Get Some REST With The New Tungsten API

Introduction

Standards allow modern systems to advance, and the REST specification is one very important example.

Since this post is about the Tungsten API in specific, let me simply say this - a RESTful API adheres to six principles: Client-server, stateless, cacheable, uniform interface, layered system and optionally code on demand.

At Continuent, our focus has been to constantly improve the product, with multiple goals. For example, some of the target principles we employ are (in alpha order):

  • data availability
  • data integrity
  • enterprise-quality stability
  • manageability/user interface
  • security
  • watchability/monitoring

As part of the watchability/monitoring goal, our upcoming version 7.0.0 includes a proper RESTful API - what we call APIv2. This API will be fully documented and public.

A Brief History

Tungsten has had an API for a long time, what we call APIv1, which was private/unpublished. This API existed only in the Manager, and was not fully REST-compliant. Additionally, many of the calls resulted in massive response payloads and significant overhead within the Manager process itself.

The original version of the Tungsten Dashboard was built upon Tungsten APIv1, and was correspondingly slow due to the APIv1 implementation.

With the advent of APIv2, the Tungsten Dashboard has been updated to make use of the new calls. The result is much improved performance overall - both from the user perspective and with the lowered impact on the cluster nodes.

As we move forwards with a dual-pronged approach to provide automated cluster deployment, APIv2 will be at the core of our Kubernetes and Tungsten Cloud efforts.

The Basics: Nuts & Bolts

In the new Tungsten v7.0.0 release, there will be an API for each layer of the cluster: Connector, Manager and Replicator. This allows for granular access and control of cluster data and operations.

The API is enabled by default and listens on localhost (127.0.0.1) only.

The HTTP/S GET method is used for read-only calls, and the HTTP/S POST method for write calls.

Each cluster process listens for API requests using the following ports by default:

Port Tungsten Component
8090 Manager
8096 Connector
8097 Replicator

The Basics: Security

SSL is enabled by default and uses the same keystore, certificates and aliases as the cluster TLS does.

Authentication is handled using username/password pairs for now, and an initial admin user account MUST be created before the API can be used.

If no admin user has been created, only the ping and createAdminUser API calls will be available.

Once the user has been created, add the username and password to every API call. For example to specify a user of tungsten with a password of secret when using the curl command:

curl --user tungsten:secret ...

Users created through the API are host-specific only, and will have to be re-created on each host of the cluster. At install time, the `tpm` command will handle populating the API user and password on all cluster nodes when you specify the rest-api-admin-user and rest-api-admin-pass options.

The Connectors receive their cluster configuration information from the Managers. To prevent errors, cluster configuration write access from the API has been disabled when the Connector is linked to a Manager.

Getting Started: Introduce Yourself

There are many ways to interact with the API.

The first step is to introduce yourself to the API by creating the required admin user account on all nodes if you did not do so at install time.

To take this first step, we will illustrate three methods and recommend a fourth:

  • Tungsten Dashboard
  • tapi command
  • curl command
  • Postman application - you can use Postman to accomplish this task, and you may download it for free here: https://www.postman.com/downloads/

Admin User Setup Via Tungsten Dashboard

Admin User Setup Via the tapi Command

Initialize all the cluster nodes (six in the examples) one-by-one via tapi:

shell> for num in `seq 1 6`; do
> echo Initializing host db$num
> tapi --create --host db${num}.yourdomain.com --user demouser --password demopass -v
> done

Admin User Setup Via the curl Command

Craft a JSON payload similar to the following:

{
    "payloadType": "credentials",
    "user": "demouser",
    "pass": "demopass"
}

Send that payload via curl to each node that needs initialization. For example:

shell> for num in `seq 1 6`; do
> echo Initializing host db$num
> curl -s --insecure --header "Content-Type: application/json" --request POST --data '{"payloadType":"credentials","payload":{"user":"demouser","pass":"demopass"}}' "https://db${num}.yourdomain.com:8090/api/v2/createAdminUser?I-understand-I-need-to-update-all-nodes=true"
> done 

HINT: To read from a file instead, specify to --data the file name with an @ prepended. For example, to read from the file `creds.json`:

curl -s --insecure --header "Content-Type: application/json" --request POST --data @creds.json 'https://127.0.0.1:8090/api/v2/createAdminUser?I-understand-I-need-to-update-all-nodes=true'

Admin User Test Via the tapi Command

Once the admin user account has been created, test access using the new credentials:

shell> tapi --user demouser --password demopass --status | jq
{
  "payloadType": "StatusPayload",
  "payloadVersion": "1",
  "payload": {
    "dataServiceName": "east",
    "dataSourceName": "db1-demo.continuent.com",
    "startTime": "2021-09-20T12:08:47.744 UTC",
    "uptimeSeconds": 8355,
    "state": "ONLINE",
    "isCoordinator": false,
    "isWitness": false,
    "managerPID": 19278,
    "parentPID": 19253,
    "policyMode": "AUTOMATIC",
    "coordinator": "db3-demo.continuent.com"
  }
}

HINT: Install the very useful `jq` utility to easily format json output.

Admin User Test Via the curl Command

Once the admin user account has been created, test access using the new credentials:

shell> curl -s --user demouser:demopass --insecure --header "Content-Type: application/json" --request GET 'https://127.0.0.1:8090/api/v2/manager/status' | jq
{
  "payloadType": "StatusPayload",
  "payloadVersion": "1",
  "payload": {
    "dataServiceName": "east",
    "dataSourceName": "db1-demo.continuent.com",
    "startTime": "2021-09-17T19:32:22.946 UTC",
    "uptimeSeconds": 87,
    "state": "ONLINE",
    "isCoordinator": false,
    "isWitness": false,
    "managerPID": 3812,
    "parentPID": 3781,
    "policyMode": "AUTOMATIC",
    "coordinator": "db3-demo.continuent.com"
  }
}

HINT: Install the very useful `jq` utility to easily format json output.

Admin User Setup: Record the Admin User

The last step of the introduction is to record the username and password in the tpm configuration, typically in an INI file. On all nodes, add the following two options and run `tpm update`:

rest-api-user=demouser
rest-api-pass=demopass

As mentioned earlier, the `tpm` command will handle populating the API user and password on all cluster nodes at install time when you specify the rest-api-admin-user and rest-api-admin-pass options in the configuration.

Admin User Recorded Test Via the tapi Command

Once the API admin user has been recorded with the tpm command, the tapi command will have access to that information and will use it automatically when found.

We can use this behavior to test that the tpm update command worked by excluding the --user and --password arguments to tapi:

shell> tapi --status | jq
{
  "payloadType": "StatusPayload",
  "payloadVersion": "1",
  "payload": {
    "dataServiceName": "east",
    "dataSourceName": "db1-demo.continuent.com",
    "startTime": "2021-09-21T12:53:08.149 UTC",
    "uptimeSeconds": 10564,
    "state": "ONLINE",
    "isCoordinator": true,
    "isWitness": false,
    "managerPID": 4232,
    "parentPID": 4207,
    "policyMode": "AUTOMATIC",
    "coordinator": "db1-demo.continuent.com"
  }
}

Getting Started: What Now?

Now that the API is ready for action, it is time to use it.

What are some high-payoff changes you can make using the new API?

Since the API is so big, we would like to offer some ideas for how to use it.

We will first examine how to replace the older monitoring scripts with the lighter-weight API calls, then will explore in more depth how to use the tapi tool to interact with the API to perform daily cluster admin tasks.

Get Some REST: Updated Monitoring Tools

In this section, we describe the available Nagios and Zabbix-specific tapi command options and how they function as compared to the original monitoring scripts (i.e. check_tungsten_* and zabbix_tungsten_*).

The original Nagios and Zabbix check scripts relied upon command-line tools like cctrl and trepctl which can have long execution times and high overhead.

The functionality provided by the original check scripts has been duplicated in the new tapi command.

The tapi command uses the new API to gather the monitoring information instead of the command-line tools, making it faster and more efficient for monitoring purposes. 

The tapi-based check functionality will behave exactly the same as the old cli tools, allowing for easy, drop-in replacement. 

Once the old monitoring scripts have been replaced with the new tapi-based tools, performance should improve and load should drop.

There are five available check scripts, and the associated replacement tapi commands are listed below:

Description Nagios CLI Tool Zabbix CLI Tool Nagios via tapi Zabbix via tapi
Check the host for running services (Manager, Connector and Replicator) check_tungsten_services zabbix_tungsten_services tapi --services tapi --services --zabbix
Check the manager to see if the datasource is online check_tungsten_online zabbix_tungsten_online tapi --online tapi --online -z
Check the manager to see if the cluster is in automatic policy mode check_tungsten_policy zabbix_tungsten_policy tapi --policy tapi --policy -z
Check the replicator to see that it is making progress over time check_tungsten_progress zabbix_tungsten_progress tapi --progress tapi --progress -z
Check the replicator for latency check_tungsten_latency zabbix_tungsten_latency tapi --latency -w 5 -c 10 tapi --latency -w 5 -c 10 -z

Get Some REST: What Is Available?

The new API can seem vast.

Since the tapi script was originally written to help test every available path in the API, a list of all available calls can be obtained easily.

Let’s get a look at the entire list of API calls available with the --listapi option, and also demonstrate the use of three key component options (-C, -M, -R) which help with setting the port and parts of the call path.

The list columns are: `Component`, `Unique Alias Per Component`, `Request Method`, and `API Call Path`:

shell> tapi --listapi -C
connector    addDataService       POST /connector/addDataService
connector    addDataSource        POST /connector/addDataSource
connector    configmodule         BOTH /connector/configuration/module/%s/%s
connector    configuration        GET  /connector/configuration
connector    connector            GET  /connector/configuration/module/connector
connector    datasource           GET  /connector/service/%s/datasource/%s
connector    datasources          GET  /connector/service/%s/datasources
connector    extra                GET  /connector/configuration/module/extra
connector    listAdminUsers       GET  /listAdminUsers
connector    module               GET  /connector/configuration/module/%s
connector    offline              POST /connector/offline
connector    onhold               POST /connector/onhold
connector    online               POST /connector/online
connector    processList          GET  /connector/processList
connector    reset                POST /connector/reset
connector    router               GET  /connector/configuration/module/router
connector    security             GET  /connector/configuration/module/security
connector    service              GET  /connector/service/%s
connector    services             GET  /connector/services
connector    servicesmap          BOTH /connector/configuration/module/servicesmap
connector    statistics           GET  /connector/service/%s/statistics
connector    status               GET  /connector/status
connector    userMap              GET  /connector/configuration/module/userMap

shell> tapi --listapi -M
manager      apiping              GET  /ping
manager      apiversion           GET  /tungstenVersion
manager      backup               POST /manager/control/service/%s/datasource/%s/backup
manager      clearArchive         POST /manager/control/service/%s/datasource/%s/clearArchive
manager      clearStandby         POST /manager/control/service/%s/datasource/%s/clearStandby
manager      clusterStatus        GET  /manager/cluster/status
manager      clusterTopology      GET  /manager/cluster/topology
manager      command              POST /manager/control/service/%s/datasource/%s/%s/%s
manager      createAdminUser      POST /createAdminUser
manager      dsCompositeStatus    GET  /manager/status/composite/%s/service/%s/datasource/%s
manager      dsServiceStatus      GET  /manager/status/service/%s/datasource/%s
manager      dsfail               POST /manager/control/service/%s/datasource/%s/fail
manager      dsoffline            POST /manager/control/service/%s/datasource/%s/offline
manager      dsonline             POST /manager/control/service/%s/datasource/%s/online
manager      dsping               GET  /manager/control/service/%s/datasource/%s/ping
manager      dspromote            POST /manager/control/service/%s/datasource/%s/promote
manager      dsrecover            POST /manager/control/service/%s/datasource/%s/recover
manager      dsshun               POST /manager/control/service/%s/datasource/%s/shun
manager      dswelcome            POST /manager/control/service/%s/datasource/%s/welcome
manager      fail                 POST /manager/control/service/%s/fail
manager      failover             POST /manager/control/service/%s/failover
manager      heartbeat            POST /manager/control/service/%s/heartbeat
manager      offline              POST /manager/control/service/%s/offline
manager      online               POST /manager/control/service/%s/online
manager      ping                 GET  /manager/service/ping
manager      policy               GET  /manager/service/%s/policy
manager      promote              POST /manager/control/service/%s/promote
manager      recover              POST /manager/control/service/%s/recover
manager      restore              POST /manager/control/service/%s/datasource/%s/restore
manager      serviceStatus        GET  /manager/status/service/%s
manager      setArchive           POST /manager/control/service/%s/datasource/%s/setArchive
manager      setPolicy            POST /manager/service/%s/policy/%s
manager      setStandby           POST /manager/control/service/%s/datasource/%s/setStandby
manager      shun                 POST /manager/control/service/%s/shun
manager      status               GET  /manager/status
manager      switch               POST /manager/control/service/%s/switch
manager      taskid               GET  /manager/task/%s
manager      tasks                GET  /manager/tasks
manager      troffline            POST /manager/control/service/%s/datasource/%s/replicator/offline
manager      tronline             POST /manager/control/service/%s/datasource/%s/replicator/online
manager      user                 GET  /manager/user/%s
manager      welcome              POST /manager/control/service/%s/welcome

shell> tapi -L -R
replicator   backup               POST /replicator/service/%s/backup
replicator   backupCapabilities   GET  /replicator/service/%s/backupCapabilities
replicator   check                POST /replicator/service/%s/check
replicator   createAdminUser      POST /replicator/createAdminUser
replicator   deleteAdminUser      POST /replicator/deleteAdminUser
replicator   heartbeat            POST /replicator/service/%s/heartbeat
replicator   listAdminUsers       GET  /replicator/listAdminUsers
replicator   off                  POST /replicator/offline
replicator   offline              POST /replicator/service/%s/offline
replicator   on                   POST /replicator/online
replicator   online               POST /replicator/service/%s/online
replicator   perf                 GET  /replicator/service/%s/perf
replicator   ping                 GET  /replicator/ping
replicator   restore              POST /replicator/service/%s/restore
replicator   services             GET  /replicator/services
replicator   status               GET  /replicator/service/%s/status
replicator   task                 GET  /replicator/task/%s
replicator   tasks                GET  /replicator/tasks
replicator   tungstenVersion      GET  /replicator/tungstenVersion
replicator   user                 GET  /replicator/user/%s

You can also do a simple filter of the output using -filter or -F, for example:

shell> tapi -L -F off
connector    offline              POST /connector/offline
manager      dsoffline            POST /manager/control/service/%s/datasource/%s/offline
manager      offline              POST /manager/control/service/%s/offline
manager      troffline            POST /manager/control/service/%s/datasource/%s/replicator/offline
replicator   off                  POST /replicator/offline
replicator   offline              POST /replicator/service/%s/offline

Anywhere you see `%s` means that a value must be provided for that position in the API call. When using the tapi command, you provide those values on the command line. More on that a bit later on…

When constructing a curl command, the full URL must be specified, and the paths listed would come at the end of the URL. For example:

curl -s --insecure --user demouser:demopass --request GET ‘https://127.0.0.1:8090/api/v2/manager/status’
curl -s --insecure --user demouser:demopass --request GET ‘https://127.0.0.1:8090/api/v2/manager/service/ping’

curl -s --insecure --user demouser:demopass --request GET ‘https://127.0.0.1:8097/api/v2/replicator/services’

curl -s --insecure --user demouser:demopass --request GET ‘https://127.0.0.1:8096/api/v2/connector/processList’

Note the different ports needed for each call to the different components?

The tapi command makes this task MUCH easier! For example, just specify the component and a shortened path for calls that do not require values:

tapi -M status
tapi -M service/ping

tapi -R services

tapi -C processList

There are even CLI shortcut options for some of the more frequently used API calls. Read on for more examples...

Get Some REST: Helpful Examples

Like the new API, the tapi command functionality is extensive. There are many convenience functions to help with daily admin tasks.

Let’s begin with some simple admin command examples:

tapi --getpolicy
tapi --getds
tapi --topology
tapi --getports

tapi --getmain
tapi --getsubs

tapi --status
tapi --clusterstatus (or tapi --cs)

tapi --allstats
tapi --routers
tapi --trstats
tapi --mgrstats

You can also affect the cluster state! In the next example, we show three ways to set the cluster to policy automatic and three ways to set the policy to maintenance.

tapi --setmaint
tapi --setpolicy maintenance
tapi --setpolicy m

tapi --setauto
tapi --setpolicy automatic
tapi --setpolicy a

You are able to interact with the Replicator, for example take it offline and online:

The below is the same as `trepctl -service east offline`.

shell> tapi -R --run offline
{"payloadType":"TaskPayload","payload":{"taskId":"d27ae273-e96f-4951-a07b-dac68771c106","state":"in_progress"},"payloadVersion":"1"}


The below is the same as `trepctl -service east online`.

shell> tapi -R -r online
SUCCESS: curl POST of https://127.0.0.1:8097/api/v2/replicator/service/east/online OK

Note the interesting details in the above commands:

  • The offline action returned a special task payload, containing both an ID and a state. The online action returned a simple success message. 
  • Of even greater interest is the fact that the tapi script automatically supplied the primary replicator service name in the API call path, in this example “east”.

The tapi script has some helper logic to make your life more RESTful when using the powerful --run feature.

If the target API call path that the alias points to contains a ‘%s’, then the following applies:

  • if you have specified --service {service name} (or -s {service name}) on the command line, use {service name} exclusively
  • if you have specified --allservices on the command line, then tapi will gather a list of all available replicator service names on that node and iterate through them, including composite active/active sub-services that contain _from_
  • otherwise, use the main replicator service name of the cluster node

Configuration Options

It is easy to configure the various options for the REST API, and each component has a separate set, shown below. The defaults are enclosed with [square brackets]. As always, leave off the leading `--` when using an INI file.

--manager-rest-api=[true]|false
--manager-rest-api-address=[127.0.0.1]
--manager-rest-api-port=[8090]
--manager-rest-api-authentication=[true]|false
--manager-rest-api-ssl=[true]|false

--connector-rest-api=[true]|false
--connector-rest-api-address=[127.0.0.1]
--connector-rest-api-port=[8096]
--connector-rest-api-authentication=[true]|false
--connector-rest-api-ssl=[true]|false

--replicator-rest-api=[true]|false
--replicator-rest-api-address=[127.0.0.1]
--replicator-rest-api-port=[8097]
--replicator-rest-api-authentication=[true]|false
--replicator-rest-api-ssl=[true]|false

Wrap-Up

For a more in-depth view of REST, please visit https://restfulapi.net and https://en.wikipedia.org/wiki/Representational_state_transfer.

In the blog post we covered the basics of the new REST API available with Tungsten version 7.0.0.

In a future post we will dive in into the details of the API, including the payloads and advanced functionality. Stay tuned!

About the Author

Eric M. Stone
COO and VP of Product Management

Eric is a veteran of fast-paced, large-scale enterprise environments with 35 years of Information Technology experience. With a focus on HA/DR, from building data centers and trading floors to world-wide deployments, Eric has architected, coded, deployed and administered systems for a wide variety of disparate customers, from Fortune 500 financial institutions to SMB’s.

Add new comment