Easier Log Analysis with tungsten_merge_logs

Introduction

In this blog post we explore the new command `tungsten_merge_logs` which merges log files under a directory tree into a single file.

There are many moving parts to a cluster and they are spread over multiple nodes (usually three).

When there is an issue, the logs are the key resource to find out what is going on.

The best practice is to gather the log files into one place and then read through them all.

This can be difficult with many files on multiple nodes.

For this reason, the `tungsten_merge_logs` tool was created.

The tungsten_merge_logs command is designed to aid troubleshooting by consolidating the various log files into one place (merged.log), ordered by time.

Log File Love

The three main log files are:

Component Log File Path
Connector connector.log $CONTINUENT_ROOT/tungsten/tungsten-connector/log
Manager tmsvc.log $CONTINUENT_ROOT/tungsten/tungsten-manager/log
Replicator trepsvc.log $CONTINUENT_ROOT/tungsten/tungsten-replicator/log

There is also a shortcut link for each located inside $CONTINUENT_ROOT/service_logs:

tungsten@db1-demo:/opt/continuent/service_logs # ls -l
total 36
lrwxrwxrwx 1 tungsten tungsten   66 Oct 22 15:32 connector-user.log -> /opt/continuent/tungsten/tungsten-connector/log/connector-user.log
lrwxrwxrwx 1 tungsten tungsten   61 Oct 22 15:32 connector.log -> /opt/continuent/tungsten/tungsten-connector/log/connector.log
lrwxrwxrwx 1 tungsten tungsten   62 Oct 22 15:32 manager-user.log -> /opt/continuent/tungsten/tungsten-manager/log/manager-user.log
-rw-rw-r-- 1 tungsten tungsten 1621 Nov 22 20:46 mysqld_exporter.log
lrwxrwxrwx 1 tungsten tungsten   62 Oct 22 15:32 mysqldump.log -> /opt/continuent/tungsten/tungsten-replicator/log/mysqldump.log
-rw-rw-r-- 1 tungsten tungsten 3237 Nov 22 19:49 node_exporter.log
lrwxrwxrwx 1 tungsten tungsten   68 Oct 22 15:32 replicator-user.log -> /opt/continuent/tungsten/tungsten-replicator/log/replicator-user.log
lrwxrwxrwx 1 tungsten tungsten   55 Oct 22 15:32 tmsvc.log -> /opt/continuent/tungsten/tungsten-manager/log/tmsvc.log
lrwxrwxrwx 1 tungsten tungsten   60 Oct 22 15:32 trepsvc.log -> /opt/continuent/tungsten/tungsten-replicator/log/trepsvc.log
lrwxrwxrwx 1 tungsten tungsten   63 Oct 22 15:32 xtrabackup.log -> /opt/continuent/tungsten/tungsten-replicator/log/xtrabackup.log

Gathering the Logs

Getting the logs is a simple procedure - just one command will do it for you: `tpm diag –all`. For example, get the log from all six nodes:

tungsten@db1-demo:/home/tungsten/diags # tpm diag --all
db1-demo.continuent.com|Collecting full cluster diagnostics
Note: if ssh access to any of the cluster hosts is denied, use "--local" or "--hosts=<host1,host2,...>"


db1-demo.continuent.com|Collecting diag information for db1-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db2-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db3-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db4-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db5-demo.continuent.com
db1-demo.continuent.com|Collecting diag information for db6-demo.continuent.com

db1-demo.continuent.com|Diagnostic information written to /home/tungsten/diags/tungsten-diag-2021-11-24-17-23-31.tgz

In our example, a diagnostic tarball was created called `/home/tungsten/diags/tungsten-diag-2021-11-24-17-23-31.tgz`, which you can extract to gain access to the log files from every node. Let extract the tarball and have a look:

shell$ tar xvzf tungsten-diag-2021-11-24-17-23-31.tgz
shell$ cd tungsten-diag-2021-11-24-17-23-31
shell$ ls -l
total 24
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:32 db1-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db2-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db3-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db4-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db5-demo.continuent.com
drwxr-xr-x 11 tungsten tungsten 4096 Nov 24 17:24 db6-demo.continuent.com

As you can see, we have six sub-directories, one per database cluster node.

Each directory will have it’s own set of log files, which of course you can view individually, i.e.:

shell$ vi `find . -name "*svc.log"`
12 files to edit

Merging the Logs

Now, as nice as it is to have a vi session with 12 files to see, it would be much better to merge all the log files into a single file sorted by host and time so that it is easier to understand what is happening at any given point in time.

Doing this is very easy - just run the new `tungsten_merge_logs` command (available in v6.1.16 NYR and v7.0.0 Beta). For example, let’s gather all 12 log files into one single `merged.log` in the current directory:

shell$ pwd
/home/tungsten/diags/tungsten-diag-2021-11-24-17-23-31

shell$ tungsten_merge_logs -L 1
New merged log file ./merged.log created!

shell$ vi merged.log

Sky’s the Limit

You may have noticed that there was an argument provided: `-L 1`. This tells the tungsten_merge_logs script to only gather the latest log from eact component. Tungsten logs will be rotated, so that a sequence number will be appended to the file, like this:

shell$ find . -name "trepsvc.log*" -ls| grep db2
552970 10244 -rw-rw-r--   1 tungsten tungsten 10485852 Oct 26 14:19 ./db2-demo.continuent.com/logs/trepsvc.log.9
563957 10248 -rw-rw-r--   1 tungsten tungsten 10485944 Oct 26 14:26 ./db2-demo.continuent.com/logs/trepsvc.log.8
563960 10244 -rw-rw-r--   1 tungsten tungsten 10485893 Oct 26 14:33 ./db2-demo.continuent.com/logs/trepsvc.log.7
552976 10244 -rw-rw-r--   1 tungsten tungsten 10485893 Oct 26 14:39 ./db2-demo.continuent.com/logs/trepsvc.log.6
563956 10248 -rw-rw-r--   1 tungsten tungsten 10485893 Oct 26 14:46 ./db2-demo.continuent.com/logs/trepsvc.log.5
552962 10244 -rw-rw-r--   1 tungsten tungsten 10485893 Oct 26 14:53 ./db2-demo.continuent.com/logs/trepsvc.log.4
563961 10244 -rw-rw-r--   1 tungsten tungsten 10485893 Oct 26 14:59 ./db2-demo.continuent.com/logs/trepsvc.log.3
562736 10244 -rw-rw-r--   1 tungsten tungsten 10485893 Oct 26 15:24 ./db2-demo.continuent.com/logs/trepsvc.log.2
563952 10244 -rw-rw-r--   1 tungsten tungsten 10485872 Oct 26 20:19 ./db2-demo.continuent.com/logs/trepsvc.log.1
552978  616 -rw-r-----   1 tungsten tungsten   630202 Nov  3 16:46 ./db2-demo.continuent.com/logs/trepsvc.log

Without the `-L 1` argument, tungsten_merge_logs will gather all logs, including the rotated ones.

More Information

Below is the help text you get when you run `tungsten_merge_logs -h`:

shell$ tungsten_merge_logs -h
Usage for tungsten_merge_logs:

   tungsten_merge_logs [options]

>>> Introduction <<<

The tungsten_merge_logs command is designed to aid troubleshooting by consolidating the various
log files into one place ordered by time.

>>> Getting Started <<<

With no options specified, the tungsten_merge_logs script will 
gather all log files in the current directory and below.

For example:
    cd
    tpm diag --all
    tar xvzf tungsten-diag-2021-11-15-16-37-33.tgz
    cd tungsten-diag-2021-11-15-16-37-33
    tungsten_merge_logs

would result in something like the following:

    New merged log file ./merged.log created!

>>> Operating Rules <<<

    All logs files are gathered by default.

    If you specify any of --connector, --replicator or --manager,
    then only the log files for the specifed components will be gathered.

    Using multiple options will aggregate the logs from the specified components.

>>> Examples <<<

  tungsten_merge_logs
  tungsten_merge_logs -h
  tungsten_merge_logs -R
  tungsten_merge_logs -C
  tungsten_merge_logs -M
  tungsten_merge_logs -M -R -L 1

>>> Command-Line Options <<<

    [-L|--log-limit] Specify the quantity of log files to gather (default: unlimited).
      When the wrapper rotates log files, it appends a period and an integer to the end of the log file name, when .1 is the newest and .2 is older and .3 older than that, etc.
      This parameter will gather the base log file plus limit minus one rotated files.
      Examples:
        - a loglimit of 1 means gather the base file only, i.e. trepsvc.log
        - a loglimit of 2 means gather the base file and the first backup file trepsvc.log.1
        - a loglimit of 3 means gather the base file and the first two backup files, trepsvc.log.1 and trepsvc.log.2

    [-C|--connector] Specify the connector 
    [-M|--manager] Specify the manager 
    [-R|--replicator] Specify the replicator 
    [-x|--extension {string}] Specify the file extension (default: log) Do NOT include the period.

    [-h|--help]
    [-q|--quiet]
    [-v|--verbose]
    [-d|--debug]

Summary

In this blog post we explored the new command `tungsten_merge_logs`, which merges log files under a directory tree into a single file `merged.log`.

If you are a Continuent customer, please submit a Zendesk ticket if you have any questions; otherwise, feel free to reach out to learn more about Tungsten Clustering.

About the Author

Eric M. Stone
COO and VP of Product Management

Eric is a veteran of fast-paced, large-scale enterprise environments with 35 years of Information Technology experience. With a focus on HA/DR, from building data centers and trading floors to world-wide deployments, Eric has architected, coded, deployed and administered systems for a wide variety of disparate customers, from Fortune 500 financial institutions to SMB’s.

Add new comment