When Tungsten Replication "Stalls" but Nothing Has Failed: A MySQL Metadata-Query Story

Every so often a support case is interesting not because something broke loudly, but because nothing broke at all - and yet forward progress simply stopped. This is one of those cases. A Tungsten Replicator that had been running happily for months suddenly stopped advancing its sequence number, threw no fatal error, and left everyone staring at a perfectly healthy-looking ONLINE state. The culprit turned out to have very little to do with replication itself and almost everything to do with how MySQL was answering a routine metadata question.

Here's the walkthrough, the root cause, and the fix - including a configuration change that can keep this class of problem from ever biting you.

The Topology

The environment was a fairly common shape for teams that want Tungsten's clustering guarantees upstream but need plain MySQL row-based replication (RBR) downstream:

• A standalone Tungsten Cluster (three nodes) as the source of truth.
• A standalone Tungsten Replicator subscribing to that cluster.
• That replicator re-emitting a traditional row-based binlog stream into GCP Datastream, which only speaks native MySQL replication.

In other words, the standalone replicator exists to translate Tungsten's transaction history log (THL) into the vanilla RBR that Datastream can consume. The source and target were both MySQL 8.0 on Rocky Linux 9, with the Replicator on version 7.2.0 and the cluster on 7.1.4.

The Symptom

The report was precise and, at first glance, contradictory:

• Replication was not marked as failed.
• The Replicator sequence number (seqno) was not moving forward.
• No replication stream queries were visibly executing on the target.

That combination is the classic signature of a long-running transaction - the apply thread is busy, so nothing errors, but nothing commits either. So the first question was the obvious one: had any large update, schema change, or version bump gone out recently? The answer was no.

What the customer did notice was one particular query running over and over on the target, each instance taking roughly ten seconds, most of that time spent checking permissions. As soon as one finished, another appeared for a different shard and table:

SELECT ... FROM information_schema.columns
WHERE table_schema LIKE 'shard_3158'
  AND table_name LIKE 'account'
  AND column_name LIKE '%'
ORDER BY table_cat, table_schem, table_name, ordinal_position

That query is issued by Tungsten itself. And that detail is the whole case.

Why is Tungsten Querying information_schema at All?

When MySQL logs a change in row-based format, the binary log event contains column values but not column names. To turn a raw RBR event back into a valid SQL statement against the target, the replicator needs to know the shape of the table - column names, data types, nullability, and so on. It gets that by asking information_schema.columns.

To avoid asking on every single event, Tungsten keeps an in-memory metadata cache. Once it has fetched the definition of a table, it reuses it. The cache is invalidated in two situations:

1. A DDL change is detected on a table - in which case only that one table's entry is dropped.
2. The replicator restarts - in which case the whole cache is gone and has to be rebuilt from scratch.

Hold onto point 2. It's the trap door.

What Actually Happened

Buried in the logs was a restart the customer hadn't triggered by hand:

2026/02/05 20:00:46 | Anchor file deleted. Shutting down.
2026/02/05 20:07:39 | --> Wrapper Started as Daemon

This turned out to be an automated OS patching cycle - a dnf update followed by a standard reboot. That reboot restarted both mysqld and Tungsten at roughly the same time. Two things then happened together:

• The Tungsten metadata cache was wiped by the restart, so the replicator now had to re-fetch column metadata for every shard and table it touched - and this was a heavily sharded database, so that's a lot of information_schema.columns lookups.
• MySQL, freshly restarted, was answering those metadata lookups in ~10 seconds each instead of milliseconds.

Multiply a very slow metadata query by a cold cache that needs to be rebuilt table-by-table, and apply throughput collapses. The replicator wasn't stuck on a giant transaction, and it wasn't failing - it was faithfully doing exactly what it's supposed to do, just at a glacial pace because each metadata answer it needed was taking thousands of times longer than normal. From the outside, that's indistinguishable from a stall.

A side observation reinforced that the problem lived in MySQL, not Tungsten. During the stall, a SHOW FULL PROCESSLIST snapshot was captured on the target. In the same snapshot as Tungsten's slow metadata query, the Datastream user was independently stuck on its own INFORMATION_SCHEMA.COLUMNS lookup (which is a routine capability probe that its driver issues), with several connections each "executing" for seconds at a time:

50276  data_stream  10.239.118.37:51868  NULL  Query  8  executing
SELECT * FROM INFORMATION_SCHEMA.COLUMNS
 WHERE UPPER(TABLE_SCHEMA) = 'INFORMATION_SCHEMA'
   AND UPPER(TABLE_NAME) = 'COLUMNS'
   AND UPPER(COLUMN_NAME) = 'SRS_ID'

Two unrelated clients both crawling on metadata reads points at the shared resource (the database's internal state), rather than at either client. This is also why the closing advice below says to capture the processlist during a stall: one snapshot was enough to separate a MySQL metadata-performance problem from a replication fault.

The Fix That Ended the Outage

The customer resolved it with a firm restart of the whole stack on the standalone replicator:

replicator stop
systemctl stop mysqld
systemctl disable mysqld
reboot
systemctl enable mysqld
systemctl start mysqld
# sanity-check with a few live queries
replicator start

After the reboot, metadata queries returned to normal speed, the cache refilled quickly, and the seqno started climbing again - applied latency back under a second. The degraded internal MySQL state that the first (automated, possibly rushed) restart had left behind was cleared by a clean, deliberate one.

That's an unsatisfying root cause on its own - "we turned it off and on again properly" - so the more useful part of the story is what it revealed about the metadata query, and how to make it cheap enough that a slow answer never matters as much.

The Deeper Fix: a Cheaper Metadata Query

Look closely at the original query's WHERE clause. It uses LIKE for exact values:

WHERE table_schema LIKE 'shard_3158' AND table_name LIKE 'account' AND column_name LIKE '%'

LIKE against a literal with no wildcards still forces MySQL down a pattern-matching path, and on MySQL 8's data-dictionary-backed information_schema that can turn into a full scan of the tables metadata. Tungsten has long shipped an optimized variant that swaps the pattern match for exact string comparison:

WHERE table_schema = 'shard_3158' AND table_name = 'account'
ORDER BY table_cat, table_schem, table_name, ordinal_position

The difference in the query plan is stark. In the unoptimized version, the tbl table is joined with type = ALL - a full scan of ~120,000 rows via a hash join:

| tbl | ALL    | ... | NULL      | ... | 121846 | Using where; Using join buffer (hash join) |

With the optimized query, that same access becomes an eq_ref primary-key lookup of a single row:

| tbl | eq_ref | ... | schema_id | ... |      1 | Using where |

On a live production box, the measured execution time of that query dropped from ~0.09s to effectively 0.00s once the optimization was applied. That may sound trivial, but recall the failure mode: after a restart the replicator fires this query for every table across many shards, and under a degraded MySQL state the per-query cost balloons. Shaving the baseline down and removing the full scan makes the whole metadata-rebuild phase dramatically cheaper and far more resilient to a temporarily unhappy database.

The optimized query is not on by default, but it's available from Tungsten 7.0.0 onward and enabled with a single property:

property=replicator.datasource.global.connectionSpec.usingOptimizedMetadataQuery=true

Why isn't the faster query the default? Deliberate conservatism.

LIKE and = don't always agree: for a schema or table name containing multibyte characters, the exact-comparison form can match a different row or none at all. In a metadata-fetch path that's not a cosmetic difference: the replicator would bind column metadata for the wrong table, or fail to find it, and silently mis-apply or stall. Very few deployments use multibyte identifiers, but a performance win that carries even a small wrong-metadata risk can't be switched on for everyone unseen. A future release may well reverse the polarity, making the optimized query the default and offering a setting to restore the pattern-match path for the rare deployment with multibyte schema or table names.

The recommendation is to benchmark both variants on your own replica first - schema shape and MySQL version both influence the win - then set it in tungsten.ini and roll it out.

Cleaning Up the Noise

One more thing surfaced during analysis, unrelated to the stall but worth mentioning because it made troubleshooting harder. The host still had a legacy native MySQL replica channel configured, and its I/O thread was repeatedly trying to connect and fetch a binlog file that had long since been purged:

Could not find first log file name in binary log index file (server_errno=1236)

That channel wasn't the cause of anything - but a stream of fatal 1236 errors in the log is exactly the kind of red herring that costs time during an incident. If you have an old channel that's no longer part of your intended flow, disable and remove it, so the only replication activity in the logs is the activity you actually care about.

Takeaways

A few things are worth carrying out of this case:

• A stalled seqno with no error is often not a replication bug. When the apply pipeline is healthy but not advancing, look at what the target database is doing - long-running metadata or information_schema queries are a prime suspect.
• Restarts are not free for row-based apply. A replicator restart wipes the metadata cache, and everything the replicator touches afterward has to re-fetch its table definitions. If the database is slow to answer those, a routine restart can look like an outage.
• Sequence your patching. Graceful shutdown of the replicator is good, but also control the startup order after a reboot - bring MySQL up, confirm it's answering metadata queries at normal latency, and only then start the replicator. Bake a post-patch latency check into the workflow.
• Turn on the optimized metadata query if you run a large, heavily sharded schema. It removes a full scan from a hot path and makes the metadata-rebuild phase far more forgiving.
• Keep the logs clean. Retire legacy replication channels, so incident logs contain signal, not noise.

If you want the fastest possible confirmation the next time something like this shows up, capture SHOW PROCESSLIST (and, if you can, performance_schema) during the stall and look specifically for long-running INFORMATION_SCHEMA queries and any metadata lock waits. That single snapshot is usually enough to tell a MySQL metadata-performance problem apart from a genuine replication fault - and it points you straight at the fix.