"Why Didn't Our Database Fail Over?" - When NOT Failing Over Is the Right Answer

A database failover that doesn't happen can be just as alarming as one that does. A writer node goes unresponsive for ten minutes, customers feel it, and the cluster - which exists precisely to handle this kind of thing - sits there and rides it out instead of promoting a replica. The natural first reaction is that something is broken.

But in a well-designed database clustering system, not failing over is frequently the correct behavior, and understanding why is one of the more useful things you can learn about how your cluster actually makes decisions. This is the story of one such incident, why the cluster deliberately held its position, and a tour through the state machine that made that call - including exactly which conditions do trigger a failover and why the defaults are set the way they are.

The Incident

The environment was a five-node Tungsten Cluster (Composite Active/Active, Continuent 7.2.0) running Percona MySQL 8.4 on Rocky 8, on AWS. The cluster had recently been expanded from three nodes to five for a MySQL upgrade - a common pattern: grow the node count during an upgrade, then contract once you're confident everything's healthy.

One evening at 21:11 UTC, the writer node suffered an AWS-confirmed EBS I/O stall lasting about ten minutes. The important detail is the shape of the failure:

• The EC2 instance stayed up.
• MySQL did not restart.
• Network connectivity and login to MySQL kept working throughout.
• What did not work was reading data - anything that needed InnoDB I/O hung.

The other nodes saw the writer flip into an UNKNOWN state, and the writer's own Tungsten Manager logged its master-status check timing out over and over. And yet the cluster did not promote a replica. Service recovered on its own when EBS I/O resumed at 21:35:45 UTC. The writer role was later relocated to another node by a deliberate manual switch the next morning.

So: writer unresponsive, monitor checks failing, five healthy nodes standing by - and no failover. Why?

The Root Cause: It Was Slow, Not Dead

Here's the crux. The monitor's failure was a query-results read timeout - the checker query connected and authenticated fine, then hung waiting for results because the underlying storage couldn't answer. It was not a connection failure and not an authentication failure.

That distinction is everything, because Tungsten's dataserver state map treats those outcomes completely differently. A query-results timeout is configured as non-fatal by default. Only a failure to connect (socket_connect_timeout) or a failure to log in (login_response_timeout) carries a "fail" action - and neither of those happened, because TCP connect and login to MySQL kept succeeding all the way through the stall.

Since the master never transitioned to a FAILED state, the automatic failover - which is triggered by a FAILED master - was never invoked. And because the node never dropped out of group communication (the host was up, heartbeats and host-pings stayed healthy), there was no quorum or membership event to drive a coordinator-side failover either. The reader nodes independently logged the identical state sequence for the writer, confirming a consistent cluster-wide view. Every node agreed: the writer is reachable but slow, not gone.

This is expected, as-designed behavior. Continuent intentionally treats a master that's reachable-but-slow as transient- cycling through states like UNKNOWN, SUSPECT, and TIMEOUT - rather than declaring it failed. The whole point is to avoid failing over on query slowness, lock contention, or brief I/O blips, all of which are common and self-correcting. The uncomfortable truth is that a sustained EBS hang which keeps the connection and login alive is, to the monitor, indistinguishable from ordinary transient slowness. The cluster made the same call it would make for a heavy report query or a momentary storage hiccup - and most of the time, that's exactly the call you want it to make.

During this specific incident, the writer even briefly recovered to ONLINE at a few points (around 21:21, 21:25, and 21:30) as the I/O intermittently returned. Which, as we'll see, matters a lot if you're tempted to make the rules more aggressive.

How Tungsten Cluster Actually Decides: The Statemap

The behavior above isn't magic or hard-coded - it's driven by a configuration file, statemap.properties.defaults (in cluster-home/conf/, overridable with a local statemap.properties). It maps each probe result from the database-server checker to three things:

• .state - online, stopped, timeout, suspect, or unknown
• .threshold - -1, 0, or N (where N is counted in increments of 10 seconds)
• .action - none, alert, or fail

The states mean:

• online - the database answered the probe normally.
• stopped - the database is definitively down; the TCP connection was actively refused or reset.
• timeout - the database didn't answer in time at some stage (covers connect, login, query execution, query results, and send-query timeouts).
• suspect - something is wrong, but we can't prove the database is dead. Could be network, DNS, resource limits, or even our own problem (host down, no route to host, too many connections, a MySQL error, wrong credentials, open-file-limit errors, and so on).
• unknown - no determination at all (also the fallback if a mapping is missing).

The Rule Chain

When a probe comes back bad, this is the sequence:

1. Detect the fault and raise an alarm carrying that probe's threshold.
2. A time keeper increments the alarm every 10 seconds. This is why thresholds are counted in 10-second increments - a threshold of 8 is roughly 80 seconds of sustained failure. A threshold of 0 expires immediately; a threshold of -1 never expires (it's pure logging).
3. Self-heal: if a later probe returns ONLINE before the alarm expires, the alarm is deleted.
4. Fence: when the alarm actually expires, it calls the fencing routine, and the configured action decides what happens:
   • action = fail → handle the datasource failure
   • action = alert → currently logs an "ALERT NOT IMPLEMENTED" no-op warning
   • action = none (or undefined) → log only

And the "fail" action is role-aware: for a replica it marks the node failed; for a master it initializes failover.

Which Conditions Actually Cause a Failover

Putting it together, only a handful of conditions lead to a fence-and-fail by default:

• socket_io_error → state stopped, threshold 0 → immediate fail/failover. This is the only truly instant trigger - it means the connection was actively refused or reset, i.e. the database is definitively down.
• socket_connect_timeout → state timeout, threshold 8 → fails after ~80 seconds of sustained connect timeouts.
• login_response_timeout → state timeout, threshold 8 → fails after ~80 seconds.
• no_route_to_host → state suspect, threshold 6 → fails after ~60 seconds (deliberately tolerant of transient DNS/routing blips).

Everything else - including all the other timeout and suspect conditions like host_is_down, mysql_error, too_many_connections, wrong_credentials, and crucially the query-level timeouts - ships with threshold = -1, action = none. The datasource state you see in cctrl and the logs will change, and tmsvc.log will record extensive detail, but there's no fencing and no failover. (host_is_down is left at none on purpose, because a genuinely dead host is handled by a separate path - the host-liveness rules that do heartbeat-gap detection and node fencing.)

Two global gates sit over the whole thing: only the cluster coordinator executes fencing, and maintenance mode suppresses the entire chain - in maintenance mode, alarms aren't even incremented.

One footnote worth knowing: the header comment in the shipped file says only the stopped state can trigger a failure, which is slightly out of date with the file's own contents - the three timeout/suspect entries above also carry fail actions in the defaults. Trust the entries, not the comment.

"Can We Make It Fail Over on This?" - Yes, but Read This First

The obvious follow-up is whether you can tune the rules, so a stall like this does trigger a promotion. You can - the statemap is fully customizable. But there are three real risks that the defaults exist specifically to avoid, and they're worth weighing carefully:

1. Intermittent stalls reset the counter. In this very incident, the writer briefly recovered to ONLINE at 21:21, 21:25, and 21:30. Because the self-heal step deletes the alarm on any healthy probe, a consecutive-count threshold would have reset each time - and might never have tripped at all. So making a query-results timeout fatal reliably helps only for a continuous hang; a partial, flapping stall like this one could still ride straight through.
2. False-failover risk. Any legitimate event that makes the checker query slow to return - a heavy report on the monitor connection, severe lock contention, a brief storage blip - would now fail the master and trigger a promotion once it persists for the threshold. And a promotion carries its own brief write outage plus a small-but-nonzero divergence risk. This is precisely the trade-off the default is designed to avoid: you'd be trading a rare, self-correcting slowdown for a more frequent, deliberately-induced write interruption.
3. Self-fencing dependency. Escalation to FAILED here is driven by the master's own Manager. If a future stall also impairs that Manager's JVM - for instance, if its logs or THL sit on the same volume that just stalled - it might not be able to act at exactly the moment you need it to.

None of these makes tuning wrong. They make it a decision that belongs to your risk tolerance, not a switch to flip casually.

How This One Ended

After reviewing all of the above, the team made an instructive choice: they left the failover rules alone. It was the first time they'd hit this in many years, the cost of a false failover on a busy production cluster was real, and - in their words - they were reluctant to start tweaking tried-and-tested failover behavior. Instead, they added monitoring aimed directly at the actual root cause: the health of the EBS volumes themselves.

That's often the right instinct. The cluster didn't fail to protect them; it correctly declined to fail over on a condition indistinguishable from transient slowness. The real gap wasn't in the failover logic - it was the absence of an alert on the storage layer that was the true source of the outage.

Takeaways

• A cluster that doesn't fail over isn't necessarily broken. Reachable-but-slow is deliberately treated as transient. Failing over on slowness trades a rare, self-healing event for a guaranteed write interruption plus divergence risk.
• Connection/login failures are fatal; query-results timeouts are not - by default. A storage stall that keeps TCP and auth alive looks exactly like slowness to the monitor.
• The statemap is where the decisions live. state, threshold (in 10-second increments), and action together define every fence-and-fail path. socket_io_error is the only instant trigger; connect/login timeouts fail at ~80s; no_route_to_host at ~60s; nearly everything else is log-only.
• Self-heal resets alarms, so consecutive-count thresholds only reliably catch continuous faults - flapping stalls can slip through even aggressive settings.
• Fix the root cause, not just the symptom. The most durable outcome here wasn't re-tuning failover - it was monitoring the storage layer that actually stalled.

If you take one thing from this: before changing how your cluster fails over, understand why it chose not to. In this case the answer was that it was doing its job - declining to promote on a signal it's specifically built to tolerate - and the more valuable fix lived one layer down, in the storage it sits on.