How Tungsten Cluster Protects Data Integrity While Using Asynchronous Replication

Author: erics, Posted April 29th, 2026 at 8:15:21am

Excerpt

Tungsten Cluster uses asynchronous replication, but it does not treat
failover as a blind promotion exercise. The Tungsten Manager
continuously monitors datasource health and replication state, tracks
applied and stored THL positions, chooses promotion candidates based on
explicit safety rules, and delays or blocks failover and recovery paths
that could leave transactions behind.

Blog Post

Asynchronous replication is often misunderstood. It is sometimes
described as though “async” automatically means “unsafe,” but that is
not the right way to think about Tungsten Cluster. Tungsten Cluster does
use asynchronous replication, yet it surrounds that replication model
with Manager-driven controls designed to protect data integrity during
normal operation, maintenance, switch operations, and failure
recovery.

At the center of this design is the Tungsten Manager. The Manager
monitors the replication status of each datasource in the cluster, acts
as a central source of cluster health and status, communicates
datasource-state changes to Tungsten Connector so client traffic can be
redirected, and uses a rules engine to decide whether to restart
services, swap Primaries, or reconfigure connectors. In other words,
failover is not simply “pick a replica and hope.” It is a controlled
decision based on the known state of the datasources, the replicators,
and the cluster as a whole.

The underlying replication engine is Tungsten Replicator. It extracts
change data from the source database, records complete transactions into
the Transaction History Log, or THL, and then applies THL events to
target datasources. Each transaction is recorded with an incremental
global transaction ID, which allows the system to identify transactions
and determine whether appliers in the topology have written a specific
transaction to a datasource. This THL-based model gives the Manager
meaningful replication-position information to use when deciding which
replica is safe to promote.

That position tracking matters most during failover. When choosing a
new Primary, the Manager first excludes unsafe or unsuitable replicas:
replicas that are not online, not standby replicas, marked ARCHIVE,
missing an online Manager, or lacking a replicator in an online or
synchronizing state. Among the viable candidates, it compares the last
applied sequence number and also records which candidate has the highest
stored THL sequence position. If one replica has applied the most events
but another has received more THL from the Primary, the Manager can
prefer the replica with the most stored THL records, which is especially
important in failover because that node has the best chance of
preserving the greatest amount of data once the remaining stored events
are applied.

Tungsten Cluster also includes explicit latency protection. A replica
whose applied latency is above the configured promotion threshold is not
used as a failover candidate. The documented default threshold is 900
seconds, or 15 minutes. This prevents the Manager from promoting a
replica that is too stale to be considered a safe target.

The strongest data-protection behavior appears in the failover tuning
defaults. If MySQL fails on the Primary but binary logs are still
available, Tungsten Manager and Tungsten Replicator can continue working
together to extract the remaining binary log events. With the documented
default replicator.store.thl.stopOnDBError=false, failover
is delayed until all available events have been read from the Primary
binary logs and all events have reached the replicas. The documentation
is explicit that this default protects data integrity; the alternative
setting allows faster failover but risks leaving binlog events
behind.

The second major guardrail is the THL apply wait rule. During
failover, the Manager waits for the candidate replica to apply all
stored THL events before promoting it to Primary. The documented default
manager.failover.thl.apply.wait.timeout=0 means the Manager
waits indefinitely. The reason is direct: once a replica is promoted,
any unapplied stored THL events are ignored and therefore lost. For
data-first HA, that default is exactly the posture you want: prefer a
longer failover over an avoidable loss of transactions.

Recovery is also protected. After a failover, the old Primary is not
casually allowed back into the topology. If the old Primary still
contains unextracted binlog events, or if it has more THL than the
replica that was promoted, cctrl recover fails. Tungsten
also provides the tungsten_find_orphaned command to
identify orphaned binlog or THL events that did not make it safely
through the failover path. This is an important distinction: the system
does not pretend an unsafe recovery is clean. It detects the risk and
forces the operator to deal with it.

Tungsten Cluster also uses quorum and shunning behavior to reduce
split-brain risk. The Manager documentation explains that clusters need
an odd number of members, or a witness when using an even number of
database nodes, so that a majority can be established during a network
partition. If a partition does not have a majority, the Primary in that
partition can be shunned until quorum is restored. In a complete network
partition, nodes can enter FAILSAFE-SHUN mode specifically
to protect data from split brain.

The practical takeaway is simple: Tungsten Cluster’s replication is
asynchronous, but its failover model is not careless. The Manager tracks
replication progress at both the applied and stored THL levels, filters
promotion candidates for health and latency, delays failover to extract
remaining Primary binlog events, waits for stored THL to be applied
before promotion, blocks unsafe recovery of an old Primary, and uses
quorum and shunning to avoid split-brain behavior. That combination
gives Tungsten Cluster a data-integrity-oriented HA model: asynchronous
replication with carefully designed mechanisms to keep the cluster as
consistent and recoverable as possible when failures occur.

Suggested Meta Description

Tungsten Cluster uses asynchronous replication, but Tungsten Manager
protects data integrity through THL position tracking, latency checks,
failover delays, recovery safeguards, and split-brain protection.

Suggested Social Summary

Tungsten Cluster uses async replication, but failover is governed by
Manager safeguards: THL position tracking, replica latency checks,
binlog-drain behavior, THL apply waits, unsafe recovery detection,
quorum, and FAILSAFE-SHUN protection.

Sources

• https://docs.continuent.com/release-notes.html
• https://docs.continuent.com/tungsten-clustering-8.0/index.html
• https://docs.continuent.com/tungsten-clustering-8.0/introduction-replicator.html
• https://docs.continuent.com/tungsten-clustering-8.0/manager-intro.html
• https://docs.continuent.com/tungsten-clustering-8.0/manager-failover-tuning.html

Categories: Technology Tags: