Data Synchronisation: Cross-Region Replication Patterns

Overview

A single-region financial platform fails in two ways: a regional outage takes everything down, and the regulator starts asking about your business continuity plan before the all-clear. In KSA, SAMA’s Technology Risk Management (TRM) framework is explicit: critical systems must have recovery capabilities that deliver defined RPO and RTO targets across a declared secondary site. Most banks start by deploying a warm standby in a second availability zone or data centre. The engineering question — which almost always surfaces too late — is what “in sync” actually means when you have two live copies of financial data separated by 20 milliseconds of WAN latency.

This article walks through the three main topologies, the conflict problem that every active/active deployment eventually hits, the specific configuration paths for Kafka and Postgres (the two platforms most commonly encountered alongside IBM MQ and Db2 in KSA bank estates), and the observability you need to detect drift before it becomes a reconciliation incident.

Topology choice

Three topologies dominate in practice. Choose before you build anything else — the replication mechanism, the conflict strategy, and the failover runbook all follow from this decision.

The two patterns that cause the most trouble in production are optimistic active/active (teams assume conflicts won’t happen because they partition by customer) and passive standby treated as hot (the DBA sets the standby to accept reads under normal load, but the failover runbook still assumes it needs a 15-minute warm-up). Both reveal themselves at incident time.

Conflict resolution

Conflicts in active/active arise when two regions write to the same logical row before either replication stream catches up. Four resolution strategies, from weakest to most reliable:

The most reliable pattern for financial data in active/active is write partitioning by entity: each entity (account, customer, transaction) is owned by exactly one region for writes, regardless of where reads are served. Conflicts become structurally impossible because the owning region is the single writer. The complexity shifts to routing: the application must know which region owns a given entity and route write requests accordingly — across regional boundaries if necessary.

Kafka MirrorMaker 2

MirrorMaker 2 (MM2) is the Kafka-native solution for cross-cluster topic replication. It runs inside Kafka Connect and supports bidirectional replication with offset translation — the critical feature that lets consumers in Region B resume at the correct offset after a failover, rather than replaying from the beginning of the mirrored topic.

# MirrorMaker 2 — bidirectional between Riyadh and Jeddah clusters
clusters = riyadh, jeddah

riyadh.bootstrap.servers  = kafka-riyadh.svc.internal:9093
jeddah.bootstrap.servers  = kafka-jeddah.svc.internal:9093

# TLS between regions — mandatory for financial data in transit
riyadh.security.protocol  = SSL
jeddah.security.protocol  = SSL
riyadh.ssl.keystore.location  = /certs/riyadh.keystore.jks
jeddah.ssl.keystore.location  = /certs/jeddah.keystore.jks

# Enable both directions
riyadh->jeddah.enabled  = true
jeddah->riyadh.enabled  = true

# Topics to replicate — whitelist beats blacklist for financial data
riyadh->jeddah.topics  = payment\.events, customer\.events, limit\.events
jeddah->riyadh.topics  = payment\.events, customer\.events, limit\.events

# Offset translation — critical for failover without replay
riyadh->jeddah.sync.group.offsets.enabled   = true
jeddah->riyadh.sync.group.offsets.enabled   = true
riyadh->jeddah.emit.checkpoints.enabled      = true
jeddah->riyadh.emit.checkpoints.enabled      = true
riyadh->jeddah.emit.heartbeats.enabled       = true

# Replication factor for mirrored topics in the target cluster
riyadh->jeddah.replication.factor  = 3

# Consumer lag limit before connector throttles; keep below 30 s for IPS flows
riyadh->jeddah.consumer.lag.threshold.ms  = 20000

Database replication

Kafka handles event streams. The relational database holding the ledger, account state, and customer master needs its own cross-region strategy. The two platforms most common in KSA bank estates are PostgreSQL (for newer microservices) and IBM Db2 (for core banking). Each has a distinct replication path.

PostgreSQL streaming replication is the default HA mechanism: the primary streams WAL to one or more standbys in real time. Standbys are read-only until promoted. For cross-region hot-standby, use synchronous_commit = remote_apply on the primary for zero-RPO commits at the cost of a WAN-round-trip on every write. For async replication with a small RPO lag, use synchronous_commit = off on the inter-region standby and measure the lag via pg_stat_replication.replay_lag.

wal_level                = replica
max_wal_senders          = 5
wal_keep_size            = 1024MB

# Synchronous replication to local HA standby; async to cross-region standby
synchronous_standby_names = FIRST 1 (standby-local, standby-jeddah)
# standby-local: sync, same DC — zero RPO within region
# standby-jeddah: async by position (FIRST 1 means only local must ack)

recovery_target_timeline = latest    # follow timeline switches on failover
hot_standby              = on        # allow reads from jeddah standby
hot_standby_feedback     = on        # prevent query conflicts from bloat vacuum

Patroni is the recommended HA layer for Postgres in Kubernetes. It wraps streaming replication with automatic leader election (backed by etcd or Consul), a REST health endpoint, and a switchover/failover CLI. Cross-region Patroni requires that your distributed consensus store (etcd) also spans regions — a cross-region etcd cluster with 5 nodes, 2 per region and 1 as tiebreaker, is the standard configuration.

IBM Db2 HADR is the IBM-native solution for Db2 HA/DR. Configure HADR with HADR_SYNCMODE NEARSYNC (primary waits for the log to reach the standby buffer but not be applied) for a good balance between RPO and WAN round-trip cost. SUPERASYNC gives the lowest latency but risks losing the in-flight log on primary failure.

# On primary (Riyadh)
UPDATE DB CFG FOR PAYMENTSDB USING
  HADR_LOCAL_HOST    'db2-primary.riyadh.svc.internal'
  HADR_LOCAL_SVC     50900
  HADR_REMOTE_HOST   'db2-standby.jeddah.svc.internal'
  HADR_REMOTE_SVC    50901
  HADR_REMOTE_INST   'db2inst1'
  HADR_SYNCMODE      NEARSYNC
  HADR_TIMEOUT       120
  HADR_PEER_WINDOW   120;   -- prevents split-brain automatic takeover within 120 s

START HADR ON DB PAYMENTSDB AS PRIMARY;

# Monitor replication health — run from primary
GET SNAPSHOT FOR DATABASE ON PAYMENTSDB | grep -E 'HADR|replay'

1. Enable cross-region connectivity

   Open the HADR port (Db2: TCP/50900–50901, Postgres: TCP/5432) between data centres. Use TLS/mTLS for the replication stream — unencrypted replication of financial data over a WAN is a SAMA TRM control failure.

2. Take a base backup to the standby

   Postgres: pg_basebackup -h primary -D /var/lib/postgresql/data -P -R. Db2: offline backup restore or db2inidb standby initialisation.

3. Start replication and verify lag

   Postgres: SELECT * FROM pg_stat_replication on primary; check sent_lsn - replay_lsn. Db2: GET SNAPSHOT FOR DATABASE and look at HADR log gap running.

4. Configure automatic failover

   Postgres with Patroni: leader election is automatic when the primary fails a health check. Db2: use IBM High Availability Cluster Multi-Processing (HACMP) or a custom script calling TAKEOVER HADR ON DB. Neither should fire before a peer window timeout to prevent split-brain.

5. Test failover end-to-end quarterly

   Perform a planned switchover, measure actual RTO, verify the application reconnects correctly, and walk the standby back to primary. Simulate network partition, not just a clean shutdown. SAMA expects evidence of tested DR procedures in the TRM review.

6. Document failback

   After a failover, the old primary becomes the new standby. Re-sync it from the new primary (another base backup or HADR reinitialisation) before promoting it back. Skipping this step causes divergent timelines and data loss.

Consistency & RPO/RTO

CAP theorem in a WAN multi-region deployment resolves to: you either sacrifice consistency (allow stale reads or lost writes during a partition) or availability (wait for confirmation from the second region before committing). Financial services almost universally chooses the latter for write paths — the ledger must be correct — and tolerates brief unavailability over a wrong balance.

Translate this into concrete targets:

• RPO (Recovery Point Objective) — how much data can be lost. For IPS-cleared payments: effectively zero (synchronous replication). For analytics and reporting replicas: seconds to minutes.
• RTO (Recovery Time Objective) — how long until the service is back. Patroni-managed Postgres: 10–30 seconds. Manual Db2 HADR takeover: minutes. Application-level reconnect logic adds another 10–60 seconds to the RTO you present to the business.

Failover & failback

Automated failover requires three things to be in place before the incident: a consensus mechanism that detects failure faster than the timeout, a promotion sequence that does not cause split-brain, and application-layer reconnection logic that handles the new primary endpoint without a restart.

The most common production failure in KSA multi-region deployments is the network partition that is not a real primary failure. The primary is up in Region A; the standby in Region B cannot see it due to a WAN link issue. If the standby promotes itself, both regions believe they are primary. Both write. Replication resumes when the link recovers, and the two divergent timelines must be reconciled manually — with potentially lost or duplicated payment records. The prevention is a peer window (Db2) or TTL-based fencing (Patroni + etcd): the standby will not promote until the peer window has expired and it has confirmed the old primary is fenced (e.g., its leader lock in etcd has expired).

scope: payments-cluster
name: pg-jeddah-standby

etcd3:
  hosts: etcd-r1.svc:2379,etcd-r2.svc:2379,etcd-jeddah.svc:2379  # 3 etcd across regions

bootstrap:
  dcs:
    ttl: 60                # leader key TTL; standby waits this long before considering failover
    loop_wait: 10
    retry_timeout: 30
    maximum_lag_on_failover: 1048576   # 1 MB; beyond this, standby will NOT auto-promote
    synchronous_mode: true             # only acked standbys eligible for promotion

postgresql:
  parameters:
    synchronous_commit: remote_apply  # zero RPO for synchronous standby
    wal_level: replica
    hot_standby: "on"
  use_pg_rewind: true                 # fast re-sync of old primary after failback

Failback is the step most runbooks skip. After a failover, the old primary is stale. The fastest path back: use pg_rewind (configured in Patroni as use_pg_rewind: true) to re-sync it from the new primary’s divergence point rather than running a full base backup. pg_rewind can only be used if the old primary was a sync standby at time of failure; otherwise a base backup is required. Either way, test failback in the DR exercise, not just failover.

SAMA data residency

SAMA’s Cloud Computing Regulatory Framework and TRM framework together impose three constraints on cross-region replication for regulated financial data:

• Data must remain within KSA. Both the primary and the replication target must sit in KSA data centres or a SAMA-approved cloud region in the Kingdom. Replicating to a region outside KSA — even as a cold DR copy — requires an explicit regulatory exemption.
• Encryption in transit is mandatory. All replication streams crossing any network segment — even private links between data centres — must use TLS 1.2+ with a certificate chain the bank controls. Certificate expiry on a replication channel has caused production incidents; automate rotation.
• Audit trails must be replicated. If the primary fails, the regulator expects the audit log to survive with zero gaps. A separate replication stream for audit tables (or a Kafka topic used exclusively for audit events) with a stronger consistency guarantee than the main data replication is the correct design. Separating audit from operational data in the replication architecture also makes the compliance evidence cleaner.

Observability

Cross-region replication fails silently. The primary keeps writing; the standby falls behind; the lag alert is not set; the DBA discovers the standby is 4 hours behind when they need it for a planned maintenance failover. The minimum instrumentation every multi-region deployment must have:

groups:
  - name: replication
    rules:
      # Postgres streaming replication lag > 30 s → page
      - alert: PostgresReplicationLagHigh
        expr: pg_replication_lag_seconds{standby="jeddah"} > 30
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Postgres cross-region standby lag {{ $value }}s — RTO breach risk"

      # MirrorMaker 2 consumer lag > 5000 messages → warn
      - alert: MM2ConsumerLagHigh
        expr: kafka_consumer_group_lag{group="mirrormaker2"} > 5000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "MM2 replication lag {{ $value }} msgs — check WAN link"

      # MM2 heartbeat not seen in 60 s → connector is down
      - alert: MM2HeartbeatMissing
        expr: time() - kafka_mm2_heartbeat_timestamp > 60
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "MM2 heartbeat absent — cross-region Kafka replication down"

      # Db2 HADR log gap > 10 MB → warn
      - alert: Db2HADRLogGapHigh
        expr: db2_hadr_log_gap_bytes > 10485760
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Db2 HADR log gap {{ $value | humanize }} — standby drifting"

Beyond lag, build a data-age dashboard: pick a high-write table (e.g., payment_transactions), track the timestamp of its most recent row on both primary and standby, and alert if the standby’s most recent row is more than N seconds older than the primary’s. Replication metrics at the platform layer (LSN, log gap) measure pipe capacity; data-age metrics measure actual business impact.

Decision guide

Choosing between active/passive, active/hot-standby, and active/active comes down to four questions. Answer them in order — the topology falls out:

1. What is the maximum acceptable data loss (RPO)?

   Zero RPO → synchronous replication required → WAN round-trip adds to every write latency → active/hot-standby with synchronous_commit = remote_apply. Non-zero RPO (e.g., 5 s for analytics) → async replication → all three topologies are viable.

2. Can the second region serve live reads under normal operation?

   Yes → active/hot-standby or active/active. No → active/passive. Hot-standby reads with slight lag are acceptable for most reference data; they are not acceptable for balance reads that precede an authorisation decision.

3. Do multiple regions receive independent writes to the same data?

   Yes → active/active, with mandatory write partitioning or conflict resolution. No → one of the simpler topologies. If you cannot answer “no” with confidence, you do not yet have the write ownership model required for active/active and should not proceed.

4. What is the operational cost ceiling?

   Active/active adds conflict resolution logic, write-routing infrastructure, and significantly more complex failover testing. Budget 3–4× the engineering effort of active/passive. If that budget is not approved, active/hot-standby delivers 90% of the availability improvement at a fraction of the cost.

5. Does the regulator require tested DR within a defined RTO?

   SAMA TRM specifies that banks must document and test business continuity plans. If your approved RTO is under 30 minutes, active/passive with manual failover is likely insufficient — Patroni-automated Postgres failover or a tested Db2 HADR takeover procedure is required.