The primary site suffers a genuine outage — power failure, disaster, or an incident severe enough to cross the DR threshold. Monitoring detects primary is unreachable or degraded beyond a defined limit, and a human incident commander formally declares a disaster and authorizes failover. This decision is deliberate, not automatic reflex — failing over (and eventually back) is itself a real operational risk, not a free action.
TriggerEvery DR plan is built around two numbers, defined long before any incident: RTO (Recovery Time Objective — how long can the business tolerate being down) and RPO (Recovery Point Objective — how much data loss, measured in time, is acceptable). These two numbers dictate how much of what follows can be pre-automated versus done manually under pressure, and how "clean" this failover can realistically be.
RTO / RPOThe secondary site's database — replicating from primary, synchronously or asynchronously — is promoted from standby/read-replica to the new writable primary. If replication was asynchronous and lagging at the moment of failure, any transactions not yet replicated are gone the instant this promotion happens. This exact loss is what RPO quantifies in advance — it isn't an accident, it's a planned, bounded trade-off.
CheckpointIdeally, automated tooling verifies replication lag and data integrity immediately before or after promotion. Under genuine crisis time pressure, this step can be compressed or skipped — a real, acknowledged operational risk that good DR runbooks explicitly plan for rather than pretend won't happen.
VerifyApplication servers at the secondary site — possibly sitting in a scaled-down standby state — are started or scaled up to production capacity, with configuration now pointing at the local (secondary) database connection instead of primary's.
App StartupGlobal traffic management — DNS-based failover or a GSLB appliance — updates so the public hostname now resolves to the secondary site's IP. This is not instantaneous for every client: DNS caching means some users see the switch immediately while others, depending on their resolver's cached TTL, don't see it until that TTL expires.
DNS CutoverA low TTL configured on these records well ahead of time — as part of DR planning, not during the event — is what keeps this cutover fast for most users. If TTL was left high, failover could effectively take hours to reach some users regardless of how quickly every other system recovered, and there's nothing to be done about it mid-incident.
TTLAs DNS propagates, new client connections land on the secondary site. Existing sessions to the now-dead primary are simply lost — the same as any hard network failure — and must reconnect fresh, this time resolving to the new address.
ReconnectMessage queues, caching layers, other internal microservices, and third-party integrations with IP allowlists each need their own reconnection or reconfiguration — some may need manual intervention if they were hardcoded to primary's IP rather than using the same DNS-based approach the public-facing service uses.
DependenciesThe DR team runs a defined set of smoke tests and health checks against the secondary site's actual production-serving stack — real transactions, real data reads/writes — before formally declaring the failover complete. "The servers responded to a ping" is not the same claim as "the service is genuinely healthy."
ValidateEnd users regain service, though sometimes at somewhat reduced capacity if the secondary site was deliberately sized for standby rather than full production load — a common, intentional cost trade-off made at DR-design time, not a failure of the failover itself.
RestoredOnce primary is repaired and independently verified healthy, a second, carefully planned cutover — usually scheduled during a low-traffic maintenance window, not run as an emergency — reverses the process: replicate data back to primary, re-promote it, re-point DNS. Failback is intentionally not rushed the way the initial failover often had to be, precisely because it isn't an active emergency.
Failback| Term | Meaning |
|---|---|
| RTO | Recovery Time Objective — the target maximum duration of an outage |
| RPO | Recovery Point Objective — the target maximum acceptable data loss, measured in time |
| Synchronous Replication | Writes are confirmed at both sites before being acknowledged — zero data loss, at a latency cost |
| Asynchronous Replication | Writes are confirmed at primary and replicated to secondary afterward — some replication lag/risk |
| GSLB | Global Server Load Balancing — DNS-aware traffic steering across geographically separate sites |
| TTL | How long a DNS answer may be cached — directly controls how fast a DNS-based failover actually propagates |
| Warm / Hot Standby | A DR site kept partially or fully running/scaled, versus a cold site requiring a full build-out to activate |
| Smoke Test | A quick set of real functional checks confirming a system is genuinely healthy, not just reachable |
| Failback | The planned, deliberate reverse cutover back to the original primary site once it's repaired |
| DR Runbook | The documented, step-by-step procedure a team follows to execute a failover consistently under pressure |