00:00

Disaster Recovery in System Design

Last updated:

Redundant servers and automatic failover protect a system from everyday faults. They do not help when a whole data centre loses power, a cloud region has a major outage, ransomware encrypts the database, or an engineer deletes production data by mistake. Those events need a disaster recovery plan.

This tutorial explains how disaster recovery is measured, the four classic recovery strategies, and the practices that make a plan work when it is needed. It builds on Availability and Fault Tolerance.

1. What Is Disaster Recovery?

Disaster recovery (DR) is the set of policies, tools and procedures that restore a system and its data after a major disruptive event, such as a regional outage, data corruption or a security breach. It is part of a wider business continuity plan that keeps the business running during and after the event.

High availability tries to prevent downtime from common failures. Disaster recovery accepts that some downtime and possibly some data loss can happen, and defines how much is acceptable and how to get back within those limits.

2. Disaster Recovery Architecture

Disaster recovery across two regions Users reach the system through DNS with health checks. The primary region runs the application and the primary database. The database replicates continuously to a standby database in a recovery region, and backups are copied to object storage in another region. If the primary region fails, DNS sends users to the recovery region. DNS + healthchecks PRIMARY REGION Applicationserving Primary DB RECOVERY REGION Applicationon standby Standby DBreplica Backups (other region) continuous replication
The recovery region receives a continuous copy of the data and independent backups. If the primary region fails, DNS health checks switch users to the recovery region.

Two protections work together. Replication keeps a near-current copy of the data in another region for fast recovery. Independent backups protect against problems that replication would faithfully copy, such as accidental deletion or ransomware.

3. How Disaster Recovery Works

RPO and RTO

Every DR plan starts with two targets agreed with the business. The recovery point objective (RPO) is the maximum acceptable data loss, measured in time. The recovery time objective (RTO) is the maximum acceptable downtime.

RPO and RTO on a timeline A timeline shows the last safe copy of the data, then the disaster, then the moment service is restored. The time between the last safe copy and the disaster is the data that can be lost, measured by the recovery point objective. The time between the disaster and restoration is the downtime, measured by the recovery time objective. last safe copy disaster service restored RPO: data that may be lost RTO: downtime time →
RPO looks backwards from the disaster: how much recent data may be lost. RTO looks forwards: how long the service may be down.

A recovery, step by step

  1. Detect: monitoring and health checks reveal that the primary region is failing.
  2. Declare: the on-call team decides, following the runbook, that this is a disaster and not a short blip.
  3. Fail over: the standby database is promoted to primary, and application servers in the recovery region start or scale up.
  4. Redirect: DNS or a global load balancer sends users to the recovery region.
  5. Verify and communicate: the team checks data and functions, and informs users and stakeholders.
  6. Fail back: once the primary region is healthy, data is synchronized back and traffic returns in a planned window.

Order matters during recovery. Restore the shared foundations first, such as networking, identity, secrets and databases, then the core services, and only then the less critical ones. A dependency map of services, drawn long before any disaster, turns this into a checklist instead of guesswork under pressure.

4. Disaster Recovery Strategies

The four common strategies trade cost against recovery speed. Each step to the right keeps more of the system running in the recovery region.

Backup and restore

Data is backed up regularly and copied to another region. After a disaster, infrastructure is recreated, ideally from infrastructure-as-code templates, and the data restored. It is suitable for internal or low-priority systems.

Backup and restore strategy The primary region runs the application and database. The recovery region holds only backups and snapshots; servers and the database are created from them after a disaster. Typical recovery point and time objectives are hours. PRIMARY REGION Applicationserving Database RECOVERY REGION Backupssnapshots typical RPO: hours · RTO: hours
Only backups are copied to the recovery region; everything is rebuilt after a disaster. Cheapest, but slowest to recover.

Pilot light

The core data is replicated continuously and the database is always running in the recovery region, but application servers are not. Recovery means starting and scaling the application tier.

Pilot light strategy The recovery region keeps the database replica running at all times, while application servers exist only as templates and are switched off. After a disaster the servers are started and scaled up. Typical recovery objectives are minutes for data and tens of minutes for downtime. PRIMARY REGION Applicationserving Database RECOVERY REGION DB replicarunning Applicationswitched off typical RPO: minutes · RTO: tens of minutes
The data layer is always running and replicated, while application servers stay off until needed, like a pilot light ready to ignite the full system.

Warm standby

A complete but smaller copy of the system runs continuously. It can serve some traffic immediately and is scaled up to full capacity during recovery. It also makes regular DR testing easier because the standby is always live.

Warm standby strategy The recovery region runs a scaled-down but fully working copy: a small application fleet and a database replica. After a disaster it is scaled up to handle full traffic. Typical recovery objectives are seconds to minutes for data and minutes for downtime. PRIMARY REGION Applicationserving Database RECOVERY REGION DB replicarunning Applicationsmall, running typical RPO: seconds–minutes · RTO: minutes
A smaller, fully working copy runs all the time and can take traffic immediately, then scales up to full size.

Multi-site active-active

The full system runs in two or more regions, all serving users. A region failure only reduces capacity. It needs careful handling of data conflicts and replication, and it costs the most.

Multi-site active-active strategy Both regions run the full application and database and serve users at the same time, with data replicated between them. If one region fails, the other continues serving all users. Recovery objectives are near zero. PRIMARY REGION Applicationserving Database RECOVERY REGION · ALSO SERVING Databasereplicating Applicationserving typical RPO and RTO: near zero
Both regions serve real traffic all the time, so losing one region only shifts users to the other. Fastest recovery, highest cost and complexity.

Backups that can be trusted

Apply the 3-2-1 rule: three copies of the data, stored on two different kinds of media, one of them in a separate location. Add immutable or write-once backups so that ransomware or a compromised account cannot delete them, and test restores regularly.

5. Advantages and Disadvantages

StrategyAdvantageDisadvantage
Backup and restoreLowest cost; simpleRecovery takes hours; most data loss
Pilot lightData stays current at modest costApplication tier must start under pressure
Warm standbyFast recovery; easy to testPays for a running copy all the time
Active-activeNear-zero downtime and data lossHighest cost; complex data consistency

6. Comparing the Strategies

StrategyTypical RPOTypical RTORelative cost
Backup and restoreHoursHoursLow
Pilot lightMinutesTens of minutesMedium-low
Warm standbySeconds to minutesMinutesMedium-high
Multi-site active-activeNear zeroNear zeroHigh

For how DR relates to fault tolerance and high availability, see the comparison in Fault Tolerance.

7. Real-World Use Cases

  • Banking and payments: active-active or warm standby across regions to meet strict regulatory RPO and RTO targets.
  • E-commerce: warm standby for the checkout path, and backup and restore for internal reporting tools.
  • SaaS products: pilot light to balance cost with recovery commitments in customer contracts.
  • Ransomware recovery: immutable, off-site backups restore clean data that replication would have corrupted.
  • Microservices: each service is assigned a tier; critical services get faster strategies, others use backups only.

8. Best Practices

  • Set RPO and RTO per service with the business, and choose the cheapest strategy that meets them.
  • Test regularly: run game days and full failover drills; an untested plan is only a hope.
  • Automate infrastructure with infrastructure-as-code so the recovery region can be rebuilt reliably.
  • Write and maintain runbooks with clear roles, decision criteria and communication steps.
  • Protect backups: encrypt them, store them in another account or region, and make them immutable.
  • Replicate secrets and configuration as well as data; recovery often fails on a missing certificate or key.
  • Monitor replication lag, because it directly limits the RPO you can achieve.
  • Name owners and decision makers in advance, so nobody waits for permission while the clock runs.

9. Summary

  • Disaster recovery restores systems and data after major events that high availability cannot absorb.
  • RPO limits acceptable data loss; RTO limits acceptable downtime.
  • Strategies range from backup and restore to pilot light, warm standby and active-active.
  • Faster recovery costs more, so each service should get the strategy its targets require.
  • Immutable off-site backups and regular failover tests are what make a plan dependable.

Next, learn how systems verify identity and permissions in Authentication and Authorization.