Skip to content
EncryptTechBD
Home Industries Our Work Blog Pricing About Contact Discuss Your Project
বাংলা (BN)
Cloud, DevOps & FinOps 6 min read

Cloud disaster recovery testing: a step-by-step guide

How to run a cloud recovery test that proves your RTO and RPO, from scenario selection and failover to business acceptance and a report that drives fixes.

E
EncryptTechBD · 0 comments
Long row of computer servers in a rack
Photo: CSIRO / Wikimedia Commons (BY)

A disaster recovery plan is a hypothesis. It says that if a region fails, a database corrupts or a ransomware event encrypts production, the business can be running again within an agreed time and with an agreed amount of data loss. Until someone tests it, that claim is unproven, and the gap between the plan and reality is usually discovered at the worst possible moment.

Testing is not the same as checking that backups ran. It is the deliberate rehearsal of recovery, measured against business outcomes. This guide gives a sequence that works for a small cloud estate, assuming the person running it has other responsibilities that day.

Prerequisites

  • An agreed recovery time objective (RTO) and recovery point objective (RPO) for each critical service, signed by the business rather than only by IT.
  • A service inventory with dependencies: domain name, DNS, TLS certificates, identity provider, email, payment gateway, courier API, object storage.
  • Written runbooks that someone other than the author has followed at least once.
  • Working access to the recovery account, including break-glass credentials stored somewhere that does not depend on the primary environment.
  • Budget approval for the test, because a full failover can double infrastructure cost for a few hours.
  • A communication plan with named spokespeople and a customer-facing holding statement.

Choose the test type first

  • Tabletop: the team talks through a scenario using the runbook, with no systems changed. Cheap, fast, and good for finding missing contacts and unclear ownership.
  • Component restore: restore one database or one service into an isolated environment. Proves the backup and the procedure, not the whole failover.
  • Partial failover: move a defined slice of the estate, such as the order service and its database, to the recovery region.
  • Full failover: move every critical service and run the business from the recovery environment for a defined period.

Start with a tabletop exercise before scheduling a live failover.

The step-by-step test

  1. Pick one scenario, not a generic drill. Region outage, database corruption, accidental deletion, ransomware, DNS or certificate failure and provider account suspension produce different recovery paths. Testing all of them at once tests none of them properly. Output: a scenario statement and the services in scope.
  2. Write the success criteria in business language. "A customer can place an order, pay, and receive an order confirmation" is testable. "The application is up" is not. Output: a short acceptance script for each critical journey.
  3. Book the window and notify the people who need to know. Support, finance and the payment provider should hear it from you, not from a status page. Output: a notice with start time, duration, impact and contact.
  4. Freeze changes and capture the baseline. Record application versions, database size, configuration and certificate expiry dates before touching anything. A test against an unknown build proves little. Output: a baseline sheet with timestamps.
  5. Verify the backups before relying on them. Check the last successful job, the restore point timestamp and the retention copy in the recovery account. A job status is not a restore. Output: the chosen restore point and its age.
  6. Restore data into an isolated environment first. Measure the time from start to a usable database. Do not point production traffic at it yet. Output: restore duration and any errors.
  7. Bring up compute, network and identity. Confirm the virtual network, security groups, load balancer, secrets and service accounts exist in the recovery region. Missing identity is a frequent failover blocker. Output: a service-by-service start log.
  8. Switch traffic deliberately. Lower the DNS time-to-live well before the test, then make the change and record propagation time. If the domain or certificate is managed by a third party, confirm who can change it at 2 a.m. Output: the traffic switch time and the observed propagation delay.
  9. Validate the application and its integrations. Log in, load a product, create a test order, send a test payment, check that the confirmation email arrives and that the courier API accepts a booking. Output: a pass or fail for each integration.
  10. Run the business acceptance script. Ask an operations person, not an engineer, to complete the journey. They will notice the missing invoice template or the wrong currency symbol. Output: a signed acceptance note.
  11. Test communication and escalation. Publish a status update, answer a simulated customer question using the holding statement, and confirm the escalation call reaches a human. Output: a timeline of who was contacted and when.
  12. Measure against the objectives. Record actual RTO and actual RPO, including the data loss window. State the difference plainly, even when it is uncomfortable. Output: a comparison table with target and actual.
  13. Roll back or fail forward, then clean up. Confirm the return path works, remove test resources, and check that no test order or payment remains in production. Output: a cleanup checklist signed by the owner.
  14. Write the report and assign the fixes. Every finding needs an owner and a date. A report without owners is a diary entry. Output: a remediation register reviewed at the next operations meeting.

Evidence checklist

  • Scenario statement and scope.
  • Baseline versions, database size and certificate dates.
  • Restore point used and its timestamp.
  • Timed restore and failover log.
  • Integration test results with screenshots or logs.
  • Business acceptance sign-off.
  • Actual RTO and RPO compared with targets.
  • Remediation register with owners and dates.

Common mistakes

  • Testing the backup system instead of the recovery process.
  • Forgetting DNS, TLS certificates, email and the identity provider.
  • Running the test with the person who wrote the runbook, then discovering nobody else can follow it.
  • Leaving credentials in the primary environment, so a real outage locks the team out of the recovery account.
  • Declaring success because infrastructure started, without a business transaction completing.
  • Skipping the rollback test, which leaves the business stranded in the recovery environment.

Metrics

  • Actual RTO against target, by critical service.
  • Actual RPO and the measured data loss window.
  • Percentage of critical services covered by a test in the last twelve months.
  • Number of findings closed within the agreed date.
  • Time to switch traffic, including DNS propagation.

Local conditions

Many Bangladeshi businesses run on a single cloud region chosen for latency, with a local provider for DNS or email that is outside the main cloud account. That split is fine until the recovery plan assumes a single vendor. Map every dependency, including the domain registrar, the SMS gateway and the payment provider. Keep a printed contact list, because the recovery plan that lives only in the affected system is not a recovery plan.

Where this fits in our work

We design and test recovery procedures under Managed Technology & Cybersecurity, including backup verification and failover runbooks. Scope and commercial terms are on the pricing page. For the budget side of the same decision, read what disaster recovery testing costs.

Sources

Services that deliver this

Want this implemented properly?

Pricing →

Every engagement starts with a written scope. If a requirement is outside it, we quote it separately before starting.

Responses (0)

No comments yet. Add the first one below.

Leave a response

Your email address will not be published. Required fields are marked *.

Keep reading

More field notes