PIVTR-D
RAWS playbook · Provider and accidental

Regional outage or service disruption

An AWS Region or service fails, and your workload runs degraded or fails over to another Region. Use break-glass access prepared in advance, and log every emergency change and temporary access grant so you can revert it later.

In the us-east-1 event of October 19 and 20, 2025, an empty DNS record for the regional DynamoDB endpoint broke DynamoDB. Error rates stayed high from 11:48 PM PDT on October 19 to 2:40 AM on October 20. Console sign-in for IAM users, IAM Identity Center sign-in in us-east-1, and AWS Support cases also failed, and STS errors and latency lasted until 9:59 AM.

Look here

LogSearch for
AWS Health Dashboard, service healthPublic events for every Region, with no sign-in needed: https://health.aws.amazon.com/health/status
AWS Health APIOpen issues that affect your account. The API needs Business Support+, Enterprise Support or Unified Operations. If us-east-1 fails, --region us-east-2 returns eventually consistent data: aws health describe-events --region us-east-1 --filter "eventTypeCategories=issue,eventStatusCodes=open"
Single-Region control planesControl planes that run in one Region and can fail with it. IAM, Organizations, Account Management, Route 53 public and private DNS, CloudFront, WAF and ACM for CloudFront, and Shield Advanced run their control planes in us-east-1. Amazon Application Recovery Controller (ARC), Network Manager and Global Accelerator run theirs in us-west-2. Their data planes keep working. Record changes in a Route 53 public hosted zone move to us-west-2 within about 60 minutes only when accelerated recovery was on beforehand
Route 53 health checksStatus from the Route 53 health checkers. Calculated health checks show only in the Route 53 or CloudWatch console: aws route53 get-health-check-status --health-check-id <id>
CloudWatchError rate and latency on your own load balancers, APIs and databases in each Region
CloudTrail deliveryWhen the trail last delivered logs, from LatestDeliveryTime. Delivery takes about 5 minutes on average and is not guaranteed: aws cloudtrail get-trail-status --name <trail>

Contain

#Item
1Open an incident record that logs every emergency change and temporary access grant, with who made it and who will revert it
2Sign in with break-glass access prepared in advance. Creating or updating IAM resources during the outage depends on the us-east-1 control plane
3Let Route 53 failover records with health checks move traffic on their own. Editing records during the outage depends on the us-east-1 control plane unless the zone has accelerated recovery
4Shift traffic with ARC routing controls. Try another cluster endpoint if one fails: aws route53-recovery-cluster update-routing-control-state --routing-control-arn <arn> --routing-control-state Off --endpoint-url https://route53-recovery-cluster.<region>.amazonaws.com --region <region>
5Warn staff about outage-themed phishing and fake status pages, and point them to the Health Dashboard address
6Promote the Aurora global database secondary when the primary Region is down destructive, loses unreplicated writes: aws rds failover-global-cluster --global-cluster-identifier <global> --target-db-cluster-identifier <secondary-arn> --allow-data-loss

Eradicate and recover

#Item
1Fail back after AWS Health closes the event and your own checks pass. For a planned Aurora fail back, run the same command with --switchover in place of --allow-data-loss
2Revert each emergency change and temporary access grant in the record, and search CloudTrail for changes made by break-glass identities in the window
3Reconcile data written in both Regions during the failover
4Record the log delivery gaps for each source, so later investigations know which hours are incomplete
5Claim service level agreement (SLA) credits through an AWS Support case with the dates, times, resource IDs, Regions and request logs. The RDS SLA requires the claim by the end of the second billing cycle after the incident, and each service SLA sets its own terms
6Turn on Route 53 accelerated recovery for each public hosted zone, so record changes keep working if us-east-1 fails. Setup can take several hours: aws route53 update-hosted-zone-features --hosted-zone-id <zone> --enable-accelerated-recovery
7Move other recovery steps off control planes, then exercise the plan with the regional outage tabletop. See also Recover
Watch out
  • AWS Support could not create, view or update cases from 11:48 PM to 2:40 AM PDT in the October 2025 event. Keep your account team contacts outside AWS.
  • Customers with IAM Identity Center in us-east-1 could not sign in during the October 2025 event. Keep a break-glass path that does not depend on the Identity Center Region.
  • Accelerated recovery covers public hosted zones only and cannot be turned on after a failover starts. During a failover you can change records but cannot create or delete hosted zones or change DNSSEC signing.
  • --allow-data-loss starts a global failover. Without it, the call defaults to a switchover.

AWS documentation: AWS Health Dashboard and AWS Health API, CLI health describe-events; Fault Isolation Boundaries whitepaper, global services; Route 53 get-health-check-status and accelerated recovery for managing public DNS records; ARC update-routing-control-state and routing control states; RDS failover-global-cluster; CloudTrail how it works; Amazon RDS SLA. AWS summary of the DynamoDB disruption in us-east-1, October 2025. AWS Health API, Route 53 accelerated recovery and the AWS event summary checked October 2026; other topics checked September 2026.

Adapts material from Dynamic Incident Response: A Framework for Security Teams by Joshua Wright, © 2026 The Escal Institute of Advanced Technologies, Inc. d/b/a SANS Institute, licensed under CC BY 4.0. Figures reproduced under that license. Changes were made. Not published by, endorsed by, or affiliated with the SANS Institute.