Data lost through provider or operator error
Resources or data disappear through a mistake: an AWS error, an infrastructure-as-code apply, or a cleanup script. If the deleting identity is not your own automation or operator, or its credentials are in doubt, open Deletion and ransomware. Otherwise stop the automation that deleted the data before you restore anything.
In May 2024 a Google Cloud misconfiguration deleted UniSuper's private cloud subscription in both of its geographies, and UniSuper restored from backups kept with another provider.
Look here
| Log | Search for |
|---|---|
| CloudTrail Lake | Deletes and terminations since the start time. The role and userAgent name the pipeline or script: SELECT eventTime, eventSource, eventName, userIdentity.arn, userAgent, sourceIPAddress FROM <event-data-store-id> WHERE (eventName LIKE 'Delete%' OR eventName LIKE 'Terminate%') AND eventTime > '<utc>' ORDER BY eventTime |
| CloudTrail, service events | Actions a service took on its own, with eventType, AwsServiceEvent. Changes AWS staff make inside a service may not appear in your trail |
| CloudFormation | Resources a stack deleted, marked DELETE_COMPLETE. A deleted stack needs its stack ID: aws cloudformation describe-stack-events --stack-name <stack-id> |
| Organizations | Closed or removed accounts: CloseAccount or RemoveAccountFromOrganization in the management account's trail |
| AWS Health | Issues AWS has posted for the service. The API needs Business Support+ or higher: aws health describe-events --region us-east-1 --filter "services=<service>,eventTypeCategories=issue" |
| AWS Backup | Recovery points left in a vault: aws backup list-recovery-points-by-backup-vault --backup-vault-name <vault> |
| Recycle Bin | Deleted snapshots that a retention rule kept: aws ec2 list-snapshots-in-recycle-bin |
| RDS retained backups | Automated backups kept after an instance was deleted: aws rds describe-db-instance-automated-backups |
| DynamoDB system backups | Backups made when a table with point-in-time recovery was deleted: aws dynamodb list-backups --backup-type SYSTEM |
| Secrets Manager | Secrets scheduled for deletion that can still be restored: aws secretsmanager list-secrets --include-planned-deletion |
Contain
| # | Item |
|---|---|
| 1 | Stop the pipeline before any restore: aws codepipeline disable-stage-transition --pipeline-name <pipeline> --stage-name <stage> --transition-type Inbound --reason "<case>" |
| 2 | Disable the scheduled cleanup rule: aws events disable-rule --name <rule> |
| 3 | Throttle the cleanup function to zero: aws lambda put-function-concurrency --function-name <function> --reserved-concurrent-executions 0 |
| 4 | Export CloudTrail for the window and save the plan or script output that ran. See Evidence |
| 5 | Turn on termination protection for each surviving stack: aws cloudformation update-termination-protection --stack-name <stack> --enable-termination-protection |
| 6 | Turn on deletion protection for each surviving database: aws rds modify-db-instance --db-instance-identifier <db> --deletion-protection --apply-immediately |
| 7 | Cancel any KMS key deletion that the same run scheduled: aws kms cancel-key-deletion --key-id <key> |
| 8 | Open an AWS Support case at urgent (production system down) or critical (business-critical system down), with what happened, when, and the resource IDs in the body. The Support API needs Business Support+, Enterprise Support or Unified Operations: aws support create-case --subject "<case>" --communication-body "<summary>" --severity-code urgent --issue-type technical |
Eradicate and recover
| # | Item |
|---|---|
| 1 | Restore in dependency order: network and keys, then data, then compute |
| 2 | Restore a deleted RDS instance from retained automated backups: aws rds restore-db-instance-to-point-in-time --source-dbi-resource-id <dbi> --target-db-instance-identifier <new> --use-latest-restorable-time |
| 3 | Restore a deleted DynamoDB table from its system backup <table>$DeletedTableBackup, then set auto scaling, IAM policies, alarms, tags, streams and time to live (TTL) again: aws dynamodb restore-table-from-backup --target-table-name <table> --backup-arn <arn> |
| 4 | Restore snapshots retained by Recycle Bin: aws ec2 restore-snapshot-from-recycle-bin --snapshot-id <snap> |
| 5 | Restore AMIs retained by Recycle Bin: aws ec2 restore-image-from-recycle-bin --image-id <ami> |
| 6 | Restore secrets scheduled for deletion: aws secretsmanager restore-secret --secret-id <secret> |
| 7 | Restore from AWS Backup. When the source account is gone, use a logically air-gapped vault shared through AWS RAM to a recovery account |
| 8 | Fix the cause with a manual approval before any destroy, deletion protection, and DeletionPolicy: Retain on data stores, then reopen the pipeline: aws codepipeline enable-stage-transition --pipeline-name <pipeline> --stage-name <stage> --transition-type Inbound |
- Recovery windows differ by service. Secrets Manager waits 7 to 30 days (30 by default), and a KMS key 7 to 30 days. Recycle Bin keeps snapshots and AMIs 1 to 365 days and volumes 1 to 7 days, per rule. A DynamoDB system backup exists only if point-in-time recovery was on, and lasts 35 days.
- RDS retained automated backups last only as long as the instance's backup retention period at deletion.
- A closed account can be reopened through AWS Support within 90 days. After that AWS deletes its content and the account ID is never reused.
AWS documentation: CloudTrail event history and service events; CloudFormation describe-stack-events and termination protection; AWS Health API; AWS Backup logically air-gapped vaults and CloudTrail logging; EBS Recycle Bin concepts; RDS retaining automated backups, deleting a DB instance and restore-db-instance-to-point-in-time; DynamoDB point-in-time recovery, list-backups and restore-table-from-backup; Secrets Manager deleting and restoring secrets; closing an account; Support case severity and create-case; AWS Support API reference; CLI codepipeline disable-stage-transition, events disable-rule and lambda put-function-concurrency. UniSuper and Google Cloud joint statement, May 2024. AWS Health API and Support API plan requirements and Recycle Bin retention periods checked October 2026; other topics checked September 2026.