PIVTR-D
RGoogle Cloud playbook · Provider and accidental

Regional outage or service disruption

A Google Cloud zone, region or service fails, and your workloads slow down or stop. On June 12, 2025, a policy with blank fields, replicated globally, crashed Service Control in every region, and external API requests returned 503 errors for up to about 3 hours. Fail over with the controls that still work, and record every emergency change so you can revert it.

Look here

LogSearch for
Personalized Service HealthIncidents on your projects. Events are also written to Cloud Logging for alerts: gcloud service-health events list --project=<project> --location=global
Google Cloud Service HealthBroad, multi-region incidents, at https://status.cloud.google.com with no sign-in, and in its RSS feed. Google posts narrower incidents in Personalized Service Health
Admin Activity audit log, emergency changesGrants and firewall changes made during the outage: gcloud logging read 'logName:"cloudaudit.googleapis.com%2Factivity" AND (protoPayload.methodName:"SetIamPolicy" OR protoPayload.methodName:"firewalls.insert" OR protoPayload.methodName:"firewalls.patch")' --project=<project> --freshness=7d
Log deliveryLate entries, where receiveTimestamp trails timestamp for security sources, and sources with no entries in the window
Cloud SQL replicasReplication lag of each cross-region replica before promotion. The lag is the window of writes you may lose

Contain

#Item
1Record the Service Health event ID, the impact start and your first failed check, and open an emergency change record from the templates for everything that follows
2Point DNS at the standby region. Records with a FAILOVER routing policy and --enable-health-checking switch on their own: gcloud dns record-sets update <name> --zone=<zone> --type=A --ttl=<ttl> --rrdatas=<standby-ip>
3Keep healthy backends in a second region behind the global external Application Load Balancer, which sends requests only to backends that pass health checks
4On Cloud SQL Enterprise Plus with a designated disaster recovery (DR) replica, switch to it while the primary and the replica are both online. Switchover waits for the replica to catch up, so it loses no data: gcloud sql instances switchover <replica>
5If the primary region is down, promote the cross-region replica destructive, irreversible: gcloud sql instances promote-replica <replica>
6Grant emergency access only with an expiry, and log each grant in the change record: gcloud projects add-iam-policy-binding <project> --member='user:<email>' --role=<role> --condition='expression=request.time < timestamp("<utc>"),title=<case>-expiry'

Eradicate and recover

#Item
1Revert every emergency grant, firewall rule and DNS change from the change record, and confirm each with the emergency-changes query
2Fail back after Service Health shows the incident resolved. A promoted replica is standalone, so create a new replica before relying on failover again
3Write the log gaps, with source and time range, into the case record
4Warn staff that phishing uses outage news, fake status pages and credit offers, and give them the official status address
5Claim service level agreement (SLA) credits through Google technical support within the window in each service's SLA, 60 days for Compute Engine, with logs showing downtime periods and times
6Follow the recover stage, and exercise the failover with the regional outage tabletop
Watch out
  • Identity, IAM policies, DNS and global load balancers are global services, and a fault in one can reach every region, as in June 2025. When one region fails, control planes in other regions keep working, so manage the standby from there.
  • The status page can lag. In June 2025, existing infrastructure as a service (IaaS) resources kept running and us-central1 recovered last, and Google posted its first report about an hour in because Cloud Service Health was down too.
  • IAM allow policy changes typically take 2 minutes and can take 7 minutes or longer. Group membership changes can take hours, so grant emergency roles to users directly.
  • Cloud SQL replication is asynchronous. A promotion during an outage can lose recent transactions and cannot be reversed.

Google Cloud Service Health incident report for June 12, 2025, Multiple GCP products are experiencing Service issues. Google Cloud documentation: Personalized Service Health overview and incident communication; Geography and regions; external Application Load Balancer overview; Cloud SQL cross-region replicas and disaster recovery; IAM access change propagation and temporary access; Compute Engine SLA; gcloud reference for service-health events list, dns record-sets update, sql instances switchover and promote-replica, and projects add-iam-policy-binding. Checked September 2026. Re-checked October 2026: the June 12, 2025 incident report (blank policy fields, Service Control crash loop in every region, 503 errors, about 3 hours); Compute Engine SLA (notify technical support within 60 days, with log files).

Adapts material from Dynamic Incident Response: A Framework for Security Teams by Joshua Wright, © 2026 The Escal Institute of Advanced Technologies, Inc. d/b/a SANS Institute, licensed under CC BY 4.0. Figures reproduced under that license. Changes were made. Not published by, endorsed by, or affiliated with the SANS Institute.