RGoogle Cloud playbook · Compute and containers
Stolen workload credentials
An attacker with code execution on a VM reads an access token for the attached service account from the metadata server and uses it from another host. The token works from anywhere until it expires. Disable the service account or deny its sensitive permissions to stop stolen tokens, then rebuild the VM.
Look here
| Log | Search for |
|---|---|
| Admin Activity and Data Access audit logs | Calls by the attached service account from addresses that are not the VM's: gcloud logging read 'protoPayload.authenticationInfo.principalEmail="<sa-email>"' --project=<project> --freshness=7d --format='table(timestamp,protoPayload.methodName,protoPayload.requestMetadata.callerIp)' |
| Same entries | Unexpected callerIp values: public addresses outside your ranges, internal addresses that are not the VM's, or gce-internal-ip, which marks a Compute Engine VM outside your organization or project. Calls between Google services show private. Also look for a new callerSuppliedUserAgent, such as a CLI string the workload never uses, though the user agent is not authenticated |
| Data Access audit log for iamcredentials.googleapis.com (only if enabled) | The service account impersonating other service accounts, with it as the caller: protoPayload.serviceName="iamcredentials.googleapis.com" AND protoPayload.methodName=("GenerateAccessToken" OR "SignJwt" OR "SignBlob" OR "GenerateIdToken") |
| Security Command Center | Unusual use of the service account: Persistence: New Geography, Persistence: New API Method, Discovery: Service Account Self-Investigation |
Contain
| # | Item |
|---|---|
| 1 | Snapshot each disk: gcloud compute disks snapshot <disk> --zone=<zone> --snapshot-names=<case-id>-<disk> --labels=case=<case-id> |
| 2 | Set deletion protection: gcloud compute instances update <vm> --zone=<zone> --deletion-protection |
| 3 | Disable the service account to block stolen tokens. Every workload using it loses access: gcloud iam service-accounts disable <sa-email> |
| 4 | To block only sensitive permissions, attach an IAM deny policy that names the service account: gcloud iam policies create <policy-id> --attachment-point=cloudresourcemanager.googleapis.com/projects/<project-id> --kind=denypolicies --policy-file=<file> |
| 5 | Remove bindings the service account does not need with gcloud projects remove-iam-policy-binding |
| 6 | Capture memory with your forensic tooling, as in Evidence |
| 7 | Stop the VM after the memory capture so the attacker's foothold gets no new tokens: gcloud compute instances stop <vm> --zone=<zone> |
Eradicate and recover
| # | Item |
|---|---|
| 1 | Reverse everything the service account changed in the window, using the searches in leaked service account keys |
| 2 | Rebuild the VM from a known-good image with a dedicated least-privilege service account |
| 3 | Re-enable the service account at least 60 minutes after disabling it, or 12 hours after if extended token lifetimes are allowed: gcloud iam service-accounts enable <sa-email> |
| 4 | Remove the deny policy once the rebuilt workload is verified |
| 5 | Enable Data Access audit logs for the data services the service account can reach |
Watch out
- Service account access tokens cannot be revoked. Stopping the VM leaves issued tokens valid until expiry. Only disabling or deleting the service account, or a deny policy, stops them early.
- Changing a VM's attached service account requires the VM to be stopped. Plan the stop after memory capture.