Kubernetes audit log investigations
Investigate a cluster from the API server audit log: audit policy levels and stages, reading events by user, verb, objectRef and responseStatus, spotting exec into pods, privileged pod creation, service account token abuse and secrets access, then containing without losing evidence.
01Audit policy: what the API server records
- Write an audit policy and choose between None, Metadata, Request and RequestResponse
- Read an audit event by user, verb, objectRef and responseStatus
- Spot exec into pods, privileged pods, service account token abuse and secrets reads
- Contain a cluster incident in an order that keeps evidence
Every action in Kubernetes, from kubectl get pods to a controller updating a Deployment, is a request to the kube-apiserver. The API server can write an audit event for each request, controlled by an audit policy: an ordered list of rules where the first match decides how much is recorded. Self-managed clusters enable it with --audit-policy-file and --audit-log-path; managed clusters expose it as a logging option.
| Level | Records | Use for | Cost |
|---|---|---|---|
| None | Nothing | Health checks, high-volume watches by system components | Blind spot by design |
| Metadata | User, verb, resource, namespace, name, source IP, response code | Secrets and tokens (never log their bodies), default for everything else | Low |
| Request | Metadata + request body | Pod and workload creation, RBAC changes: shows the spec that was submitted | Medium |
| RequestResponse | Metadata + request and response bodies | Narrow, high-value writes | High; can leak sensitive data into logs |
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"] # keep ResponseStarted/ResponseComplete only
rules:
# 1. drop noise first: rules are evaluated top down, first match wins
- level: None
nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
# 2. secrets and tokens: who and when, never the values
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps", "serviceaccounts/token"]
# 3. interactive access and workload creation: keep the request body
- level: Request
resources:
- group: ""
resources: ["pods", "pods/exec", "pods/attach", "pods/portforward", "pods/ephemeralcontainers"]
- group: "apps"
resources: ["deployments", "daemonsets", "statefulsets"]
verbs: ["create", "update", "patch", "get"]
# 4. RBAC changes: full detail
- level: RequestResponse
resources:
- group: "rbac.authorization.k8s.io"
resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
verbs: ["create", "update", "patch", "delete"]
# 5. everything else
- level: MetadataRule 3 includes get because newer kubectl versions open exec over a WebSocket, which arrives as an HTTP GET; older clients use a POST, recorded as create. Order matters: if the secrets rule came after a catch-all Request rule for the core group, secret values would land in your log pipeline.
| Platform | How to get audit logs | Watch out for |
|---|---|---|
| Amazon EKS | Enable the audit control-plane log type; lands in CloudWatch Logs /aws/eks/<cluster>/cluster | Off by default; the policy is AWS-managed |
| Google GKE | Cloud Audit Logs: Admin Activity always on; Data Access (get, list, watch) must be enabled | Without Data Access you will not see secret reads |
| Azure AKS | Diagnostic settings: kube-audit or kube-audit-admin | kube-audit-admin excludes get and list, so it misses secret reads |
| Self-managed | --audit-policy-file, --audit-log-path, rotation flags, or a webhook backend | Local files on control-plane nodes rotate and can be tampered with |
This is an advanced module
Investigations beyond Windows is part of the Pro pass. 3 more sections — the deep dives, the final design, follow-ups and the test — unlock with it.
Already a cyart.io or studio.cyart.io member? Your pass code is in your Studio account.