Design a Kubernetes admission control service
Design the policy service every write to 50 Kubernetes clusters passes through: validating and mutating webhooks, an OPA Gatekeeper or Kyverno style policy engine, cosign image signature verification, a latency budget the API server can live with, a defensible fail open versus fail closed choice and a safe multi-cluster rollout.
01Understand the problem
- Trace a request through mutating and validating admission in the API server
- Design a policy engine service with a latency budget and caching
- Choose failurePolicy (fail open versus fail closed) per policy and avoid webhook deadlocks
- Verify cosign signatures by digest and roll policies out across many clusters safely
What they probe: where admission sits in the request path, what happens to the cluster when your webhook is down, how you keep p99 low when signature checks call a registry, why tags are not enough, and how a policy change reaches 50 clusters without an outage. Where candidates slip: failurePolicy: Fail on everything including kube-system, verifying a tag instead of a digest, no audit of objects that predate a policy, enforcing on day one.
When a client creates or updates an object, the API server authenticates and authorises it, then runs admission: first mutating admission (may change the object, for example add defaults or pin an image to a digest), then schema validation, then validating admission (may only accept or reject). Built-in plugins run in-process; dynamic admission webhooks send an AdmissionReview over HTTPS to your service and wait. Only then is the object written to etcd. Your service sits in the synchronous path of every matching write.
Functional requirements
- Enforce on create and update: no privileged pods, required labels, resource limits, allowed registries, signed images only.
- Mutate where it helps: inject defaults, resolve image tags to digests.
- Audit existing objects against new policies and report violations without blocking.
- Manage policies as code and roll them out across 50 clusters in stages.
Non-functional requirements
- Added latency p99 < 100 ms per admission request, including signature checks.
- Peak 500 matched requests/s in the largest cluster during deploy waves.
- A webhook outage must never stop the cluster healing itself (rescheduling pods after a node dies).
- Every deny is explainable (policy, rule, message) and logged.
This is an advanced module
Design problems: cloud & runtime security is part of the Pro pass. 4 more sections — the deep dives, the final design, follow-ups and the test — unlock with it.
Already a cyart.io or studio.cyart.io member? Your pass code is in your Studio account.