Kubernetes Production Readiness Checklist

Kubernetes clusters rarely fail in interesting ways. They fail because a pod had no memory limit, a liveness probe restarted a busy service into a cascade, or nobody ever tested a restore. This is the Kubernetes production readiness checklist we run before any cluster takes real traffic — eight groups of concrete checks, each one a thing you can verify today. Work through it top to bottom; the ordering roughly matches how often each category causes a production incident.

1. Workloads and resources

Most Kubernetes outages start at the workload spec, not the control plane. These are the non-negotiables for every Deployment and StatefulSet:

Every container sets resource requests AND limits. Requests drive scheduling; without them the scheduler packs nodes blind. Remember the asymmetry: CPU is compressible — hit the limit and you get throttled. Memory is incompressible — exceed the limit and the kernel OOMKills the container. Set memory limits from observed usage plus headroom, not guesses.
Readiness, liveness, and startup probes are configured — and configured differently. Readiness gates traffic; liveness restarts the container. A liveness probe must never fire on a pod that is merely slow or busy — that turns a latency blip into a restart storm. Point liveness at a trivial in-process check, readiness at real dependency health, and use a startup probe for slow-booting apps instead of stretchinginitialDelaySeconds.
Graceful shutdown works end to end. The app handles SIGTERM, stops accepting new work, drains in-flight requests, and exits beforeterminationGracePeriodSeconds expires (default 30s — raise it if drain takes longer). Verify with a rolling restart under load: zero 5xx means it works.
PodDisruptionBudgets exist for every service that matters. Without a PDB, a node drain or cluster upgrade can evict all replicas at once. SetminAvailable or maxUnavailable so voluntary disruptions can never take you to zero.
At least two replicas, spread across failure domains. UsetopologySpreadConstraints across zones and nodes (or pod anti-affinity) so both replicas of a service are not on the same node when it dies.

resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    memory: 512Mi   # memory limit = request avoids surprise OOMKills
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: api

2. Autoscaling

HPA scales on a metric that reflects load. CPU utilization is a percentage of the request — if the request is tiny, the HPA thrashes at idle. For request-driven services, scale on requests-per-second, queue depth, or latency via custom or external metrics.
Node autoscaling is in place. Cluster Autoscaler works everywhere; Karpenter (AWS) provisions right-sized nodes faster and bin-packs better. Either way, pod autoscaling without node autoscaling just produces Pending pods.
You have actually tested both directions. Run a load test and watch scale-up latency (image pull time counts). Then watch scale-down: does the PDB allow it, do pods drain cleanly, does the app tolerate losing warm instances?
HPA and fixed replica counts do not fight. Removereplicas from manifests managed by an HPA, or GitOps will reset it on every sync.

3. Security

No privileged containers; run as non-root. SetrunAsNonRoot: true, allowPrivilegeEscalation: false, drop all capabilities and add back only what is needed, and use a read-only root filesystem where the app allows it (mount an emptyDir for scratch space).
Pod Security Standards are enforced. PodSecurityPolicy is gone (removed in 1.25); label namespaces with the built-in Pod Security Admission —restricted for app namespaces, baseline minimum everywhere.
NetworkPolicies with default-deny. Flat pod networking means any compromised pod can reach everything. Apply a default-deny ingress policy per namespace, then explicitly allow required flows. Confirm your CNI actually enforces NetworkPolicy.
RBAC is least-privilege. No workload uses cluster-admin; CI deploy credentials are namespace-scoped; audit who holds broad ClusterRoleBindings.
Secrets do not live in plaintext env vars or git. Use a secrets manager (Vault, AWS Secrets Manager, etc.) via External Secrets Operator or CSI driver, and enable encryption at rest for Secrets in etcd (managed platforms usually offer this).
Images are scanned and pinned. Vulnerability scanning in CI (Trivy or similar), signed images (cosign) with admission-time verification if your compliance bar requires it, and pulls only from registries you control.

4. Networking and ingress

TLS is automated with cert-manager. Certificates renew themselves; nobody gets paged for an expiry. Verify the renewal actually happened at least once.
Ingress timeouts are deliberate. Align proxy read/send timeouts with upstream behavior — a 60s default in front of a 5s-budget API hides failures; a short one in front of long-polling breaks it.
Rate limiting exists at the edge. Per-client limits at the ingress (or a WAF in front) so one noisy client cannot exhaust the whole service.
Cluster DNS is sized and monitored. CoreDNS is a common silent bottleneck — watch its latency and error metrics, run NodeLocal DNSCache on large clusters, and set sensible ndots for chatty workloads.
The full ingress path has a health check. Something outside the cluster probes DNS, then load balancer, then ingress, then service. Internal probes all green while the LB is down is a classic blind spot.

5. Observability

Metrics: Prometheus (or a compatible managed backend) scraping the cluster, node exporters, kube-state-metrics, and application metrics — with retention and storage sized for real cardinality.
Logs: centralized and searchable (Loki or an ELK stack), with retention policy set. Pod logs vanish with the pod; if it is not shipped, it never happened.
Traces: OpenTelemetry instrumentation on request paths that cross more than one service — without traces, "which hop is slow" is guesswork.
Dashboards per service showing the golden signals: latency, traffic, errors, saturation.
SLOs defined, alerting on symptoms not causes. Page on error-rate and latency burn against the SLO — the things users feel — not on "CPU above 80%" or "pod restarted". Cause-based alerts go to a dashboard or ticket queue.
Alert fatigue is actively managed. Every page must be actionable and urgent. If an alert fires and the runbook says "usually fine, resolve it", delete the alert. On-call that ignores pages has no alerting at all.

6. Reliability and disaster recovery

etcd is backed up. On managed control planes (EKS, GKE, AKS) the provider handles etcd, but you still need your API objects recoverable. Self-managed: scheduledetcdctl snapshot save shipped off-cluster.
Persistent volumes and cluster state are backed up with Velero (or your platform's equivalent) — namespaces, PVs via snapshots or restic/kopia, on a schedule, stored outside the cluster's blast radius.
Restore has been tested, recently. A backup you have never restored is a hope, not a plan. Restore into a scratch cluster quarterly and time it — that number is your real RTO.
Multi-AZ where the platform allows. Nodes across at least two zones, workloads spread (see section 1), and zonal PV gotchas understood — a pod cannot follow its volume to another zone unless the storage supports it.
Failure has been rehearsed. Kill a node, drain a zone, delete a random pod under load. Chaos tooling is optional; actually doing the exercise is not. A production readiness review that never broke anything on purpose reviewed nothing.

7. Delivery

Deploys are declarative with a real rollback. GitOps (Argo CD or Flux) gives you drift detection and rollback-by-revert; a plain pipeline is fine ifkubectl rollout undo or a pinned re-deploy is tested and documented.
Immutable image tags only. Deploy by digest or a unique tag per build (git SHA). Never :latest, never mutable tags — "same tag, different image" makes rollback meaningless and debugging miserable.
Progressive delivery for risky services. Canary or blue-green (Argo Rollouts, Flagger) with automatic rollback on SLO burn, so a bad release hits 5% of traffic instead of 100%.
No human applies manifests to prod by hand. If someone cankubectl apply from a laptop, your git history is fiction.

8. Cost and housekeeping

Requests are right-sized against real usage. Over-requested CPU and memory is the top Kubernetes cost leak — compare requests to actuals (VPA in recommendation mode, Goldilocks, or Prometheus queries) and reclaim the gap.
Idle capacity gets reclaimed. Scale-to-zero for dev/preview environments off-hours; consolidation enabled in Karpenter or aggressive scale-down in Cluster Autoscaler.
ResourceQuotas and LimitRanges per namespace so one team's runaway job cannot starve the cluster, and every pod gets defaults even when someone forgets section 1.
Cost is attributed. Namespace or label-based showback (OpenCost or your cloud's tooling) — unowned spend only ever grows.

How to use this checklist

Do not treat this as a gate you pass once. Run it as a working session with the team that will carry the pager: mark each item pass, fail, or not applicable, attach evidence (a manifest, a dashboard link, a restore log), and turn every fail into a ticket with an owner. Re-run the whole thing after major upgrades and once a quarter — clusters drift, and the checks that pass today quietly rot.

The two groups teams most often skip — tested restores and symptom-based alerting — are exactly the two that decide whether an incident is a blip or a headline.

If you want a second set of eyes on your cluster before launch, our Kubernetes consulting engagements run this review with your team and fix the gaps. And if section 5 was where your gaps clustered, we help teams stand up observability and SLOs that page on what users actually feel.