Production readiness for a multi-tenant cluster estate
This is an illustrative scenario describing how we typically approach this kind of engagement. It does not describe a specific client, and no results are claimed.
Challenge
A typical scenario: clusters grew organically, upgrades are feared, configuration differs between clusters and tenants can affect each other.
Architecture
- Clusters defined in Terraform and bootstrapped through GitOps
- Namespace-per-tenant model with quotas, network policies and RBAC
- Kyverno policies for security and configuration baselines
- Karpenter for node provisioning and consolidation
Approach
- Assess clusters against a production readiness checklist
- Define a standard cluster baseline
- Rebuild or converge clusters to the baseline one at a time
- Establish a tested, repeatable upgrade process
Technology
EKS · Karpenter · Kyverno · Argo CD
Implementation
- Codify cluster add-ons and policies
- Introduce tenant onboarding through pull requests
- Add disruption budgets and readiness checks to workloads
- Run upgrade rehearsals in non-production first
Intended outcome
The aim is clusters that are consistent, upgradable on a schedule, and safe for multiple teams to share.
Lessons learned
- Converge gradually rather than rebuilding everything at once
- Policies in audit mode before enforcement avoid surprises
- Upgrades become routine only when practiced