AI Infrastructure8 min read
[DRAFT — AWAITING APPROVAL]
Kubernetes for AI and ML Workloads
When Kubernetes is the right substrate for AI workloads, and what you need to add to make it work.
Kubernetes is a strong foundation for many AI and ML workloads, but out of the box it is designed for long-running services, not GPU-heavy batch jobs and latency-sensitive inference.
When Kubernetes is a good fit
- You already run Kubernetes and want one operational model for apps and ML
- Multiple teams share a pool of GPUs and need quotas and fairness
- You need portable infrastructure across clouds or on-premises
What you need to add
- GPU device plugins or operators (for example the NVIDIA GPU Operator) and node pools per accelerator type
- Batch scheduling and queueing with tools such as Kueue or Volcano, including gang scheduling for distributed training
- Model serving layers such as KServe, Triton or vLLM-based deployments with autoscaling
- Storage designed for large datasets and checkpoints
- GPU-aware observability: utilization, memory, queue wait time
Separate training and inference
Training jobs tolerate queues and preemption; inference does not. Use separate node pools, priorities and scaling policies so a large training job cannot starve user-facing endpoints.
When to consider alternatives
If a team has no Kubernetes experience and a small, steady workload, a managed ML platform may deliver value faster. The right answer depends on scale, team skills and how much control you need.