[DRAFT — AWAITING APPROVAL]
GPU Infrastructure and Scheduling
Gang scheduling, queues, quotas and topology: making expensive accelerators actually get used.
GPUs are among the most expensive resources in the cloud. Poor scheduling leaves them idle while jobs wait in queues, which is the worst of both outcomes.
Why default scheduling falls short
The default Kubernetes scheduler places pods one at a time. Distributed training needs all workers to start together, and multi-team clusters need queues, quotas and priorities that a first-come, first-served model cannot provide.
Core scheduling capabilities
- Gang scheduling so distributed jobs start all-or-nothing
- Queues with quotas and fair-share across teams
- Priorities and preemption to protect inference and urgent work
- Topology awareness for interconnect-heavy training
- Borrowing of unused quota so idle capacity is not wasted
Sharing smaller workloads
Not every workload needs a full GPU. Multi-Instance GPU (MIG) partitions supported GPUs into isolated slices; time-slicing allows sharing without hard isolation. Choose based on isolation and performance requirements.
Measure what matters
Track GPU utilization and memory per job, queue wait time, job success rate and cost per training run or per inference request. These numbers show whether capacity planning and scheduling policies are working.