AI InfrastructureEXAMPLE SCENARIO
Shared GPU platform for training and inference
This is an illustrative scenario describing how we typically approach this kind of engagement. It does not describe a specific client, and no results are claimed.
Challenge
A typical starting point: several ML teams each run their own GPU instances. Some sit idle while others wait for capacity, inference shares nodes with training, and nobody can see who is using what.
Architecture
- Kubernetes cluster with separate GPU node pools for training and inference
- Kueue for queueing, quotas and fair-share between teams
- KServe for model serving with autoscaling
- GPU metrics exported to the existing observability stack
Approach
- Inventory current GPU usage, workloads and team needs
- Design node pools, queues and priority classes
- Pilot with one training team and one inference service
- Roll out to the remaining teams with documented golden paths
Technology
Kubernetes · Kueue · KServe · NVIDIA GPU Operator
Implementation
- Install and harden the GPU Operator and node pools
- Define queues, quotas and borrowing rules per team
- Standardize serving templates with autoscaling and health checks
- Add dashboards for utilization, queue wait time and cost per team
Intended outcome
In an engagement like this, the goal is one governed GPU pool, so inference is protected from training jobs and teams can see their own usage and cost.
Lessons learned
- Agree on quota and priority policy with team leads before enforcing it
- Keep training and inference isolated from day one
- Treat GPU utilization as a shared metric, reviewed regularly