GPU Orchestration on Kubernetes in 2026: DRA, MIG, and Time-Slicing for LLM Serving

In production generative AI architectures, GPUs represent both the highest infrastructure cost and the primary scalability bottleneck. Allocating hardware accelerators (NVIDIA H100, L40S, A100) to LLM ServingRuntimes (vLLM, KServe, TGI) or auxiliary microservices (embedding models, rerankers, RAG pipelines) requires orchestration granularity far beyond the legacy model of static, whole-node reservations.

Scaling on Kubernetes and Red Hat OpenShift AI relies on three complementary allocation mechanisms: Time-Slicing, Multi-Instance GPU (MIG), and the adoption of Dynamic Resource Allocation (DRA) introduced in modern Kubernetes releases.

---

1. The Limitations of Legacy NVIDIA Device Plugins (1 Pod = 1 GPU)

Historically, the k8s-device-plugin exposed GPUs as discrete integer resources (nvidia.com/gpu: 1). While this approach suits heavy distributed training workloads (PyTorch, Ray Train), it proves inefficient for inference:

• Severe VRAM Underutilization: Running a lightweight embedding model (0.5 GB VRAM) or a reranker on an 80 GB GPU locks the entire accelerator away from other workloads.
• Lack of Hardware Isolation: Unconstrained application-level sharing leads to memory bus contention and unpredictable spikes in Time to First Token (TTFT).
• Topology Blindness: Inability to express NVLink affinity or inter-GPU bandwidth requirements without cumbersome manual scheduling setups.

---

2. Comparative Matrix: GPU Allocation Strategies

Strategy	Memory & Compute Isolation	Recommended Use Case	Key Advantages	Trade-offs
Time-Slicing	None (Temporal Sharing)	Dev, testing, low-traffic auxiliary embeddings	Zero hardware configuration, high oversubscription	Risk of cross-tenant OOMs, no latency guarantees
NVIDIA MIG	Strict Hardware Slicing (VRAM + SMs)	Multi-tenant inference, collocated ServingRuntimes	Deterministic QoS, full failure domain isolation	Rigid profiles (e.g., 1g.10gb, 3g.40gb), high-end GPUs only (A100/H100)
DRA (Dynamic Resource Allocation)	Dynamic via CDI / Structured Parameters	Heterogeneous clusters, advanced LLMOps pipelines	Declarative hardware matching, decoupled scheduler	Requires modern Kubernetes (1.30+) and compatible driver stack

---

3. The Power of Dynamic Resource Allocation (DRA)

Dynamic Resource Allocation (DRA) and the Container Device Interface (CDI) fundamentally modernize accelerator management on Kubernetes:

• Structured Parameters: Instead of a simple integer request, workloads define ResourceClaims based on required VRAM capacity, NVLink topology, or compute capability.
• On-Demand Allocation: The Kubernetes scheduler matches claims dynamically during scheduling passes without requiring cluster operators to statically carve node profiles.
• Optimized Co-scheduling: Native coordination between GPU allocations, NUMA affinity, and high-throughput network fabric for vLLM Tensor Parallelism.

---

4. Implementation on Red Hat OpenShift AI (RHOAI)

Within Red Hat OpenShift AI, these capabilities are delivered as an enterprise-grade platform:

• NVIDIA GPU Operator on OpenShift: Automated deployment of drivers, CDI, MIG profiles, and DCGM telemetry export via Prometheus.
• KNative & KServe Scale-to-Zero: Idling inactive ServingRuntimes to immediately release GPU slices for high-priority training or batch jobs.
• Elastic Scheduling with Ray / KubeRay: Coordinated distributed compute across heterogeneous infrastructure.

---

5. Advance Your AI Infrastructure Engineering with Aperta Scientia

To design, secure, and operate scalable AI infrastructure:

• AS300 Curriculum — AI Platform Engineer (399h): The intensive live program to master OpenShift AI, GPU partitioning, vLLM, KServe, and prepare for the official Red Hat Certified Specialist in AI/ML (EX267) certification.
• AS200 Curriculum — DevOps & Platform Engineering (399h): The fundamental platform engineering foundation covering hardened OpenShift clusters, Ansible automation, and GitOps.

👉 Explore the AS300 AI Platform Engineer Curriculum | Explore the AS200 DevOps Curriculum | Contact our Engineering Advisors

---

6. Technical References & Documentation

• Kubernetes Dynamic Resource Allocation (DRA): Official Kubernetes Documentation
• NVIDIA Multi-Instance GPU (MIG) Architecture: NVIDIA MIG User Guide
• Red Hat OpenShift AI Hardware Acceleration: OpenShift AI Acceleration Guide