How do GPUs work in Kubernetes?

Kubernetes treats a GPU as an extended resource that a vendor device plugin advertises on each node, such as nvidia.com/gpu. A Pod requests a whole number of GPUs in its resource limits, the scheduler places it on a node with that many free GPUs, and the container runtime exposes those devices to the container. The node needs the GPU driver and a container toolkit installed before any of this works.

The software stack on a GPU node

A CPU node runs the kubelet and a container runtime. A GPU node needs three more pieces:

The Kubernetes documentation puts it plainly: an administrator installs the vendor’s drivers on the nodes and runs the vendor’s device plugin, after which the cluster exposes a resource such as amd.com/gpu or nvidia.com/gpu. Device plugin support for GPUs has been stable since Kubernetes 1.26.

The CUDA libraries inside your container image and the driver on the node must match. Each CUDA release needs a minimum driver version, which NVIDIA lists in its CUDA compatibility guide. A container that needs a newer driver than the node has cannot initialize CUDA.

How a Pod requests a GPU

GPU requests follow stricter rules than CPU and memory:

• You set GPUs in limits. Kubernetes uses the limit as the request.
• If you set both requests and limits, they must be equal.
• You request whole GPUs. There is no 0.5 GPU and no overcommit.

apiVersion: v1
kind: Pod
metadata:
  name: cuda-job
spec:
  restartPolicy: OnFailure
  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: trainer
      image: nvcr.io/nvidia/pytorch:24.10-py3
      resources:
        limits:
          nvidia.com/gpu: 1

Once a GPU is allocated to a container, no other container gets it until the Pod ends. That is simple and safe, and it is also why GPUs often sit idle: a small model holding a whole H100 uses a fraction of it. Several ways to split a GPU between workloads exist, covered in MIG, time slicing and MPS.

Choosing a GPU model

nvidia.com/gpu: 1 asks for a GPU, not a specific one. When a cluster has L4, A100 and H100 nodes, you pick the model with node labels. The Kubernetes docs recommend Node Feature Discovery to label nodes automatically. NVIDIA’s GPU Feature Discovery builds on it and adds labels such as nvidia.com/gpu.product and nvidia.com/gpu.memory, which a nodeSelector or node affinity rule can match.

Keeping other workloads off GPU nodes

GPU nodes are expensive, so you rarely want a logging agent replica or a web server to fill their CPU and memory. The usual pattern is to taint GPU nodes, for example with nvidia.com/gpu=present:NoSchedule, so that only Pods with a matching toleration land there. The ExtendedResourceToleration admission controller can add that toleration automatically to any Pod that requests an extended resource such as nvidia.com/gpu.

The NVIDIA GPU Operator

Installing and upgrading drivers, toolkit and plugins by hand on every node gets old quickly. The NVIDIA GPU Operator packages them as Kubernetes workloads. It manages the driver, the NVIDIA Container Toolkit, the device plugin, GPU Feature Discovery, the DCGM Exporter for monitoring and the MIG Manager, among other components. It can also run on nodes where the driver and toolkit are already installed and manage only the remaining parts.

Monitoring GPUs

Kubernetes metrics cover CPU and memory, not GPUs. The DCGM Exporter publishes GPU metrics from NVIDIA’s Data Center GPU Manager to Prometheus: memory used, temperature, power, errors and profiling counters such as SM activity. Those metrics answer the question that matters for cost, which is how busy each GPU actually is. See how to reduce GPU costs for which metric to trust.

What comes after device plugins

The device plugin model has limits: a Pod asks for a count, it cannot describe the device it wants, and devices cannot be shared or reconfigured per request. Dynamic Resource Allocation (DRA) replaces counts with claims. A driver publishes devices and their attributes in ResourceSlice objects, administrators define DeviceClass objects, and Pods reference a ResourceClaim that selects devices by attribute. Several containers or Pods can share one claim.

The core of DRA reached general availability in Kubernetes 1.34. Vendor support is still catching up. As of November 2026, NVIDIA’s DRA driver for GPUs officially supports ComputeDomains for multi-node NVLink systems, while GPU allocation through DRA is not yet officially supported and is disabled by default. For most clusters today, the device plugin is still how GPUs are scheduled.

Scaling GPU nodes

The cluster autoscaler and node auto-provisioning add a node when a Pod is pending because no node has a free GPU, and remove it when the GPU sits empty. With GPUs, the order of events matters: the new node must boot, install or load the driver, and start the device plugin before the scheduler sees nvidia.com/gpu capacity. See how Kubernetes autoscaling works for the general mechanics.

GPUs on Cloudfleet

Cloudfleet installs the NVIDIA drivers and NVIDIA Container Toolkit on GPU nodes, both on cloud nodes created by node auto-provisioning and on self-managed servers. When a Pod requests nvidia.com/gpu and no node can run it, Cloudfleet provisions a GPU node, and labels such as cfke.io/accelerator-name pick the GPU model across providers, as the GPU workloads documentation describes.