How do you reduce GPU costs?

You reduce GPU costs by paying only for GPU time that does useful work. In order of impact: measure how busy each GPU really is, shut down GPUs that sit idle, run each job on the smallest GPU it fits, pack more work onto each GPU, and only then optimize the price per hour through reservations, spot capacity and provider choice. Most savings come from the first three steps, not from negotiating the hourly rate.

Measure utilization correctly

The number most dashboards show is misleading. nvidia-smi reports GPU utilization as the percent of time during which one or more kernels was executing on the GPU, as NVIDIA’s NVML and PyTorch’s documentation define it. A single small kernel running on a few of the GPU’s streaming multiprocessors (SMs) counts as 100% busy.

DCGM profiling metrics measure what you actually pay for. SM activity (DCGM_FI_PROF_SM_ACTIVE) is the fraction of time at least one warp was active on a multiprocessor, averaged over all multiprocessors. A GPU can show high utilization in nvidia-smi while SM activity stays low. Collect SM activity and memory use per GPU with the DCGM Exporter, then tie them to the Pods and teams that requested the GPUs. That tells you where the waste is before you change anything.

Stop paying for idle GPUs

An idle GPU costs the same as a busy one. As of November 2026, AWS lists an 8-GPU p6-b200.48xlarge at about $114 per hour on demand in US East, so one forgotten node costs about $2,700 a day. Common sources of idle time:

• Notebooks and development environments left running overnight and on weekends.
• Inference replicas sized for peak traffic that run all day.
• Training jobs that finished while their nodes stayed up.

Autoscaling that removes empty GPU nodes, and scaling inference to zero replicas when there is no traffic, close most of that gap. The trade-off is cold start. A new GPU node needs minutes to boot and load drivers, and a large model needs more time to load its weights from storage. Keep a warm minimum for latency-sensitive services and scale everything else to zero. The mechanics are in how Kubernetes autoscaling works.

Match the GPU to the job

Training large models needs H100, H200 or Blackwell GPUs with fast interconnects. Most inference, embedding, reranking and fine-tuning of small models does not. OVHcloud’s price list, for example, starts the L4 at $1.00 per hour and the H100 at $2.99 per hour as of November 2026. A model that fits in an L4’s 24 GB costs a third as much to serve there. Check memory first with how much GPU memory an LLM needs, then benchmark cost per token or per request on two or three GPU types. The cheapest GPU per hour is not always the cheapest per token.

Put more work on each GPU

Two techniques raise the useful work per GPU-hour:

• Batching. Inference servers such as vLLM use continuous batching, which adds new requests to the batch while others are still generating. One GPU then serves many concurrent users instead of one at a time.
• Sharing. Small models that use a fraction of a GPU can share it through MIG, time slicing or MPS. See GPU sharing in Kubernetes for which one fits.

Quantization helps too. FP8 weights take half the memory of BF16, so a model may fit on one GPU instead of two, or leave room for a larger KV cache and more concurrent requests on the same GPU.

Choose the buying model

Once utilization is known, the buying model matters:

Spot GPUs are cheaper but the provider can reclaim them, so jobs must checkpoint and resume. Committing beyond the real baseline turns into paying for idle capacity again. The trade-offs are compared in GPU procurement in a shortage.

Compare providers

The same GPU costs very different amounts at different providers. In Cloudfleet’s snapshot of public on-demand list prices on 4 November 2026, an H100 ranged from $2.45 to $11.06 per GPU per hour, with specialized GPU clouds usually below the hyperscalers. The full table is on the GPU Kubernetes page. Compare more than the hourly price: some providers charge per GPU and others per 8-GPU node, and CPU, memory, local disk and networking may or may not be included.

Watch storage and egress

GPU jobs move large datasets, checkpoints and model weights. Moving them out of a provider, or between regions, can add a meaningful share to the bill. Keep training data next to the GPUs that read it, and cache model weights in the region where inference runs. Cloud egress fees explains how providers charge for it.

Use quotas as guardrails

Cloud GPU quotas are usually seen as an obstacle, but they also cap spend. Set quotas per project or team at the level you budget for, and add budget alerts per account. Kubernetes ResourceQuota objects can cap requests.nvidia.com/gpu per namespace, so one team cannot take every GPU in a shared cluster. The same discipline applies to the rest of the cluster, covered in how to reduce Kubernetes costs.

GPU costs on Cloudfleet

On Cloudfleet, node auto-provisioning creates a GPU node when a Pod requests nvidia.com/gpu and removes it when the work is done, and time slicing or MPS lets several Pods share one GPU on the Pro and Enterprise tiers. When several connected clouds offer the requested GPU, Cloudfleet provisions the most cost-effective one, and providers bill you directly with no markup on compute, as the GPU workloads documentation and GPU Kubernetes page describe.