GPU Cloud Pricing: AWS, Azure and GCP for AI and ML Workloads 2026

GPU cloud instances are the most expensive compute resources available in public cloud, with on-demand prices ranging from $0.90/hour for an entry-level GPU to $98/hour for the largest NVIDIA H100 cluster instances. As AI and machine learning workloads have become central to enterprise technology strategy, GPU cloud costs have become a significant and often poorly understood portion of cloud bills. This guide breaks down how GPU pricing works across AWS, Azure, and GCP, compares specific instance families, and provides practical strategies for reducing GPU costs without sacrificing training throughput.

GPU Instance Families: AWS vs Azure vs GCP

Each cloud provider offers multiple GPU instance families, optimized for different use cases ranging from inference serving to large model training. The GPU hardware itself is typically NVIDIA, though the specific generation and configuration varies by instance family and provider.

Understanding NVIDIA GPU Generations and Performance

The GPU hardware generation has a profound impact on training throughput per dollar. An NVIDIA H100 SXM5 delivers approximately 3.35× the FP16 throughput of an A100, and approximately 9× the throughput of a V100. This means that while an H100 costs roughly 3× more per hour than an A100, it completes training jobs roughly 3× faster — making the total cost of a training run roughly equivalent while dramatically reducing wall-clock time.

For inference workloads, the calculus is different. Inference is often latency-constrained rather than throughput-constrained, meaning that throwing a more powerful GPU at the problem doesn't proportionally reduce cost — you may need the same number of instances regardless of GPU generation to meet latency SLAs. For inference, T4 GPU instances often provide the best cost efficiency per query.

AWS's proprietary Inferentia2 chip (used in inf2 instances) offers competitive inference economics for PyTorch and TensorFlow models, with pricing approximately 30% below equivalent NVIDIA GPU inference instances and better throughput for batch inference workloads that can tolerate compilation overhead.

Spot and Preemptible GPU Instances: 60–90% Savings

The single most impactful cost lever for ML training workloads is Spot/Preemptible GPU instances. AWS Spot pricing for p3.2xlarge (V100) is approximately $0.918/hour versus $3.06/hour on-demand — a 70% saving. For a training run that takes 100 hours on-demand ($306), the same job on Spot costs approximately $92 — assuming no interruptions.

The risk with Spot is interruption. AWS can reclaim Spot instances with a 2-minute warning when capacity is needed for on-demand customers. For long training jobs, this means you must implement checkpointing — saving model state to durable storage (S3/GCS/Azure Blob) at regular intervals so the job can resume from the last checkpoint rather than starting over.

Reserved GPU Instances: When to Commit

For production inference serving — where a GPU instance runs 24/7 to serve model predictions — reserved pricing makes strong economic sense. A 1-year reserved p3.2xlarge in AWS costs approximately $1.22/hour, versus $3.06 on-demand — a 60% saving. Over a full year, that single instance saves $16,500 on the compute cost alone.

Reserved GPU instances make sense when: the workload runs more than 60% of the time (making Spot unreliable due to interruption risk), the capacity requirement is predictable, and the commitment period aligns with your model lifecycle. Model refreshes that require instance type changes every 6 months argue against 3-year commitments.

Managed AI Services vs Raw GPU Compute

Cloud providers offer managed AI services that abstract away GPU management — SageMaker (AWS), Azure Machine Learning, and Vertex AI (GCP). These services add a platform premium of 20–40% over raw GPU instance pricing in exchange for managed training infrastructure, experiment tracking, model registry, and deployment automation.

AWS SageMaker Training Jobs using ml.p3.2xlarge costs $4.131/hour versus $3.06/hour for a raw EC2 p3.2xlarge — a 35% platform premium. Azure ML compute clusters add a similar premium. The managed services are worth the premium for teams that value the ML lifecycle management features; teams that can self-manage infrastructure should use raw GPU instances directly.

For inference specifically, managed endpoints add even larger premiums. SageMaker Real-Time Inference on ml.g4dn.xlarge costs $0.736/hour versus $0.526 for raw EC2 — a 40% premium. At scale, this can represent hundreds of thousands of dollars per year in avoidable costs for teams that can manage their own inference infrastructure with containers and a load balancer.

Worked Cost Examples: Training and Inference

Training a large language model (70B parameter) on AWS: Using 8× p4d.24xlarge instances (8 A100 GPUs each = 64 A100s total) at $32.77/hour each = $262/hour total. A full training run of 100 hours costs $26,200 on-demand. Using Spot instances at 70% discount: $7,860. With checkpointing overhead adding 10% to total time: $8,646 on Spot.

Production inference serving for a BERT-based NLP model on GCP: Using 4× n1-standard-8 with T4 GPUs at $0.462/hour each = $1.848/hour = $1,349/month. At GCP committed use discount (1-year): $944/month. Serving 10 million inference requests/month, total cost per 1,000 requests: $0.094.

GPU Cost Optimization Strategies

Implement checkpointing for all training jobs. This is the prerequisite for using Spot instances safely. Save model checkpoints to S3/GCS/Azure Blob every 15–30 minutes. With checkpointing, Spot interruptions cost at most 15–30 minutes of lost work, not entire training runs.

Use mixed-precision training (FP16/BF16). Training in FP16 instead of FP32 approximately doubles training throughput on modern NVIDIA GPUs, cutting training time and cost in half. Most modern ML frameworks support this with a one-line configuration change.

Right-size your inference fleet. Many teams over-provision inference capacity to handle peak load, running expensive GPU instances at 10–20% utilization during off-peak hours. Use auto-scaling with target tracking policies to scale inference capacity to actual demand, and consider serverless inference options (SageMaker Serverless, Azure Container Apps) for workloads with spiky or unpredictable traffic.