GPU Cloud Pricing: AWS, Azure and GCP for AI/ML Workloads

GPU Instance Families Across AWS, Azure, and GCP

All three providers organize GPU instances around the same underlying NVIDIA hardware generations, but wrap them in provider-specific instance families: AWS uses the P-series (P4, P5) and G-series (G5, G6), Azure uses the NC, ND, and NG series, and GCP uses the A2 and A3 machine families alongside Cloud TPUs (GCP's own AI accelerator, not GPU-based). The underlying chip — not the instance family name — is what actually determines training throughput, so cost comparisons should be done GPU-model to GPU-model rather than instance-family to instance-family.

GPU Hourly Pricing Comparison (On-Demand, 2026)

Per-GPU on-demand pricing, US region, single GPU. Multi-GPU instances (8x H100, etc.) scale roughly linearly plus a networking/NVLink premium.

On-Demand vs Reserved vs Spot for GPUs

GPU capacity is in persistently short supply relative to demand, which changes the usual cloud cost-optimization playbook. Reserved/committed GPU pricing (1–3 year terms) typically saves 30–50%, similar to CPU instances, but availability is the real constraint — providers frequently limit or waitlist on-demand H100 and A100 capacity in popular regions, making a capacity reservation valuable even before considering the discount.

Multi-GPU and Cluster Pricing

Training large models requires multi-GPU nodes connected via high-bandwidth interconnect (NVLink within a node, InfiniBand or equivalent between nodes). An 8x H100 node costs roughly 8x the per-GPU rate plus a 10–20% premium for the NVLink/InfiniBand networking hardware — an AWS p5.48xlarge (8x H100) runs approximately $98–110/hour on-demand. Multi-node training clusters add further networking costs for inter-node bandwidth (EFA on AWS, InfiniBand on Azure/GCP) that are often billed separately from compute.

Storage and Networking for AI Workloads

GPU compute is usually not the only significant cost in an AI/ML pipeline. Training datasets stored in object storage (S3, Blob Storage, GCS) and read repeatedly across epochs benefit from provider-native high-throughput storage tiers, and moving large datasets between regions or out of the cloud for a multi-cloud training setup can generate substantial egress charges — a 10TB training dataset transferred cross-region costs $200–800 in transfer fees alone before any training compute runs.

Cost Optimization Tips for ML Training

• Right-size the GPU to the model: Fine-tuning and inference for small-to-mid models frequently runs fine on A10G/T4-class GPUs at a fraction of A100/H100 cost — reserve the largest GPUs for training runs that actually saturate them.
• Use spot for checkpointed training: Jobs with checkpointing every few minutes can tolerate spot interruptions and capture 60%+ savings with minimal wall-clock time penalty.
• Reserve capacity ahead of large training runs: Given GPU scarcity, committing to capacity 2–4 weeks ahead of a planned training run is often the only way to guarantee availability, independent of cost.
• Monitor utilization, not just cost: A100/H100 instances left idle between training runs are the single biggest source of AI infrastructure waste — GPU utilization monitoring should be a standing FinOps practice for any ML team.