Market economics
Inference vs Training Markets
Training wants dense, reliable blocks of accelerators. Inference wants high utilization, low latency, batching, caching, and routing. They share GPUs, but they do not buy compute the same way.
"combining of one or more inference requests into a single batch"
Key facts
Training is a capacity-block market
Training large models consumes GPUs in blocks. Multi-GPU nodes, high-bandwidth interconnect, fast storage, and network topology matter because one weak link can hold the job back. The buyer is not simply renting eight accelerators. They are buying a synchronized system that can keep the job fed and recover from failure.
This is why training buyers often value reserved capacity, cluster scheduling, local NVMe, InfiniBand or equivalent networking, and support access. The meter is nominally GPU-hour, but the decision is closer to cost per successful training run.
Inference is a utilization and latency market
Inference can be sold as raw GPU instances, managed serverless workers, model APIs, per-token endpoints, or per-call tools. The right meter depends on traffic shape. A steady production model with high QPS can justify dedicated GPUs. A bursty or experimental workload may be cheaper on serverless inference or managed model APIs even if the apparent token price looks higher.
The biggest economic difference is batching. A GPU that is expensive at low utilization can become competitive when requests are packed efficiently. NVIDIA Triton calls this dynamic batching. vLLM describes continuous batching, prefix caching, and PagedAttention. These are not implementation footnotes; they are the difference between selling raw accelerator time and selling profitable low-latency tokens.
Per-token pricing hides hardware and exposes product economics
A managed API price says nothing about which GPU served the request. That is the point. The provider absorbs hardware mix, scheduling, speculative decoding, caches, model replicas, reliability engineering, abuse handling, and routing. Buyers pay for a finished token stream.
The resulting comparison is product-like. The cheap model can be expensive if it needs longer prompts, more retries, tool calls, or more output to reach the same answer quality. The expensive model can be cheap if it solves the task in fewer attempts or supports a cacheable long context. Buyers should measure cost per accepted task, not price per nominal token alone.
Batch inference and real-time inference should be budgeted separately
Batch inference can trade latency for lower unit cost. Managed APIs often expose batch discounts, and self-hosted stacks can schedule offline jobs when GPUs would otherwise be idle. Real-time inference trades in the other direction: p95 latency, cold starts, safety checks, and availability matter.
A practical compute budget usually has three lanes: dedicated capacity for predictable baseline traffic, burst capacity for spikes, and batch capacity for offline jobs. Treating those as one blended compute line hides the decisions that actually move cost.