Inference cost per token is the cost of generating one token on the hardware serving a model. It is not the price a provider charges per token, although the two are related. Most of the factors that move it act through one ratio: hourly hardware cost divided by the tokens that hardware realizes per hour. This article separates the mechanisms that the sources support from the magnitudes that rest on vendor figures, and gives the source for each number.
Definitions: production cost versus rate-card price
A GPU-hour is one hour of one accelerator's time, priced at the rental rate the operator pays. Throughput is the number of tokens generated per second. Production cost per token is therefore the rental rate divided by tokens per hour. Inworld AI's guide states that a GPU-hour costs the same whether the system emits 2,000 or 27,000 tokens per second, so the entire difference lies in throughput (Inworld AI, vendor guide).
A rate-card price is what a customer pays per million tokens. Providers bill input and output tokens at separate rates, so a customer's bill equals input tokens times the input rate plus output tokens times the output rate (Tetrate, practitioner guide). That is a billing formula. It says nothing about the provider's production cost. No source reviewed measures how much of a frontier closed model's price is GPU cost, margin or overhead.
Utilization and batch size
Batch size is the largest lever in the evidence. Decode, the phase that generates tokens one step at a time, is limited by memory bandwidth. At batch size 1 a single request cannot keep a modern GPU busy, so most of the hardware's capacity goes unused. Adding requests to the batch spreads the same memory reads across more tokens (Inworld AI).
Vendor tables put numbers on this. Spheron reports one H100 setup falling from about $258 per million tokens at batch size 1 to about $2.30 at batch size 256 (Spheron benchmark). A second Spheron page describes the same change as roughly 100x (Spheron cost-per-million-tokens). The model, precision and sequence lengths behind these tables are not recorded in the sources reviewed, and both pages are vendor material. Treat the 100x figure as one configuration, not a general property of GPUs.
An academic energy study gives the neutral version of the point. Larger batches can lower energy per token by sharing fixed costs, but the effect on energy per request depends on request shape and platform (arXiv 2608.28044).
Why output tokens cost more than input tokens
CloudZero reports that output tokens cost roughly five times input tokens across major providers (CloudZero). That is a rate-card observation, not a measured cost split. The explanation offered is the prefill/decode split. Prefill processes the prompt in parallel and is compute-bound; decode generates tokens sequentially and is memory-bound (Inworld AI). In the author's reading, this is consistent with the price gap, but no source reviewed provides a measured production cost for each phase.
Context length and the KV cache
KV cache memory grows linearly with context length, because each token in the context is stored as a key-value pair (arXiv 2603.20397). That memory competes with batch size for GPU memory. The JAX Scaling Book notes that a smaller KV cache allows a larger batch size and context at the generation step (JAX Scaling Book). The mechanism is well supported. The dollar effect in any given deployment is not established. A "3x to 5x context bloat" figure in one consultancy blog is unsourced and is not used here.
Model size, precision and hardware
Smaller models and lower-precision weights move fewer bytes per token, and cheaper hardware lowers the hourly rate. The sources support these mechanisms. The magnitudes are weaker. Mirantis says a 70B model raises per-token cost by two to three times relative to a 7B model, with no method given (Mirantis). Packet.ai reports that moving an H200 running Llama 4 70B from BF16 to FP8 cut cost per output token from about $0.21 to $0.13 per million, a 38% reduction (Packet.ai). Packet.ai is cloud marketing, and confidence is low. Mirantis also attributes an 85% saving from batching 32 requests, at 20% more latency. No original measurement behind that figure was found. Treat all three as unverified.
Why self-hosting figures disagree
The same hourly rate can produce very different costs per token. Universeblend shows one H100 rate yielding between $0.17 and $8.74 per million output tokens, depending only on throughput (Universeblend). I checked the arithmetic against that page's own table, but the source has low authority. Rental rates vary too. Inworld reports public NVIDIA B200 list rates from $3.49 to $14.24 per GPU-hour across clouds in April 2026 (Inworld AI).
Before comparing two figures, ask for the rental rate, batch size or concurrency, sequence lengths, and realized tokens per second. Vendors often leave some of these out, and the result moves with each.
Per-token price versus total cost
A lower price per token does not guarantee a lower bill. Token efficiency, meaning the number of input and output tokens a model needs to finish a task, can dominate. In one example attributed to Artificial Analysis through an arXiv preprint, Sonnet completes a benchmark at roughly $817 total, against $1,888 for Grok-4 (arXiv 2511.11788). I did not check this against Artificial Analysis directly. The energy study makes the general point that a lower average cost per token need not mean a lower total request cost (arXiv 2608.28044). One blog reports reasoning models consuming internal tokens at up to 100x their visible output (AI Superior). Confidence is low.
Price declines and margins
Epoch AI measured the cheapest price to reach fixed benchmark scores. The annual rate of decline varies by performance milestone, from 9x to 900x per year (Epoch AI). Epoch names smaller models and more cost-effective hardware as well-known drivers, and says other factors are hard to determine from public information. Epoch also says reduced margins may explain some of the drops, but it found no clear evidence for this.
Secondary 2026 coverage reports a newer figure of about 47% per quarter, roughly 13x per year, from a report titled "The plunging price of thought" (Pasquale Pillitteri; Yano AI). I have not checked that against an Epoch primary, and it does not reconcile with the 9x to 900x series. Do not treat the two as one series.
For open models, 2024 data analysed in arXiv 2506.04645 found providers serving models within a factor of two of maximum tokens per second, and API markups over GPU rental costs appearing to be under a factor of two. The sample excludes Groq and uses dated data. It does not describe frontier closed models.
What is not established
- No primary measurement of cost per token across batch size, context length and output length on one fixed setup was found.
- The margin share of frontier closed-model prices is unknown.
- Effects of speculative decoding and continuous batching in production lack independent measurement.
- Pope et al. (2023), "Efficiently scaling transformer inference," is cited in a reference list (arXiv 2609.24298) but was not retrieved. It should be read directly.
- The 2026 decline figures have not been reconciled across sources.
When you compare cost-per-token figures, ask for four inputs: the hourly rate, the batch size, the realized tokens per second, and the tokens used per task. Without them, a figure is the vendor's assumption, not a measurement.