The price of a fixed level of AI performance has fallen roughly 1,000x in three years (a16z, LLMflation 2024) — GPT-3-quality output that cost $60 per million tokens in late 2021 sold for $0.06 by late 2024. Stanford HAI’s 2025 AI Index measured a 280-fold drop for GPT-3.5-equivalent quality in about 18 months (Stanford HAI, AI Index 2025). Epoch AI tracks a median decline near 50x per year across benchmarks, accelerating to 200x per year since January 2024 (Epoch AI, LLM Inference Price Trends 2025). Yet total spending is climbing the other way: The Information reported OpenAI’s 2025 inference bill near $8.4 billion, about four times the prior year. This analysis consolidates data from Epoch AI, Stanford HAI, a16z, Google, NVIDIA, Menlo Ventures, and 12 other primary sources on where inference economics stand in mid-2026.
TL;DR
- GPT-3-equivalent output fell from $60 per million tokens in 2021 to $0.06 by 2024, a 1,000x drop (a16z, LLMflation 2024).
- Querying GPT-3.5-equivalent quality fell from $20 to $0.07 per million tokens, a 280-fold cut in about 18 months (Stanford HAI, AI Index 2025).
- Benchmark price-performance decline ranges 9x to 900x per year; median 50x, rising to 200x since January 2024 (Epoch AI, 2025).
- Isolated algorithmic efficiency progress runs about 3x per year, on top of hardware gains (The Price of Progress, arXiv 2025).
- Market-average NVIDIA H100 on-demand rental was about $3.61 per GPU-hour across 42 providers in 2026 (getdeploying, 2026).
- NVIDIA’s GB300 NVL72 hit $0.12 per million tokens, described as 35x lower cost per token than Hopper (NVIDIA, InferenceMAX 2025).
- Google processed 3.2 quadrillion tokens per month by mid-2026, roughly 7x year over year (Google I/O 2026).
- Enterprise LLM API spend doubled from $3.5 billion to $8.4 billion in six months (Menlo Ventures, 2025).
- OpenAI’s 2025 inference cost reached about $8.4 billion, roughly 4x the prior year (The Information, 2025).
- Combined training-plus-inference infrastructure was sized at $251 billion in 2025, growing to $672 billion by 2029 (Bloomberg Intelligence, via Cerebras S-1 2026).
- The median Gemini text prompt uses 0.24 watt-hours, down 33x in 12 months (Google, 2025).
1. The Price Collapse for Equivalent Performance
The headline number in inference economics is not any single price — it is the slope. For a fixed quality bar, the cost of running a model has been halving on a scale of months, not years, and the fall has steepened since early 2024. The divergence across benchmarks matters: cheap commodity quality is falling fastest, while frontier reasoning quality holds its price. Epoch AI measures a median decline of 50x per year, jumping to 200x per year when restricted to data since January 2024. The floor keeps dropping because three forces stack: competition, better silicon, and better algorithms. Isolating the last, a research team put pure algorithmic efficiency at roughly 3x per year.
| Metric | Value | Source |
|---|---|---|
| GPT-3-equivalent quality (MMLU ~42) price | $60/M tokens (Nov 2021) to $0.06/M (2024), a 1,000x drop | a16z, LLMflation (2024) |
| LLMflation rate for equal performance | ~10x cheaper per year | a16z, LLMflation (2024) |
| GPT-3.5-equivalent quality (MMLU 64.8%) price | $20/M (Nov 2022) to $0.07/M (Oct 2024), 280-fold | Stanford HAI, AI Index (2025) |
| Cross-benchmark price-performance decline | 9x to 900x per year; median 50x | Epoch AI, Inference Price Trends (2025) |
| Post-Jan-2024 acceleration | Median 200x per year | Epoch AI, Inference Price Trends (2025) |
| GPT-4 quality on GPQA Diamond | 40x per year price drop | Epoch AI, Inference Price Trends (2025) |
| Frontier benchmark price decline | 5x to 10x per year | The Price of Progress (arXiv, 2025) |
| Isolated algorithmic efficiency progress | ~3x per year | The Price of Progress (arXiv, 2025) |
Outlier note: Epoch’s own dataset behind these figures rests on 36 unique price observations across six benchmarks, so single-benchmark rates (the 900x extreme) are noisier than the pooled median.
2. Hardware and GPU Economics
Falling token prices ride on falling compute prices. Rental rates for the workhorse NVIDIA H100 kept sliding through 2025 into 2026 as specialized clouds undercut hyperscalers, and each new accelerator generation resets the cost-per-token floor. NVIDIA’s InferenceMAX benchmarks, published October 2025, put the GB300 NVL72 at $0.12 per million tokens — 35x lower cost per token than the Hopper generation. The vendor framing is promotional, so treat the multiples as best-case, but the direction matches the independent price trackers below. Gartner projects the trend runs for years: inference on a trillion-parameter model should cost more than 90 percent less in 2030 than in 2025.
| Metric | Value | Source |
|---|---|---|
| H100 on-demand average, 42 providers | ~$3.61 per GPU-hour | getdeploying (2026) |
| H100 cloud price drop from peak | 64% to 75% | GPU price trackers (2026) |
| AWS H100 price cut, June 2025 | ~44% | GPU price trackers (2026) |
| ML hardware performance growth | 43% per year, doubling every 1.9 years | Stanford HAI, AI Index (2025) |
| ML hardware cost decline / energy efficiency | -30% per year / +40% per year | Stanford HAI, AI Index (2025) |
| NVIDIA GB300 NVL72 cost per token | $0.12/M, 35x lower than Hopper | NVIDIA, InferenceMAX (2025) |
| Blackwell cost per million tokens | 15x lower than prior generation | NVIDIA, InferenceMAX (2025) |
| Projected 2030 cost, 1T-parameter inference | >90% lower vs 2025 | Gartner (2025) |
Context: the same NVIDIA disclosure claims a $5 million GB200 NVL72 buildout generates about $75 million in token revenue, a 15x return — a supplier’s ROI pitch, not an independent audit. For the broader silicon picture, see our AI chips statistics.
3. The Token Explosion
Cheaper tokens did not shrink the bill; they detonated demand. As per-token prices fell, providers began measuring throughput in quadrillions, and agentic workloads that fire dozens of calls per task push volume higher still. Google reported processing 3.2 quadrillion tokens per month by mid-2026, roughly 7x its year-earlier rate. This is the Jevons paradox in real time: the more efficient the resource, the more of it gets consumed. Gartner notes agentic queries alone consume 5 to 30 times more tokens than a standard chatbot turn, so the volume curve is steepening as products shift from single answers to multi-step agents.
| Metric | Value | Source |
|---|---|---|
| Google token throughput | 3.2 quadrillion/month (mid-2026), 7x YoY | Google I/O (2026) |
| Google token growth | 480T/month (May 2025) to 980T (July 2025) | T. Tunguz, Token Race (2025) |
| Microsoft Foundry throughput | 100T tokens in a quarter; 50T single-month record | T. Tunguz, Token Race (2025) |
| Microsoft large-customer count | 250+ customers on track for >1T tokens/year each | Microsoft, via T. Tunguz (2025) |
| Together.ai throughput | 2 trillion tokens/day (Sept 2025) | T. Tunguz, Token Race (2025) |
| Open-source share of inference volume | ~1% to 3% | T. Tunguz, Token Race (2025) |
| Agentic query token multiplier | 5x to 30x more tokens than a chatbot query | Gartner (2025) |
Context: agentic AI accounted for less than 1 percent of Azure’s overall inference activity in the cited window, so the 5x-30x multiplier is applied to a small but fast-growing base. For where those workloads head next, see our AI agents statistics.
4. Enterprise Spend and the Margin Paradox
Here is the paradox in one line: unit prices are collapsing, yet buyers and sellers are both spending more. Enterprises shifted budget from experimentation to always-on production, where inference never stops. Menlo Ventures found 74 percent of AI builders now say the majority of their workloads are inference, up from 48 percent a year earlier. On the supplier side, inference is the dominant cost line: The Information reported OpenAI’s 2025 inference bill near $8.4 billion, roughly four times the prior year and well above its own $6.6 billion forecast. Margins are the pressure point — the same reporting put OpenAI’s gross margin at 33 percent, below its 46 percent target.
| Metric | Value | Source |
|---|---|---|
| Enterprise LLM API spend | $3.5B (late 2024) to $8.4B (mid-2025) | Menlo Ventures (2025) |
| Builders with inference-majority workloads | 74%, up from 48% a year earlier | Menlo Ventures (2025) |
| Enterprise generative-AI investment | $1.7B (2023) to $37B (2025) | Menlo Ventures (2025) |
| OpenAI 2025 inference cost | ~$8.4B, ~4x YoY (vs $6.6B forecast) | The Information (2025) |
| OpenAI 2026 inference cost projection | ~$14.1B | The Information (2025) |
| Anthropic 2025 inference cost | ~$2.7B, >3x growth | The Information (2025) |
| OpenAI gross margin | 33% (fell from 40%, missed 46% target) | The Information (2025) |
| Barclays consumer inference chip capex, 2026 | ~$120 billion | Barclays Research (2025) |
Context: Anthropic’s API business is reported to run gross margins above 80 percent, a reminder that the “inference loses money” story is model-mix and pricing dependent, not universal. For the software economics underneath, see our SaaS statistics.
5. Cost Levers: Open Models, Caching, and Batch
Sticker prices overstate what disciplined teams actually pay. Open-weight models, prompt caching, and asynchronous batching each cut effective cost by an order of magnitude, and they stack. DeepSeek’s R1 runs about 96 percent cheaper than OpenAI’s o1 on a per-token basis, roughly a 27x gap. The frontier tier tells the opposite story: OpenAI’s o1 output price of $60 per million tokens is identical to what GPT-3 cost at launch in 2021, so premium reasoning has not gotten cheaper even as commodity quality cratered. The savings live in the levers below.
| Metric | Value | Source |
|---|---|---|
| DeepSeek R1 API price | $0.55/M input, $2.19/M output | DeepSeek pricing (2026) |
| OpenAI o1 API price | $15/M input, $60/M output | pricepertoken (2026) |
| DeepSeek R1 vs OpenAI o1 | ~96% cheaper (~27x) | pricepertoken (2026) |
| OpenAI o1 output vs GPT-3 launch | $60/M, identical to 2021 GPT-3 | a16z, LLMflation (2024) |
| DeepSeek R1 cached input | $0.14/M tokens | DeepSeek pricing (2026) |
| DeepSeek V4 cached input | $0.03/M tokens, a 90% discount | DeepSeek pricing (2026) |
| OpenAI Batch API discount | Flat 50% for a 24-hour window | OpenAI Batch API (2026) |
| Cache + batch, GPT-5.4 cached input | $0.625/M, 75% below standard | OpenAI Batch API (2026) |
Outlier note: output tokens are priced 3x to 5x higher than input tokens across major providers because generation is sequential while input processing parallelizes, so caching helps input-heavy workloads most. For how open-weight models reshape this, see our cloud computing statistics.
6. Market Size and Forecasts
Aggregate market sizing for AI inference is wide, because analysts draw the boundary differently — some count only inference chips, others the full software-plus-services stack. Treat the spread as a range, not a contradiction. Bloomberg Intelligence, cited in Cerebras’ S-1, sized the combined training-plus-inference infrastructure market at $251 billion in 2025, growing to $672 billion by 2029, with inference expanding more than twice as fast as training. The narrower inference-chip slice grows steadily rather than explosively, while broad “AI inference” definitions land near a quarter-trillion dollars by 2030.
| Metric | Value | Source |
|---|---|---|
| AI inference market, 2025 to 2030 | $106.15B to $254.98B, 19.2% CAGR | MarketsandMarkets (2025) |
| AI inference market, 2024 to 2030 | $97.24B to $253.75B, 17.5% CAGR | Grand View Research (2025) |
| AI inference market, 2026 to 2034 | $117.80B to $312.64B, 12.98% CAGR | Fortune Business Insights (2026) |
| AI inference chip market, 2026 to 2030 | $20.51B to $36.97B, 15.9% CAGR | The Business Research Company (2026) |
| Combined training + inference infra, 2025 to 2029 | $251B to $672B, ~28% CAGR | Bloomberg Intelligence, via Cerebras S-1 (2026) |
| Inference vs training growth | Inference grows >2x faster | Bloomberg Intelligence, via Cerebras S-1 (2026) |
Divergence note: 2030 estimates for “AI inference” span roughly $255B (MarketsandMarkets) to $313B by 2034 (Fortune Business Insights); the gap reflects scope and base-year differences, not a data error.
7. Energy and Cost per Query
The per-query resource footprint is falling as fast as the dollar price, and 2025 was the year the biggest provider finally published numbers. Google disclosed that its median Gemini text prompt uses 0.24 watt-hours of electricity — and that this figure fell 33x over the prior 12 months. Independent estimates land in the same neighborhood, which is a useful cross-check on vendor math. The caveat carries weight: these are medians, and complex reasoning or agentic tasks consume several to tens of watt-hours, so the average bill rises as products shift toward heavier workloads.
| Metric | Value | Source |
|---|---|---|
| Median Gemini text prompt energy | 0.24 watt-hours | Google Cloud (2025) |
| Median Gemini prompt water / carbon | 0.26 mL / 0.03 gCO2e | Google Cloud (2025) |
| Energy per median Gemini prompt, 12-month change | Fell 33x | Google Cloud (2025) |
| Carbon per median Gemini prompt, 12-month change | Fell 44x | Google Cloud (2025) |
| Average ChatGPT query energy | ~0.34 watt-hours | OpenAI (Sam Altman, 2025) |
| Typical query estimate | ~0.3 watt-hours | Epoch AI (2025) |
| Complex / agentic query energy | Several to tens of watt-hours | Google Cloud (2025) |
Context: Google’s disclosure covers full-system power including idle capacity and data-center overhead (PUE), a broader boundary than the chip-only figures often quoted elsewhere, so it is not directly comparable to back-of-envelope estimates.
Summary: AI Inference Cost by the Numbers
| Metric | Value | Source |
|---|---|---|
| GPT-3-equivalent price | $60/M (2021) to $0.06/M (2024), 1,000x | a16z, LLMflation (2024) |
| GPT-3.5-equivalent price | 280-fold drop, late 2022 to Oct 2024 | Stanford HAI, AI Index (2025) |
| Cross-benchmark price decline | Median 50x/year; 200x/year since Jan 2024 | Epoch AI (2025) |
| Algorithmic efficiency progress | ~3x per year | The Price of Progress (arXiv, 2025) |
| H100 on-demand average | ~$3.61 per GPU-hour, 42 providers | getdeploying (2026) |
| H100 price drop from peak | 64% to 75% | GPU price trackers (2026) |
| ML hardware cost / energy efficiency | -30% / +40% per year | Stanford HAI, AI Index (2025) |
| NVIDIA GB300 NVL72 cost per token | $0.12/M, 35x lower than Hopper | NVIDIA, InferenceMAX (2025) |
| Google token throughput | 3.2 quadrillion/month, 7x YoY | Google I/O (2026) |
| Enterprise LLM API spend | $3.5B to $8.4B in six months | Menlo Ventures (2025) |
| Inference-majority workloads | 74% of builders, up from 48% | Menlo Ventures (2025) |
| OpenAI 2025 inference cost | ~$8.4B, ~4x YoY | The Information (2025) |
| Anthropic 2025 inference cost | ~$2.7B, >3x growth | The Information (2025) |
| DeepSeek R1 vs OpenAI o1 | ~96% cheaper (~27x) | pricepertoken (2026) |
| AI inference market | $106B (2025) to $255B (2030), 19.2% CAGR | MarketsandMarkets (2025) |
| Combined train + inference infra | $251B (2025) to $672B (2029) | Bloomberg Intelligence, via Cerebras S-1 (2026) |
| Median Gemini prompt energy | 0.24 Wh, down 33x in 12 months | Google Cloud (2025) |
| Gartner 2030 cost projection | 1T-param inference >90% cheaper vs 2025 | Gartner (2025) |
| Agentic query token multiplier | 5x to 30x more tokens | Gartner (2025) |
| OpenAI Batch API discount | Flat 50% for 24-hour window | OpenAI (2026) |
Methodology and Sources
Data was gathered by aggregating figures from primary reports, company disclosures, regulatory filings, named price trackers with public methodology, and peer-reviewed analysis published in 2024 through 2026, prioritizing 2025-2026 data and tracing trade-press mentions back to the originating source.
- Andreessen Horowitz (a16z), Welcome to LLMflation (2024) — a16z.com
- Epoch AI, LLM Inference Price Trends and How Much Energy Does ChatGPT Use? (2025) — epoch.ai
- Stanford HAI, 2025 AI Index Report, Research and Development chapter (2025) — hai.stanford.edu
- The Price of Progress: Algorithmic Efficiency and the Falling Cost of AI Inference (arXiv:2511.23455, 2025)
- Gartner, AI inference cost forecast, reported via CIO Dive (2025)
- NVIDIA, Blackwell InferenceMAX Benchmark Results (October 2025)
- Google / Google Cloud, Measuring the Environmental Impact of AI Inference (2025); MIT Technology Review reporting (2025)
- Tomasz Tunguz, The Trillion Token Race (2025); Google I/O 2026 keynote (Sundar Pichai)
- Menlo Ventures, 2025: The State of Generative AI in the Enterprise (2025)
- The Information, OpenAI and Anthropic inference-cost reporting (2025)
- Bloomberg Intelligence, market sizing cited in Cerebras Systems S-1 filing, U.S. SEC (2026)
- Barclays Research, consumer AI inference capex estimate (2025)
- DeepSeek, API pricing documentation (2026); pricepertoken.com model pricing (2026)
- OpenAI, Batch API documentation (2026); Sam Altman public statement on query energy (2025)
- getdeploying and other NVIDIA H100 cloud price trackers (2026)
- MarketsandMarkets, Grand View Research, and Fortune Business Insights, AI inference market reports (2024-2026); The Business Research Company, AI inference chip market (2026, via GlobeNewswire)
Data watch: Stanford HAI publishes the AI Index annually (next edition expected around April 2027); Menlo Ventures refreshes its State of Generative AI report yearly; Epoch AI updates its inference-price and trends trackers continuously; Google is expected to update its environmental-impact disclosure annually; and Gartner and the market-research firms reissue their inference forecasts on rolling cycles. Frontier API prices change frequently, so per-token figures are point-in-time.
Last updated: July 10, 2026.
We review and update this page quarterly as new data is published.