Explain Like Chai · 9 min read
Why AI Is So Expensive to Run: Models, Chips and Data Centres Explained
Frontier AI feels cheap at the chat box and expensive behind the scenes. Here is a plain-language map of models, chips, power, and data centres, plus what you can control on your bill.
"Key Takeaways"
- AI is expensive because training and serving frontier models needs specialised chips, huge clusters, and data centres that draw serious electricity.
- The International Energy Agency reports global data centre electricity demand grew quickly in 2025, with AI-focused facilities growing even faster. Lawrence Berkeley National Laboratory publishes scenario estimates for US data centre use toward 2030.
- Microsoft Research estimated roughly 0.3 watt-hours per query as a median for frontier-scale serving under stated assumptions. Versions of that work differ slightly on the exact median; treat the figure as approximate. Extended test-time scaling can raise energy use by roughly an order of magnitude under their scenarios.
- Your API receipt is mostly tokens and model tier, not the full infrastructure bill. You control model choice, prompt length, and when a frontier model is worth it.
Summarize this article with
When a chat reply arrives in two seconds, it is easy to assume AI is mostly software. The experience is light. The system behind it is not.
Running today's large models is a factory job. You need accelerators built for matrix math at scale, racks wired into clusters, cooling that can handle heat density ordinary servers never see, and staff who keep all of it alive while traffic spikes at random hours.
That is why labs talk about billions for training runs and why providers charge per million tokens even when the consumer app feels bundled. This piece separates the stack into parts you can picture, cites published estimates where they exist, and ends with levers you actually hold.
Three layers: model, chip, building
Start with the model. A frontier system is not one file on a laptop. It is weights and infrastructure spread across many machines so a single request can use a slice of a much larger pool.
Next come the chips. Training and serving at scale lean on GPUs and other AI accelerators. Supply, packaging, networking, and how fully a cluster is utilised all feed into cost. When demand outruns supply, list prices and cloud rates move.
Finally the data centre. Rows of servers need power, backup power, cooling, fibre, physical security, and compliance work. AI-heavy sites push power density harder than a typical corporate rack from a decade ago.
Product teams pay for all three at once. Your subscription or API bill is the retail layer on top.
Where the money goes (simplified)
| Layer | What it does | Why it shows up on your bill |
|---|---|---|
| Model | Stores and runs a very large neural network | Bigger models and longer outputs mean more compute per request |
| Chips and networking | Matrix math fast enough to serve many users | Premium tiers and 'thinking' modes burn more accelerator time |
| Data centre | Power, cooling, space, operations | Providers recover infrastructure through pricing, caps, and plan gates |
| Your usage | Tokens in, tokens out, tool calls, agents | This is the line item you can steer week to week |
Training is a one-time earthquake; serving is the monthly rent
Training a frontier-class model is a capital event. You rent or own a large cluster for weeks or months, run optimisers over trillions of tokens, checkpoint often, and hope the run does not fail late.
Independent trackers publish training energy and cost estimates because labs rarely share full bills. Epoch AI's public estimate for Grok-4 training is on the order of roughly 310 gigawatt-hours and a median cost around $490 million. Treat that as an outside estimate, not an official xAI disclosure. It is useful for scale, not for quoting as fact in a board deck.
Serving is different. It is continuous. Every query, every agent loop, every long document you paste in adds marginal electricity and chip time. That is why usage meters exist even when the marketing says 'unlimited with fair use.'
Estimates are maps, not receipts
When a research group or analyst publishes a training cost or energy number, read the assumptions.
Hardware generation, utilisation, energy mix, and whether the number includes networking and storage all change the story. Round numbers are for orientation.
Power and the IEA view
Data centres were already growing before the AI boom. AI workloads added a sharper wedge.
In its Key Questions on Energy and AI work, the International Energy Agency reported that global data centre electricity demand grew by about 17% in 2025, with AI-focused facilities surging by about 50% in the same year. Those are IEA figures from its reporting, not a guess from this desk.
The same IEA framing looks ahead toward 2030 with scenario context: demand can keep climbing as more inference runs 24/7 and more regions build AI-ready capacity. Exact future terawatt-hours depend on policy, efficiency gains, and how concentrated AI stays in a few large clouds.
Lawrence Berkeley National Laboratory's US data centre energy update is another anchor. Its scenarios point to on the order of about 649 terawatt-hours of US data centre electricity use in 2030, with a range across cases. Cite that as an LBNL scenario estimate, not a promise etched in stone.
What one query costs in energy (with caveats)
Microsoft Research published work in September 2025 that estimated energy per query for frontier-scale serving under explicit assumptions about model size, hardware, batching, and utilisation.
Across versions of that research, the median sits around 0.3 watt-hours per query (one public write-up cites about 0.34; a later arXiv revision cites about 0.31). That is small in household terms, but it multiplies fast at global traffic. Use the rounded figure, not a single decimal, when you repeat the claim.
The same line of research looked at test-time scaling: when a model uses far more tokens at inference (on the order of fifteen times a typical query in their scenario), the median energy estimate rose by roughly an order of magnitude. Exact point estimates differ slightly by revision. Again, that is under their stated scenario, not a universal constant for every app you use.
The practical lesson is not to memorise watt-hours. It is to see why 'thinking' models, agent loops, and long outputs are priced differently from a short reply.
Why 'thinking' and agents feel pricier
- They generate more tokens before you see an answer, which raises output-side metering on APIs.
- They may call tools repeatedly, each round trip adding input and output tokens.
- They keep context hot longer on the provider side, which competes with other tenants for the same cluster.
- They tempt you to use a frontier model for tasks a smaller model would handle fine.
API list prices are not the whole infrastructure story
When OpenAI, Google, Anthropic, or others publish dollars per million tokens, you are seeing a retail unit designed for developers. It reflects model tier, competition, and margin. It does not itemise the diesel backup generator or the underwater fibre lease.
Enterprise contracts add another layer: committed spend, private capacity, support, and safety features. Consumers see subscriptions with soft caps. Neither format exposes the full data centre ledger.
That separation is healthy for beginners. You do not need to model a substation to choose a model. You do need to know that someone pays for the substation, and that cost flows into which features stay free and which get gated.
Chips: why everyone names the same few vendors
AI training at scale still clusters around a small set of accelerator families. NVIDIA's data centre GPUs dominate headlines, but the full stack includes custom silicon from large clouds, networking chips, and memory bandwidth that can bottleneck before raw FLOPs do.
Lead times matter. A lab can announce a model months before it has enough stable capacity to serve the world cheaply. That is one reason new tiers roll out gradually and why 'available' and 'affordable at your volume' are not the same.
For you, the chip war shows up as plan changes, region availability, and occasional queueing during peak launches.
What you can control this week
You cannot pick the grid mix for a hyperscaler campus. You can pick the model, the prompt shape, and the workflow.
Default to the smallest model that passes your quality bar. Escalate to frontier only for tasks that need long reasoning, fragile code, or multimodal depth you verified.
Shorten the context you send. Large pasted PDFs are convenient and expensive. Summarise first with a cheaper pass if the task allows.
Turn off agent autonomy for chores a chat completion can do. Loops are powerful and token-hungry.
Measure one real workflow before and after a model swap. Tokens per successful task beats vibes.
Practical cost levers
| Lever | Example | Trade-off |
|---|---|---|
| Model tier | Flash or mini class for drafts; frontier for final pass | May need a two-step habit |
| Output length | Ask for bullet limits and structured sections | Less prose, more editing by you |
| Context window | Link or summarise instead of dumping full files | Requires upfront organisation |
| Thinking mode | Use high reasoning only on failed medium runs | Slower, higher meter |
| When not to use AI | Skip the model for facts you can look up in ten seconds | Saves money and hallucination risk |
Cooling, fibre, and why the building is not generic
Data centres are not oversized closets. AI racks run hot. Cooling design, liquid loops in some sites, and the cost of moving heat out of the hall show up in lease rates and cloud margins even if you never see a chiller on a pricing page.
Networking is the other quiet bill. A model shard on one machine is useless if tokens cannot move between GPUs fast enough. That is why hyperscalers talk about custom fabrics and why a startup renting spot capacity still inherits those economics indirectly.
Geography matters too. Power price, grid carbon intensity, and local permitting for new builds all influence where training runs land and which regions get new capacity first. Users feel that as latency and as which model tier launches in which country.
A team habit that matches the physics
Pick a default model for the week and name it in your team chat. When someone reaches for the frontier tier, they add one line on why. That single habit cuts surprise invoices more than debating watt-hours in abstract.
When a vendor ships a fast, cheaper model class, test it on real tasks before assuming the old default still earns its price. Efficiency gains from better models and better chips eventually show up in list prices, but not evenly or all at once.
If you operate in price-sensitive markets, read regional pricing explainers alongside global launch posts. The same model name can sit on different plans, credits, and tax treatment depending on where you subscribe.
Frequently Asked Questions
Is AI electricity use mostly from training or from everyday chat?
Both matter, but everyday inference grows with users. Training spikes when a new frontier run finishes. Serving accumulates quietly with every product embed and agent loop. Policy discussions often focus on training because the numbers are dramatic; operators feel serving every day.
Should I trust watt-hour per query estimates?
Use them comparatively, not as your home utility bill. Published medians depend on hardware generation, batching, model size, and whether the query triggered long internal reasoning. They still teach that longer, heavier inference costs more energy than a tiny reply.
Why do thinking models cost more on APIs?
Providers meter tokens. Thinking models often emit many internal or visible tokens before the final answer. That shows up as higher output usage even when the UI looks like one message.
Does a cheaper subscription mean the model is smaller?
Often, yes, or it means softer rate limits and fewer premium features. Read the plan table. Compare model names and caps, not just the monthly price.
Where can I learn more about context and tokens?
Start with a plain explainer on context windows and token counting. Once you see how input size drives the meter, pricing pages make more sense.
The Chai Takeaway
AI feels expensive because it is physical work disguised as magic. Labs pay for chips and power at warehouse scale. You pay for tokens and tiers at human scale. Choose models on purpose, shorten prompts, and reserve frontier tools for work that earns the bill.
Explore More Explain Like Chai