Will rising compute costs or the amount of computing resources — such as time, memory and energy required to execute a specific algorithm or process — trigger the end of the four-year-long AI boom and finally burst the burgeoning AI bubble? And what are all these AI tokens, and why are some of the largest and most profitable tech companies in the world clamping down on the use of these tokens?
Let me explain what’s going on: In the aftermath of OpenAI’s unveiling of its generative AI chatbot ChatGPT in November 2022, the focus was initially on the training of frontier AI models, which required a lot of graphics processing units (GPUs) such as Nvidia’s Hopper, Blackwell and the latest Vera Rubin chips. More than US$450 billion ($580.6 billion) has been spent on GPUs over the past four years. Global AI-related capex for this year is estimated at around US$750 billion. About US$150 billion will be spent on GPUs alone, or a fifth of the total AI capex.
Training AI models may be expensive, but it is just a one-time cost. But running the model, whether it is Google’s Gemini or Anthropic’s Claude, each time you or I ask a question, generate an image or automate a workflow by using an AI agent to pay a utility bill or schedule an appointment, is an ongoing expense that keeps on growing the more we use all these fancy AI models and AI agents.
Global AI capex is forecast to soar to US$1.2 trillion next year. The rising cost of compute is increasingly difficult to justify because soaring capex has yet to boost productivity or profits. Not surprisingly, large companies like Microsoft, Walmart, Uber and Accenture have begun rationing access to AI tokens to cut costs and are instructing staff to use AI tools more efficiently. As compute costs have risen, AI platforms have moved from flat monthly pricing to token-based or usage-based pricing. Indeed, heavy usage can easily outpace the subscription fee you pay.
See also: Xi promises AI for all in debut at China’s top tech summit
A “token” is the basic unit of data processed by AI models to read and write. When you type a message to Gemini or Claude, the text is broken up into small chunks. A token is usually a word, part of a word or a punctuation mark. “Understanding” might split into “under” and “standing”, while a common word like “the” is one token.
As a rough rule, one token is about three-quarters of an English word, so 1,000 tokens is roughly 750 words. The AI model works token by token. Here’s how: Let’s say you ask Claude about the best place to eat seafood in your favourite city. It reads your question as a sequence of tokens and generates its response by predicting one token at a time. Every single token, in and out, requires a pass through the model’s computation.
Tokens have emerged as the fundamental economic unit of the AI industry. Indeed, tokens are how AI is priced. Application Programming Interface (API) access to AI models is billed per million tokens. Think of tokens as the product that is being sold. So, the cost of running AI comes down to the cost per token. Over the past year, token consumption has exploded. New “reasoning” models don’t just answer your queries. They think step by step before responding, generating thousands of hidden tokens of internal reasoning for a single question. AI agents that browse the web, write code or undertake multi-step tasks can burn through millions of tokens per task. Token volumes are growing 14 times annually, propelled by agentic workloads and highly elastic demand.
See also: DeepSeek champions China’s bid to flood world with cheap AI
Each processed token requires GPU compute, electricity and data centre capacity. So, when you read that hundreds of billions of dollars are being spent on AI chips and data centres, you might wonder how many tokens the world needs to process all the queries that would be generated, and at what cost? Nvidia, Google and OpenAI now report token-processing volumes as evidence of AI demand, the way barrels were once used as a measure for oil or kilowatt-hours for electricity. AI tokens, not to be confused with crypto tokens, which are a completely different thing, are becoming something like a commodity input for the AI sector, with their own supply chain including GPU chips, high bandwidth memory (HBM) and power, falling unit costs, and rapidly compounding demand.
The price of tokens is falling fast due to intense competition among AI developers and a huge surge in infrastructure efficiency, which in turn has triggered a fierce price war. It’s a classic Jevons paradox: Cheaper tokens unlock new use cases, which in turn lead to consumption of a lot more tokens. Companies such as Walmart, Uber and Accenture realise that their employees are using far more AI tokens now than they were last month or six months ago.
A recent study by Epoch AI, an AI research institute, reported prices declining between nine times and 900 times per year across benchmarks, with a median of 50 times per year. Indeed, the pace of price declines is still accelerating. When GPT-3 launched nearly six years ago, a certain capability level cost US$60 per million tokens; by March this year, multiple models exceeded that benchmark at US$0.06 per million tokens. And enterprise spending data shows the average cost per million tokens across major providers fell from roughly US$10 to US$2.50 in a single year. “This effectively enables the compute supply to expand, as more tokens can be monetised on the same fleet of xPUs,” Barclays AI analyst Ross Sandler noted in a recent report.
The economics of tokens is changing as AI models complete the same task for a fraction of the cost with far fewer tokens. When selling inference workloads, such as Anthropic tokens through AWS Bedrock, Amazon earns both traditional infrastructure fees as well as a revenue share with Anthropic, which is highly incremental to margins. This type of API workload has taken off in recent months with broader AI adoption and the agentic AI take-off. “We’re also seeing inference margins inflect at the AI lab level,” Sandler says.
Anthropic’s “adjusted” operating margin is now on track to turn positive to 5% in the April-June quarter, up from -13% in the previous quarter. A year ago, Anthropic didn’t even expect to turn a profit until 2028. If an AI lab like Anthropic is now making 70% inference margin on current-generation models, it implies that, given the same revenue per unit of intelligence, previous-generation models were running at around -20% margins, the Barclays analyst says. “In reality, the AI labs are likely passing on some cost savings into better rate limits for users, but this dynamic is likely playing into the better margins we’re seeing at the labs to some extent.”
OpenAI CFO Sarah Friar recently noted that token costs are still falling. “The main input is compute,” she noted. “The good news on compute is that there is a massive deflationary curve on cost.” Cost depreciation from GPT-4 to GPT 5.4 was 97%. With the recently released GPT-5.5, OpenAI passed some of the savings back to customers. It doubled the price on GPT-5.5, but the cost to the customer fell 20% to 30% per token because it’s far more efficient.
Over the next 12 months, supplies of GPUs and HBM are likely to remain tight as demand soars, driven by agentic AI. Agentic workflow can trigger 10 to 20 large language model (LLM) calls to complete a single user task. So even as per-token price declines with efficiency improvements, analysts expect frontier-model pricing to stay firm.
Sink your teeth into in-depth insights from our contributors, and dive into financial and economic trends
Another chokepoint is advanced chip packaging capacity. By mid-2027, foundry giant TSMC is expected to complete the expansion of its Chip-on-Wafer-on-Substrate or CoWoS. Moreover, Samsung would have ramped up its HBM4 production, relieving some pressure on memory capacity. However, significant new HBM capacity is likely to come online only in late 2027 when SK Hynix and Micron plant expansions are complete. Until then, HBM supply will remain tight.
The most problematic bottleneck for AI compute is still power. US data centre construction is being delayed in many states because of power. Of the 16 gigawatts of US data centre capacity that were set to open in 2026, only 5GW is currently under construction. Utility interconnection queues in markets like Northern Virginia, Phoenix and Dallas can take four to seven years. That means a proposed new data centre joining the queue now should not expect utility power before 2029. Chokepoints include high-power transformers that now have a three-to-five-year delivery schedule and switchgear, the equipment connecting a data centre campus to the grid, is sold out through 2028. Research firm Gartner projects 40% of AI data centres in the US are likely to be power-constrained by 2027.
Will rising compute burst the AI bubble or at least temper growth? Rising input costs like higher HBM prices, GPU rentals and cost of power are unlikely to burst the bubble in the foreseeable future, though they are already tempering growth. Memory and storage component costs have surged 90% per quarter over the last two quarters. GPU rentals have risen by 20% to 30% at AWS, while Blackwell rental rates are up about 20% year to date, with server rack component costs swinging as much as 40% in a week. Microsoft recently noted its own storage and memory costs rose more than 2.5 times and expects these could double again late next year. Higher input costs and falling token costs could lead to a margin squeeze. Here’s why: Token prices are falling faster than inference demand can monetise, while hyperscalers double GPU orders despite sluggish revenue growth, and capex keeps accelerating while revenue growth stalls.
More could go wrong for AI Frontier models. “What if the payoff takes longer than consensus assumes?” asks Apollo Global’s Torsten Slok. It’s a pressing question given that token prices are declining, and Chinese models are gaining ground, both in their share of the world’s most-used models and in token usage. Investors are also watching out for any signs of demand destruction. Microsoft recently cancelled most of its direct Claude Code licences after employee AI usage grew so large that “the cost of compute is now far beyond the costs of the employees”. The average enterprise AI budget in the US has gone from US$1.2 million to US$7 million in two years, mostly because companies are aggressively deploying AI. If customers continue capping token usage, revenues for Frontier Labs and Hyperscalers will collapse.
Lastly, the AI buildout, which was initially all paid for by the internal cash of the tech giants, is now being financed with loans collateralised by the chips themselves. A default that floods the market with older GPUs will likely trigger a cascading collapse.
Assif Shameen is a technology and business writer based in North America
