Industry-wide initiative to reduce AI inference costs through model optimization, quantization, and edge-device deployment gains momentum.
Global companies are waging a "cost diet war" to reduce soaring artificial intelligence costs. Competition to lower inference costs—the expenses incurred each time AI is used—has intensified beyond mere usage reduction, with companies now investing heavily in hardware and software optimization.
According to a recent report from The Information, OpenAI engineers have cut existing inference costs to less than half through a new optimization technique. Though OpenAI has not disclosed the full methodology, the number of graphics processing units required for inference has dropped significantly to around 200 using this approach. The test, conducted on non-registered ChatGPT accounts handling low-usage, simple tasks, demonstrated that software-based improvements can substantially enhance inference efficiency and reduce costs.
**The Cost Pressure**
AI companies face mounting pressure to reduce inference costs as usage explodes. AI agents, which autonomously make judgments and execute tasks, dramatically increase token consumption—the fundamental unit of data processing in AI systems. Goldman Sachs Research projects that global monthly token consumption from AI agents will balloon from 5.6 trillion this year to 11.77 quadrillion by 2030, a twenty-one-fold increase. "We are starting to hear that companies are facing existential crises due to AI inference costs," said JR Storment, director of the FinOps Foundation under the Linux Foundation.
**Software Solutions**
AI companies are deploying multiple software-based strategies. OpenAI's recent optimization combines quantization (reducing computational precision without sacrificing performance), fixed-batch processing to lower computational load, and model routing, which automatically directs simple queries to cheaper models. DeepSeek has developed "DeepSeek Sparse Attention (DSA)," which uses a separate context management tool called an "indexer" to process only task-relevant tokens rather than comparing all token pairs. DeepSeek reports this maintains prior performance while cutting inference costs by more than half.
**Hardware Race**
Hardware innovation is accelerating in parallel. OpenAI unveiled its first in-house inference chip, "Jalapeño," co-developed with Broadcom. By incorporating AI into the design process, the team delivered final blueprints to the foundry in nine months—a task that typically requires two to three years. OpenAI states the chip "significantly outperforms existing top-performing AI accelerators in terms of performance per watt."
Major tech companies are racing to develop dedicated inference chips. Google has designed separate chips for training and inference since its eighth-generation TPU. Microsoft introduced the inference-dedicated "Maia 200" in January. NVIDIA acquired Groq, a startup specializing in inference-optimized language processing units, last year, integrating its low-latency capabilities into NVIDIA's AI architecture.
Domestic players Rebellions and FuriosaAI are targeting the market with inference-specialized chips offering higher performance and lower power consumption than NVIDIA GPUs, reducing total usage costs. Rebellions has supplied its second-generation chip "Rebel 100" to Saudi Aramco, while Furiosa has begun mass production of its "Renegade" chip.
**The Paradox**
AI inference unit costs are declining. Yet total inference costs continue rising as AI usage accelerates—a manifestation of Jevons' paradox, where improved efficiency drives increased consumption. OpenAI's annual inference expenditure is projected to rise from $8.4 billion in 2025 to $14.1 billion in 2026, according to market research firm Sacra. One tech industry source observed: "The surging inference demand will ultimately drive the need for memory essential to powering AI."