Despite declining token costs, businesses continue to see their AI tab climb.
The price of a token — the chunks of text that large language models (LLMs) use to generate responses to prompts and that AI companies use to bill their customers — has collapsed, noted Jeff Barrington, managing director of Windsor Drake, an investment banking and M&A advisory firm in Toronto.
“They’re down about 98% since early 2024, yet enterprise AI bills keep rising,” he told TechNewsWorld. “Cheaper compute just gets consumed faster, and 73% of enterprises blew past their original AI budgets last year.”
“Uber handed 5,000 engineers a coding assistant in December and burned its entire annual AI budget by April,” he added.
One reason for the growing consumption: Companies are moving from pilots to production, and from chatbots to agents. “An agent does not answer one prompt,” Barrington explained. “It fans that prompt out into planning, tool calls, retrieval and verification, which can turn a single instruction into hundreds of calls and five to 30 times the tokens.”
“Feeding in messy, unstructured data makes it worse, because the model burns tokens just parsing the mess,” he said.
“Clean, structured documents cut the bill in reverse — less to interpret, fewer tokens,” he continued. “Inference is now about 85% of enterprise AI budgets, so that is where the money goes.”
Token Hungry Agentic AI
AI flipped software from a fixed-cost license to a metered utility, asserted Jon Knisley, head of AI enablement and value at Abbyy, a global intelligent automation company.
“Every prompt, retrieval, and loop is on the meter, so spend rises with adoption,” he told TechNewsWorld. “But the real jump is architecture.”
Moving from a chatbot to an agent raises token consumption per task by one to two orders of magnitude,” he said.
He added that a large portion of a business’s AI bill is waste. “In agentic workloads, actual output is often only 5% to 15% of the tokens consumed,” he explained. “The rest is context overhead.”
“PDFs are a perfect example,” he noted. “They were designed for rendering, not understanding, so every PDF that enters a pipeline forces the model to burn tokens reconstructing layout before it can extract the facts. You’re paying a reasoning model to do a formatting job.”
“Token consumption is also accelerating because organizations are expanding the number of workflows where AI is delivering value. As AI becomes embedded across more business functions and everyday tasks, token usage naturally increases because more interactions flow through these models,” explained Edward Rothschild, CTO of Adronite, an AI coding and software intelligence company in Seattle.
He added that newer generations of AI models often consume more tokens than their predecessors, even when completing tasks that older models were already capable of handling.
“In many cases, this is because they’re designed to reason more deeply, taking additional steps, evaluating more context, and performing more internal processing before producing an answer,” he told TechNewsWorld. That extra reasoning can improve quality, but it also increases token consumption.
Beyond Token Costs
The big message here is that the consumption of AI tokens is exploding as companies move beyond simple chatbots into copilots — autonomous agents that repeatedly read documents, retrieve data, call tools, and communicate with other systems, observed Mark N. Vena, president and principal analyst at SmartTech Research, a technology advisory firm in Las Vegas.
“Companies are also feeding models larger context windows and more unstructured material, often sending the same background information with every request and paying to process it again,” he told TechNewsWorld.
However, he pointed out that the model invoice is just the tip of the iceberg, as companies also need to clean and organize data, integrate legacy systems, assess output quality, monitor usage, enforce security policies and keep multiple AI platforms working together.
“Poorly defined data and inconsistent business terminology cause semantic debt that forces models to process more information and employees to spend more time correcting answers, resolving conflicts and maintaining complicated retrieval pipelines,” he added.
Bihag Karnani, a senior product manager at Google, explained that organizations may have invested months, if not years, finding out that their internal data is in a lot worse shape than they initially thought.
“Their internal data may be structurally inconsistent throughout their system, and it would take a substantial amount of time to clean and transform this data so that it is usable as context,” he told TechNewsWorld. “Every single hour spent by a team of engineers working to make data LLM-ready is an AI cost that doesn’t appear on your model spend invoice.”
“Semantic debt can also be a large expense,” he added. “When different departments in an organization define different terms, taxonomy, and definitions for the same concept, the model is receiving conflicting or unclear context about the information, which negatively impacts output quality,” he said.
“Fixing semantic debt requires organizational alignment work, not technical work,” he continued, “and is typically under-estimated.”
AI Exposes Organizational Complexity
Rob Enderle, president and principal analyst of the Enderle Group, an advisory services firm in Bend, Ore., noted that a critical oversight in enterprise AI planning is treating unstructured data as a plug-and-play input for LLMs.
“Indexing massive, unstructured file repositories directly into vector databases for RAG leads to context bloating, where models continuously ingest low-value, duplicate, or irrelevant context,” he told TechNewsWorld.
“To optimize long-term economics, organizations should implement a pre-ingestion pipeline that cleans, summarizes, and structures raw data before it reaches the model,” he advised. “Investing upfront in deterministic data parsing and semantic layering yields a compound return. It permanently suppresses token consumption per query, reduces retrieval noise, and substantially increases the accuracy of model outputs.”
Parker Johnston, agentic field CTO at Insight Enterprises, a solutions integrator in Tempe, Ariz., agreed that organizations should do their discovery and data readiness work before they scale because the ceiling on what agents can do is set by the ground they stand on.
“Understand what data you have, who can access it, and which systems deserve to be connected,” he told TechNewsWorld. “Otherwise, you end up paying premium rates for an incredibly capable agent to wade through data that is duplicated, poorly governed, or irrelevant.”
“The real opportunity is not just cheaper tokens,” he continued. “It is building an environment where you can keep handing AI more of the load and trust that the capability, and the spend, are pointed at something that matters.”
LLM costs often mirror organizational complexity, added Dustin Engel, co-founder and principal consultant at Elegant Disruption, a strategy and AI-focused consulting firm in Philadelphia.
“Unclear ownership, inconsistent processes, scattered knowledge and years of unstructured content all show up quickly once AI is inside the workflow,” he told TechNewsWorld. “The model bill is the receipt.”
“AI can make messy information more useful, but only if the company also invests in structure, governance and better operating design,” he warned. “Otherwise, you are paying a model to work around problems that should have been solved upstream.”
“Treat information as infrastructure and measure AI by the value it creates, not the volume of usage it generates,” he said.
Read the full article here
