How enterprises manage AI inference costs.

AI inference is becoming a major operating cost. Why it is hard to budget and forecast, and how companies manage it.

Carry Research8 min read

When a company builds AI into a product or a process, it takes on a cost that is easy to start and hard to stop. Every customer conversation, document reviewed or line of code suggested consumes inference, and the bill grows with usage. That cost becomes part of the business, and it stays there.

This article looks at why that cost is becoming a major line on the profit and loss statement, why it is hard to budget and forecast, what its volatility means for listed companies, and the tools companies use to manage it.

Where the cost comes from

Inference costs tend to arrive in three waves. The first is experimentation: teams try models, build prototypes and run tests, and the spend is small and easy to ignore. The second is launch: a feature reaches customers or employees, and usage starts to climb. The third is scale: the feature becomes part of how the business works, and inference becomes a recurring operating cost.

Each wave also tends to use more tokens per task than the last. Production systems add instructions, safety checks, retrieved documents and conversation history to every request. Agents that plan and call tools may make many model calls to finish one job. A task that cost a fraction of a cent in a prototype can cost much more once it is built properly. How AI inference is priced explains how those tokens add up.

A growing line on the P&L

For most companies today, inference is still a modest cost. That is changing quickly. Gartner expects worldwide end-user spending on foundation generative AI models to roughly double in 2026, to about $23 billion, and notes that usage-based pricing makes that spending harder to predict.

As AI moves from pilots into products and everyday work, the bill moves with it: from an IT experiment to an operating cost that sits alongside cloud hosting, salaries and rent. For software companies that build AI into what they sell, inference lands in the cost of revenue and moves gross margin directly. Elsewhere it sits in operating expenses, where it competes with every other use of the budget. Over the next few years, for many companies, it is likely to become one of the larger lines on the P&L.

Unit costs fall, total costs rise

Finance teams are often told that AI is getting cheaper, and for any given capability that is true. The price of a fixed level of capability has generally fallen, often quickly. Yet many companies see their total inference bill rise year after year.

Illustration: the cost of each task falls from 100 to about 22 over four years, while total spend rises from 100 to about 338. 0100200300Year 1Year 2Year 3Year 4 Total spend Cost per task
Illustration, year 1 = 100. If the cost of each task falls 40% a year while usage grows 2.5 times a year, total spend still rises 50% a year.

Both statements hold at once because usage grows faster than unit prices fall. More features use AI, more customers and employees use those features, and each task uses more tokens as products become more capable and agents take on longer jobs. In the illustration above, the cost of each task falls by almost four-fifths over three years, yet total spend more than triples.

Falling unit costs are good news, but they do not make the total predictable. Both lines move, neither moves smoothly, and the company controls only part of either.

Why switching takes time

In principle, a company can move to a cheaper model the day one appears. In practice, switching takes time, for good reasons.

  • Prompts and workflows. They are tuned to a model's behavior, and need retuning when the model changes.
  • Quality. It has to be re-tested against the company's own standards before a new model reaches customers.
  • Reviews. Security, privacy and compliance reviews often approve a specific vendor and data location.
  • Contracts. Committed-spend agreements tie usage to a provider for a period.

Many companies take weeks or months to move a production workload to a new model, and some move only once or twice a year. So the cost of inference is carried forward. A company that launches an AI feature this year is, in effect, committing to buy inference for as long as the feature runs.

A company that launches an AI feature this year is, in effect, committing to buy inference for as long as the feature runs.

What makes it hard to budget

Inference spending is the product of two things that both move: how much is used and what each unit costs. Usage often grows faster than planned when a feature succeeds. Unit prices change as providers release models, reprice existing ones and retire older versions.

For a finance team, that combination is uncomfortable. The direction of travel for the price of a given capability has generally been downward, but the path is uneven, and a company's own costs depend on which models it uses and when it moves. A budget built on today's rate card can be wrong in either direction within a quarter.

Consider a company that budgets $2 million for inference next year, based on current prices and expected usage. If usage runs 40% ahead of plan and prices hold, it spends $2.8 million. If usage is on plan but its main model is replaced by a dearer successor and switching takes six months, the overrun comes from price instead. If a cheaper model arrives early and the company switches quickly, it may spend well under budget. These figures are invented, but each of these paths is common, and the finance team needs to tell them apart.

Why volatility matters for listed companies

For a listed company, the question is not only how large a cost is but how predictable it is. Public companies give investors guidance on revenue, margins and earnings for the coming quarter or year. A cost that can swing sharply within a quarter makes that guidance harder to give and easier to miss.

Investors and executives both care about this. In a well-known survey of more than 400 financial executives, 78% said they would give up some economic value to report smoother earnings, because missing targets or reporting volatile results is seen as making earnings less predictable and weighing on the share price.

The effect reaches the cost of capital. Research on US companies has found that firms with less favorable earnings attributes, including less smooth earnings, tend to have a higher cost of equity. Hedging material input costs has been associated with higher valuations: among US airlines, fuel hedging has been linked to firm values roughly 5% to 10% higher.

Inference is not yet large enough to move the share price of most companies. For those building AI into what they sell, it may soon be. Managing its volatility before it becomes material is easier than explaining it to investors afterwards.

Who inside the company carries it

Inference exposure often has no single owner. Engineering chooses models and controls how many tokens each task uses. Product decides which features use AI and how widely they roll out. Procurement negotiates with providers. Finance sets the budget and explains the variance. Treasury, where a company has one, manages financial risks such as currency and interest rates, but rarely inference.

As the cost grows, companies tend to give it an owner, set a budget per feature or per unit of work, and report on it like any other material cost. That is usually the point at which the question of managing the price and budget, not just the usage, comes up.

Ways to manage the exposure

Companies already use several tools, and they work best together.

  • Supplier agreements. Committed-use discounts and fixed-price arrangements reduce unit costs and add predictability, at the price of flexibility.
  • Engineering choices. Routing simpler tasks to smaller models, caching repeated input, batching work that can wait and controlling output length reduce volume and cost.
  • Measurement. Tracking cost per unit of work, not just total spend, shows whether savings come from better prices or less usage.
  • Hedging. Index-linked contracts can offset movements in the market price of inference, so that a company's budget is protected even when its own supplier terms cannot be changed quickly.

Each tool addresses a different part of the exposure. Supplier agreements fix the price with one provider. Engineering reduces how much is needed. Measurement shows what is happening. Hedging addresses the market price itself, independently of which provider the company uses or when it switches. Designing a hedge that fits walks through how to set one up.

Treating inference like other operating costs

Energy, freight and raw materials were once costs that companies simply absorbed. As markets and benchmarks developed, they became costs that could be planned, contracted and hedged. Airlines manage fuel, manufacturers manage metals and utilities manage power through a mix of contracts, engineering and market instruments, with a benchmark at the center of each.

Inference is on the same path. The companies that treat it as a managed exposure, rather than a line item that surprises them, will be able to build on AI with more confidence.

Choosing the right reference

Hedging needs a reference that tracks the exposure. Carry publishes two. CICI tracks the price of inference at constant capability, which fits companies that need a defined level of capability and will use whichever model provides it best. CRTI tracks the realized cost the market pays per million tokens, which fits companies whose spending follows prevailing market prices. Comparing a company's own cost history with each index is a good way to choose.

Carry's role is to provide the reference prices and the market that make that possible: trusted benchmarks for the cost of intelligence, and a place to manage the exposure to them.

This article is for information only and is not investment advice. Derivatives involve substantial risk of loss and are not suitable for all investors. The budget figures and the chart are hypothetical illustrations. Sources: Gartner, "Gartner Forecasts Worldwide AI Platforms and Models Market to Grow 63% in 2026" (20 July 2026); J. Graham, C. Harvey and S. Rajgopal, "The Economic Implications of Corporate Financial Reporting" (2005); J. Francis, R. LaFond, P. Olsson and K. Schipper, "Costs of Equity and Earnings Attributes" (2004); D. Carter, D. Rogers and B. Simkins, "Does Hedging Affect Firm Value? Evidence from the US Airline Industry" (2006).

Newsletter

New research, in your inbox.