When a company uses an AI model through an API, it is not billed by the hour, by the seat or by the question. It is billed by the token. Knowing what a token is, and the handful of rules that sit around it, explains most of what an inference bill contains.
What a token is
An AI model does not read words. It reads tokens: short pieces of text, often a whole common word, sometimes part of a longer one, a number or a punctuation mark. Before a request reaches the model, a tokenizer splits the text into these pieces, and every piece is counted.
For English text, a useful rule of thumb is that one token is about four characters, or about three-quarters of a word. A 750-word document comes to roughly 1,000 tokens. The ratio differs in other languages, and different models split the same text in different ways, so the same request can use a different number of tokens on two different models.
Images, audio and files are converted into tokens too, at rates each provider publishes.
Why conversations cost more as they go on
Models do not remember earlier messages on their own. In a chat, the application sends the whole conversation so far with every new message, so each turn costs more input than the one before.
Take a conversation in which each question and each answer is about 200 tokens, on top of 1,000 tokens of standing instructions. The first turn sends 1,200 tokens. The tenth sends the instructions, the nine earlier questions and answers and the new question: about 4,800 tokens. Over ten turns, the conversation sends about 30,000 input tokens to produce 2,000 tokens of answers.
This is one reason caching matters so much in practice: most of each request repeats what was sent before. It is also why the context window, the most a model can handle in a single request, shapes both what an application can do and what it costs.
The price per million tokens
Providers publish their prices in US dollars per million tokens. The unit is large because single requests are small: a short question and answer may use a few hundred tokens, while an application serving thousands of users can use billions in a month.
Almost every rate card has at least two prices for each model:
- Input tokens. Everything sent to the model: the instructions, the question, the conversation so far and any attached documents.
- Output tokens. Everything the model writes back.
Output is usually priced several times higher than input. A model can read a prompt all at once, in parallel, but it writes its answer one token at a time, and each new token takes a full pass through the model. Writing ties up the hardware for longer, so it costs more.
A worked example
Take a model priced, for illustration, at $2 per million input tokens and $8 per million output tokens. A customer-support request sends 3,000 tokens of instructions, customer history and question, and receives a 500-token answer:
- Input. 3,000 tokens at $2 per million is $0.006.
- Output. 500 tokens at $8 per million is $0.004.
That is one cent per request. At 100,000 requests a day it comes to $1,000 a day, or about $30,000 a month. The prices are invented, but the shape is typical: in many applications most of the tokens are input, while output carries the higher price.
What changes the bill
The headline rates are only the starting point. Several common features change what a request actually costs.
- Cached input. Many applications send the same long instructions or documents with every request. Providers can store that repeated part and charge much less each time it is reused; at some, a cached read costs a tenth of the normal input price or less.
- Batch processing. Work that does not need an immediate answer, such as sorting a backlog of documents, can be sent as a batch. Several large providers offer about half off for batches that complete within 24 hours.
- Reasoning. Reasoning models think before they answer, and that thinking is generated text. It is generally billed as output, even when it is not shown, so the same visible answer can cost much more from a reasoning model than from one that answers directly.
- Long requests. Some models charge a higher rate when a single request is very long.
- Speed and capacity. Some providers charge more for faster or guaranteed service, and less for flexible service that can wait.
Agents multiply the calls
A simple application makes one model call per request. An agent, which plans a task, calls tools such as search or a database, reads the results and decides what to do next, may make dozens. Each call sends the instructions, the task so far and the latest results, and each produces output, often including reasoning.
So the cost of a finished task can be many times the cost of a single call. For agents, cost per task, rather than price per token, is the number that matters, and it depends as much on how the agent is designed as on the rate card.
Why the rate card is not the whole story
Two models with the same prices can produce very different bills. One may need more tokens to represent the same text. Another may answer in fewer words, or get a task right without retries. A cheap model that succeeds half the time can cost more per finished task than an expensive one that succeeds the first time.
The reverse also holds. As models improve, a task that once needed a large model can often be done by a smaller one, and the cost of getting it done falls faster than any single rate card suggests.
This is why comparing prices across models, or following how the price of inference changes over time, is harder than reading a list. Rates change often, new models arrive with new prices, and the capability behind each price keeps moving. Measuring price at constant capability looks at that problem in detail.
A rate card tells you what one model costs today. A benchmark tells you what inference costs, and how that is changing.
Paying per token or renting the hardware
Paying per token is not the only option. Companies can also run open-weight models on GPUs they rent or own, paying for the hardware by the hour instead of for each token.
Which is cheaper depends mostly on utilization. A GPU that is busy around the clock produces tokens at a low cost each. One that sits idle half the time doubles the cost of every token it does produce. Per-token pricing leaves that utilization risk with the provider, who can spread it across many customers; running models yourself keeps it, along with the engineering work. How GPU compute is priced covers the hardware side.
What to measure
Teams that manage inference costs well tend to track a few numbers beyond the monthly bill.
- Cost per unit of work. The cost of a resolved ticket, a reviewed document or a completed task, which captures both price and efficiency.
- Input and output split. Which side of the rate card drives the bill, and so which savings matter most.
- Cache hit rate. The share of input tokens served from cache at the lower rate.
- Model mix. How much of the work runs on which models, and whether simpler tasks could move to smaller ones.
Following the price over time
For a team that only needs this month's bill, the rate card is enough. For anyone who budgets a year ahead, signs a long-term contract or wants to manage the risk of prices changing, the question becomes what the market price of inference is, and how it moves.
That is the question a benchmark answers. CICI, the Carry Inference Cost Index, measures the price of AI inference at constant capability, so every move reflects a change in the cost of intelligence itself. CRTI, the Carry Realized Token Index, measures what the market actually pays, in US dollars per million tokens. Both are published daily on Carry Indices.
The prices in the example are illustrative and do not refer to any provider. Caching, batch processing and reasoning are described in general terms; each provider's own price page sets its terms.