Token pricing models explained: input, output and cached
Billing for API access to large language models (LLMs) is almost without exception based on units processed: tokens. Although this looks simpler than renting fixed infrastructure, in practice it often causes confusion about how the invoice is built up. Cost per call is not determined by a single fixed rate but by a combination of factors: input tokens, output tokens, cached status and additional forms of processing.
To keep a grip on spending in software development and integrations, a thorough understanding of these pricing structures is necessary. In this article we cover how token prices are constructed, why input and output are valued differently, how context caching works and which hidden variables affect the total bill.
The basics: separate rates for input and output
Providers of AI models apply separate rates for processing the question (input or prompt) and generating the answer (output or completion). In almost every case the output rate is considerably higher than the input rate.
This price difference follows directly from the underlying hardware architecture and processing technique of transformer models:
- Input processing (parallel): When a prompt is sent to the model, the graphics card (GPU) can process all input tokens simultaneously. This operation makes optimal use of the hardware's parallel compute power. Processing time per token is therefore low, which translates into a lower rate.
- Output generation (autoregressive): Generating an answer happens step by step. The model computes one token, adds it to the context, and then computes the next token. This sequential process requires the entire model matrix to be pulled through GPU memory again at every step. That causes a memory bandwidth bottleneck and demands more compute per generated token, which explains the higher output rate.
Core rule: Generating a shorter answer saves more money directly than shortening the question by a comparable number of tokens.
Cached input: how context caching lowers costs
In many applications — asking questions about a fixed document set, or using extensive system prompts — a large part of the input text is repeated across successive requests. To keep the model from reprocessing this constant text on every API call, vendors offer context caching .
What gets cheaper?
When part of the input is successfully read from the cache (a cache hit), the server does not have to repeat the compute-intensive pre-processing. Vendors charge a reduced input rate for this. In practice, this discounted rate for cached input tokens lies considerably below the standard input rate.
Conditions for context caching
Caching does not work automatically on arbitrary texts. A number of strict conditions apply:
- Stable prefix: The cached text has to sit at the exact start of the prompt. As soon as a single character at the start changes, the cache loses its value for the rest of the text.
- Minimum length: Caching requires a minimum input size. For very short prompts the administrative overhead of the cache does not outweigh the gain.
- Time to live (TTL): Caches are held in server memory for a limited time. If a cache goes untouched for a certain period, it is cleared.
Why caching carries costs of its own
Holding context in the fast memory (VRAM) of a GPU cluster is expensive. Some platforms therefore charge an additional rate for storing the cache per hour or per minute that it stays active. For applications with infrequent requests, storing a cache can work out more expensive than simply reprocessing the standard input tokens.
Other items on the API bill
Besides standard input and output tokens, additional forms of processing can appear on the invoice and affect the cost picture.
| Item | Description | Effect on cost |
|---|---|---|
| Reasoning tokens | Internal intermediate steps in specialized models. | Billed as output tokens, which raises the total price per request. |
| Tool calls & function calls | Parsed JSON structures for external integrations. | Count as input and output tokens, including the schemas sent along. |
| Embeddings | Vector representations of text for search systems. | Fixed rate per input token processed. |
| Multimodal input | Images, documents or audio. | Converted into an equivalent number of text tokens based on resolution or duration. |
Vigilance is warranted in particular with reasoning models . These models generate 'trains of thought' invisible to the user in order to reach an accurate answer. All these internal tokens fall under the more expensive output rate, so the final price of a single call can climb sharply even when the eventual answer is very short.
From tokens to total cost per request and per user
The basic formula for computing the cost of one API request is as follows:
Kosten = (Ongecachte invoertokens * Tarief_Invoer)
+ (Gecachte invoertokens * Tarief_Cached_Invoer)
+ (Uitvoertokens * Tarief_Uitvoer)
+ Eventuele opslagkosten_cacheIn practice, actual monthly invoices often come out higher than an initial estimate based on this formula. That is because estimates overlook a series of structural factors:
- System prompts on every request: An instruction, rule set or role definition has to be sent along with every individual request if the API is stateless.
- Growing conversation history: In a chat application, every successive question resends the full preceding conversation. The tenth message in a session therefore costs a multiple of the first. To understand the impact of this growth, insight into the context window is essential.
- Retries and error handling: Automatic retries on network errors, invalid JSON output or unsatisfactory answers generate extra token consumption without being directly visible in the eventual user experience.
Why a lower token price is not always cheaper
It is a common fallacy to assume that a model with a lower rate per million tokens automatically produces a lower total bill. Total spending is determined by the price per token multiplied by the number of tokens used.
Various characteristics of a model affect how many tokens are needed to complete a specific task:
Verbosity
One model is naturally concise, while another formulates extensive, chatty or detailed answers. A model that uses twice as many words to give the same answer halves the price advantage of a lower token rate.
Number of attempts needed
Less capable models often require more complex, longer prompts with multiple examples (few-shot prompting) to reach the desired result. If a more compact model needs three attempts to deliver a correctly formatted JSON structure, the eventual processing is more expensive than one successful call to a more advanced model.
Comparing fairly: tokenizers and evaluations
When comparing model vendors, it is necessary to be careful with direct comparisons based purely on token prices. A crucial factor here is the tokenizer that the model uses.
A tokenizer converts raw text into numerical tokens. Different model families use different tokenizers with varying vocabulary sizes. The same Dutch paragraph can be split into 100 tokens by one model while another breaks the same text into 130. A price comparison per token is only clean when it is measured on the same input text and the same desired output.
For a realistic picture it is wise to base the selection process on a structured approach. Working through the guide on choosing a suitable model helps weigh functionality against eventual operational spending. Analyzing the ratio between quality versus cost also gives insight into what performance is achievable within a given budget.
Levers for cost control
Developers and system architects have several knobs to turn in order to keep token consumption in check:
- Optimize prompt length: Remove superfluous formatting, instructions and redundant examples from the system prompt.
- Set a context window policy: Truncate old messages from a conversation history or summarize preceding messages once a certain threshold is reached.
- Enforce output limits: Set the parameter
max_tokensto keep a model from generating endless text when it goes off the rails. - Model routing: Use light, cheaper models for simple tasks (such as classification or summarization) and deploy heavier models only for complex reasoning steps. For the concrete implementation of cost control in production, look at the options for cost monitoring through API gateways.
