Why Output Tokens Are More Expensive: From KV Cache to Agent Cost Engineering

This post is based on a simple interview:

The questions in this article can also be addressedWhat exactly is the model route solving: from Agent cost, delay to reasoning control?、MCP (Model Context Protocol)How the concept of a relatively close read together is developed in different contexts.

Why is the price of input token in the big model usually much lower than output token? Technically, is this a reasonable price?

This is interesting because it looks simple, but it can continue to follow the reasoning system and application architecture. If a person who is an Agent developer has really handled costs and delays, it cannot simply answer "output is more expensive because it is slower to produce." From GPU resources to the reasoning side of the watching schedule, the problem is not light.

If it's just product-level, input and output seem to be token. One token come in, one token go out, why is the price so low?

But from the reasoning system, they are not the same kind of work.

Here. OpenAI API Pricing The official page display is an example of the Standard text token that has been used input、cached input and output Dismantling into different price slots.The same model, input, capched input, unput, was explicitly detached into different resource forms.

This is because the reasoning stage itself is asymmetrical.

Prefill and Decode

Large model reasoning can be broken down into at least two stages:Prefill and Decode。

Prefill The process is for the user to give the prompt. The model is read in a single input to calculate the hidden status of these input token and generate the KV Cache that will be used for subsequent decode.

Decode It's about the model itself being generated token. It can only generate the next token each time, then reconnect this new token back to the context and continue to generate the next one.

The largest difference between the two phases is parallelity.

Prefill phase, where the model can process an entire input in parallel. And the bottom mass of operations is close to the matrix, the GPU's favorite work pattern. Tensor Core of the modern GPU was designed for this high-volume matrix calculation. The model weight is reused in large quantities token as long as the watch and sequence length are sufficient. The weights are moved from the visible to the large array calculations. The utilization of the algorithm can be higher.

Decode is completely different. Self-regression generation determines that the model must be N token to produce N+1 token. Even if you had the strongest GPU, you wouldn't have finished counting the future token in advance. Each token generation is re-engineered, reading weights, reading history KV Cache, writing new KV Cache, and sampling the next token. The calculations themselves are not necessarily enough to feed Tensor Core, and most times GPU is waiting for data to move from HBM to the computing unit.

So one important instinct is:

Prefill:  一次读很多 token，尽量把 GPU 算力吃满
Decode:   一次吐一个 token，被自回归顺序和显存访问拖住

This is the first source of the cost difference between input token and output token. Input token is the main counterpart in the current request prefill, output token. It's also a graphic card, which consumes almost electricity, and Prefil spends much less time than Decode, and the depreciation of the graphic card is a significant part of the cost of language model reasoning.

KV Cache is both optimized and cost-effective

If KV Cache is not available, the model recalculates all previous token's comments key/value for each new token.

KV Cache functions to save the key and value that history has generated in each layer of attitudinal. Next time you generate a new token, the model does not recalculate all history token, but only the new token query/key/value, and then the new query looks at the history KV.

This is certainly a great optimisation. Without it, context generation is largely unavailable.

But KV Cache is not free. State-of-the-art costs of using KV Cache to continuously occupy visible, bandwidth and movement resources. It turns a request into a continuously occupied visible object.

Each request that is still being generated requires that its own KV Cache be kept in the visible memory. The longer the output, the longer the KV Cache; the more the request is made, the larger the KV Cache. Decode, at each step, not only reads model weights, but also history KV, and writes back the new token KV.

So KV Cache has two sides:

Perspective	The benefits it brings.	The cost of it.
Calculate	Token	Cannot eliminate self-returning serial generation
Organisation	Allows context to be used	Every active request takes over the sustained growth.
Movement control	Support for multi-request continuous generation	Watch size is limited by KV capacity and bandwidth
Agent	Reuse Stable Prefix	If the context is in disarray, the C.O.C. rate will drop.

The smallest system object for LLM service is not the request itself, but the KV status that is requested to be carried.

Why, Decode, watching slows.

Batching is very understanding in the Prefill phase. Multiple requests come in, put together big bats, GPU do big matrix calculations, swallow up, single token costs drop.

Decode could also catch, but its returns were not as ideal.

There are three levels of reason.

First, the basic unit of Decode is a round token. You can create "next token" in the same project, but each request can only move forward. A request to generate 200 token will experience about 200 rounds of decode. The batting is increasing the number of requests for ingestion, which does not transform the self-return chain of individual requests into a parallel chain.

Second, the bigger the bat, the more KV Cache needs to be read and maintained. Model weight reading can share a share of the proceeds in the bat, but KV Cache basically increases with the requested number and the length of the context. Decode is already easily stuck in visible bandwidth, and more KV visits will continue to squeeze bandwidth.

Third, the length of the request was inconsistent. In real-life services, some requests are soon closed, others are continuing, and others are just ready to join the decode. Modern engines will use their own data or its operation-level production to kick out the completed requests, add new requests and keep GPU flowing as much as possible. This can significantly increase the amount of vomiting, but it solves the control hole, not the physical bottlenecks of decode.

So more precisely:

Continuous watching makes Decode less waste, but not Decode becomes Prefill.

This is why many online dialogue systems are concerned about two indicators:TTFT and tokens/sec。

TTFT It is time to first token, mainly affected by queues, prefills and movement movements. This is the time that the user waited for before the first time he saw the model opening.

tokens/sec Closer to the continuous generation rate of the decode phase. Users have seen how much token it can throw up every second, mainly influenced by the decode path.

If you're a lot like Agent to plug a bunch of tool files, warehouse summaries and historical messages, TTFT will get worse. If you're still into the mind-scrutinizing and re-suming, the decode costs will continue to expand.

Pricing and Agent Dev

Input is not completely free because prefill still counts. However, as long as input can be processed in parallel, its unit cost is more easily absorbed.

Cached input cheaper, usually for prefix reuse after the impact of prompt cape. With OpenAI Prompt Caching Document is an example where the cache hit depends on matching conditions such as long prefixes, route and retention time. Official price page. Cached input Separately, it is actually making prefix reuse a user-visible cost signal.

Output is more expensive because decode consumes a more difficult serial generation time and visible bandwidth. Long output is not just a few words, but is continuously working hours in the presence of both the visible and the visible cards. There are often more expensive Outlook prices in the longer context to compensate for the excess KV Cache consumption of the visible.

If you put it in the Agent project, the price is actually a reminder to developers of three things:

Cacheable input, try to make a stable prefix.
Don't need the model to say what it says. Don't let it out. Thinking can solve complex problems, but not so many.
Agent cost optimization, not just total token numbers, but also what token is prefill, which token is decode, which token hits the cache.

For ordinary chat products, KV Cache is often a matter of reasoning service providers. Developers see only token bills and delays, and it is difficult to organize a cost-optimization scheme for using pricing and Cache, after all, no one can predict what users say and what models will output.

But for Agen developers, KV Cache will in turn influence how you organize the context.

The first four categories of context would be clearer: stabilization prefixes, such as system instructions, tool descriptions and long-term mission rules; semi-stabilization, such as summaries of the same document, the same warehouse or the same session; dynamic content, such as the results of this round of tools and erroneous information; and user-oriented outputs, such as final responses and explanations. The sequence, stability and length of these elements are usually directly controlled by developers, and not necessarily the presence of the service provider ' s bottom KV in the same example.

Different service providers. cached input It could be achieved by prompt caching, prefix caching or other internal mechanisms. The developers usually only go through.Stable prefix increases the probability of hit.; only in systems that host or explicitly expose the cache-aware route are direct management of runtime KV Cache more closely.

Steady prefix. Steady.

I'm not sure if I'm gonna get a chance to get a better picture of the situation.

Preconditions: Multiple requests share the same or a matchable token prefix.
Typical scene: Long document, question and answer, multi-cycle conversation history.
Engineering boundary: The details of the different systems are different, so it should be understood as creating stabilization prefixes as far as possible, and not as any repetition is inevitable.

This will directly affect the hint organization of Agent.

Many Agent requests are re-spelled prompt, but the order is random: time stamp is at the top, dynamic track is at the front of system program, tool list is unstable, memoory is different at each insertion. This would undermine the shared prefix. Even if the big paragraphs are the same, if a bit of dynamic is inserted, token prefix is no longer the same. The more stable the content, the more forward, the more dynamic the content, the more backward, and thus the Cache, which is used to optimize costs.

A more rational structure is usually:

稳定前缀: system prompt developer policy tool schema / tool descriptions repo instructions / docs index 相对稳定的会话状态: compressed memory selected files / selected docs

高度动态内容: latest user message latest tool result transient trace

Do not contaminate the tool output with long prefixes

Agent can easily plug the tool output directly back into the context, especially in the log, web page, search results, test output, database records.

If these elements enter history without compression, they pose two types of problems.

First, they increase the cost of follow-up. Each round is reprocessed to the longer context, and prices will rise as the context increases.

Second, the context is rapidly decomposed, and a large number of tool outputs are filled with Context, and the model will soon be compressed and re-press, and then the original target will be forgotten by the next round.

So the Age tool layer should be filtered and compressed as much as possible. Do not turn the full tool results over to the model, and let the tool service provider first return to structured summary, key fields, error code, verifiable state. When the original language is really needed, it should be expanded as needed.

A practical principle is:

The return of the tool to the model should be the minimum adequacy required for the next decision-making, rather than a complete reproduction of the outside world.

This is in line with my earlier views on MCP/Harness: the tool interface is not as big as it is good, and the tool output is not as good as it is. They all go into the context budget of the model, and eventually become the cost of prefill, cache and decode.

Long file questions and answers to make visible

Long file questions and answers are the most valuable scene of prefix caching.

If the user questions the same paper, the same code repository, the same financial paper, it is not always appropriate to randomly slice, sort and pompt the document. It would be better to have the context of the document as a stable prefix and then to put different issues behind it.

For example:

[固定任务说明]
[固定文档内容或固定文档摘要]
[引用/证据规则]
[本轮用户问题]

This is a large number of prefixes shared between multiple issues, making it easier for the service to re-use the cache.

Of course, that's not to say always stuff the whole long file in. RAG and incremental context loading remain important. The point here is that when you have decided to re-enter a material, it should be organized into a stable prefix that can be used again, rather than re-aligning it.

Mask tool instead of adding or deleting, keep application-only

I've been talking about this."How do you get the contents?"And there is another dimension that will also quietly destroy the CLA:"How does the context change?"I'm sorry. Manus is here. Context Engineering for AI Agents It gives two engineering disciplines, pre-empting this stability.

Number one: Mask, not dynamic add-and-delet toolI'm sorry. Many Agent prefers to add and reduce the definition of tools to the context in a dynamic way at the current stage, which is a double disaster: the definition of tools is usually placed in a position that is very forward to the context (part of the prefix for stabilization), and the change is to render the KV-Cache behind it large-scale invalid, each round of which is re-prefilled; and the model is confused and even hallucinating if a tool that has been removed is cited in the context. Better to have tools defined.It's always the same...and using a status machine to decode Mask logits for the wrong tool— The tool is still in context, just this step."Unoptional"I'm sorry. Compatibility prefixes for tool names (e.g.) browser_、shell_) can also be bound by group motion space without moving the prefix at all.

Number two:ContextI'm sorry. Do not turn back to the actions and observations that have already taken place — any rewriting of history would render the prefix after that all ineffective. And that's what's going on.Sequence to be sure.: The same content must be sequenced into the same bytes, for example, the key of JSON must be stable, or an invisible field can be reordered quietly to destroy the cache. In other words, prefix stability is not just required."Same content."And I'm asking you to..."Meaning same"。

These two and the front."The stable content is ahead, the dynamic content is back."It's an extension of the same principle:The whole context is operated as a stability prefix, which is not re-packaged as a temporary buffer zone.

Which indicators should be recorded

If you're really doing the Age Cost Project, recording the total number of tokens is not enough.

At a minimum, these indicators should be removed:

Indicators	Annotations	Main counterpart issues
input tokens	Inputs for current round of physical access to models	Is the context too big?
cached input tokens	Inputs for Cache Hit	Is the prefix designed well?
output tokens	Modelled token	Decode, is the cost out of control?
TTFT	First token delay	- Yes, sir. - Yes, sir.
decode latency	Time-consuming generation on a continuous basis	Is output too long and the bandwidth tight?
output/input ratio	Output-Input Ratio	Agent overexplains or emptys
cache hit rate	Prefix / KV reuse	Whether context structure disrupts cache

With these indicators, you can judge where the optimization is.

If TTFT is high, caught input is low, the problem may be re-used in context structure and prefix.

If TTFT is okay, but it's always high, the problem is probably output too long or decode too slow.

If input is large but capched input is high, it may not be bad, as long stabilization prefixes may have been reused.

If output/input radio is long-term high, especially in classification, tool route, and schema generation of such intermediate steps, it means that Agent may be doing the least of what is supposed to be done with the most expensive token.

Answer the first question.

A short answer:

This pricing is technically reasonable. Input token 's main consumption prefill calculation is easily measured by the bat and matrix; unput token 's lead time, visible bandwidth and sustained KV Cache status of the major consumption decode phase. A reasonable technical explanation is that such pricing reflects the more scarce and less affordable costs of GPU resources.

And then we'll expand on three points.

First, Prefil and Decode are different parallels. Prefill is closer to the big matrix calculation, Decode is restricted by the order of return.

Second, Decode is easier to memoory-born. For each token generated, the model weight and history KV Cache is accessed, and the visible bandwidth becomes the core bottleneck.

Third, KV Cache improved the double counting, but introduced the current cost of the continued occupation of the visible and movement control resources. But watching can lift the stale, but it can't turn the swab generation into a complete parallel. Continuous Batching, trying to squeeze the graphic cards, but with a synergy with KV-Cache, it's gonna be a gradual build-up.

Three points back to the same question: Decode needs to take a card longer, and GPU depreciation is expensive.

Finally, the project understands:

For Agent developers, the idea is to organize context into a cached, stable prefix, to compress the results of dynamic tools and to strictly control the output budget of intermediate steps.

And then you answer not just the model theory, but the connection between the reasoning system and the application architecture.