02. Prompt Length, Context Growth, and Cache Cost

In the captured Kimi trace, the first model response requested two Bash operations. After the CLI executed them, it sent a second model request containing the original system message, environment, task, the model’s tool calls, and both tool results.

The final answer hides this accumulation. Every additional Agent turn can make the next model input larger.

Four quantities that must remain separate

A context window is the maximum number of Tokens one model call can process. A Token is a discrete unit used by the model and is not equivalent to one character. Input Tokens measure one request. Cumulative input adds the input of every model call in the task.

If stable rules and tool definitions occupy S + T, and turn i adds M_i, complete-history mode produces approximately:

input_n = S + T + M_1 + M_2 + ... + M_n

The cumulative input is not the length of the final request. Earlier content is transmitted and processed again in later calls, so the task total grows faster than the visible conversation.

What the capture establishes

The successful Codex trace reported:

Cached input represented about 48.4% of that request. This does not imply a 48.4% price reduction because cached-Token pricing depends on the service and model.

Kimi’s two HTTP request bodies were approximately 94 KB and 96 KB. Byte length proves that the second body was larger, but it cannot be converted reliably into Tokens. JSON escaping, field names, Unicode encoding, and tokenization all change the ratio.

Claude Code returned 403 and Grok returned 402, so neither entered a second turn. Those failures provide no evidence about their successful history-growth behavior.

Four controls solve different problems

Prompt caching reuses computation for an identical prefix. It does not enlarge the context window and does not stop history growth. Trimming and summarization change what the model sees; caching does not.

Anthropic documents the cache order as tools, system, then messages. Stable material placed before per-turn tool results is more likely to preserve a reusable prefix. See Claude Prompt Caching.

Dynamic fields should appear late

A date, random identifier, working directory, or live status placed at the beginning of a request creates an early divergence. A long unchanged rule set after that divergence may no longer share the same cache prefix.

A useful order by change frequency is:

This is not an instruction-priority table. Priority answers which instruction wins a conflict. Ordering answers where a stable prefix can exist.

What a summary must preserve

“The user is analyzing a project” is not enough to resume an Agent. A usable summary includes:

• The current objective and explicit prohibitions.
• Resources already read or modified.
• Completed steps and their evidence.
• Tool failures, error classes, and retry counts.
• Incomplete steps.
• Operations awaiting user confirmation.

The summary also needs a reference to the history range from which it was produced. Otherwise an omission cannot be traced back to its source.

A ten-turn measurement

Every turn records request sequence, input Tokens, cached input, output Tokens, time to first Token, total latency, and newly added events. Comparing only the final turn ignores repeated work in the first nine. Comparing only the cumulative total hides the turn where growth accelerated.

The trace also records where the first changed Token appears before the cache boundary. A content digest can compare experimental prefixes without retaining sensitive text. If a timestamp is inserted before the platform rules in turn three and cache reuse drops from that turn onward, the trace can connect the change to the outcome.

Trimming and summaries need an audit path

“Keep the last N messages” is not a sufficient trimming policy. An early permission decision, failure cause, or unresolved constraint may still govern the current action. Each removal records message identifiers, the reason for removal, and the summary version that replaced them.

A summary should not overwrite its predecessor. An auditable record preserves its source range, generation model, creation time, and validation result. If “network access is forbidden” becomes “network access is allowed,” recovery requires the original history rather than another summary of the corrupted summary.

When compression should begin

A context-window percentage is only one signal. A large completed tool result can be externalized early. A short conversation containing dense constraints may be unsafe to summarize even when many Tokens remain.

A practical trigger considers remaining context, expected next output, unresolved tool calls, and evidence that must remain verbatim. Evaluation includes one task that depends on a constraint from the first turn and another that depends only on recent results. A policy that succeeds only on the second task is not sufficient for a long-running Agent.

Verifying cache layout

Run the same five-turn task while changing only a short suffix in each user message. Record input, cached input, first-Token latency, and total latency. Then move a current timestamp to the beginning of the system block and repeat.

If cache reuse begins to fail earlier in the second run, the difference comes from context layout rather than answer content. Identical final answers do not imply identical execution cost.

Why cumulative input contains a quadratic term

Let the stable prefix contain p Tokens and let every turn add m Tokens. If the client resends the complete history, call i contains approximately p + i × m input Tokens. After n calls:

total(n) = Σ(p + i × m)
         = n × p + m × n × (n + 1) / 2

The second term grows with n². A single request does not acquire a quadratic context window. The term appears because an early message is counted again in later requests. In a ten-call run, first-turn content can appear ten times; tenth-turn content appears once.

This lower-bound model omits variable tool-result sizes, output lengths, and compaction. It still exposes one runtime consequence: removing an unnecessary model round trip also removes that turn's messages from every later request.

Prefix caching changes the price and prefill latency of repeated Tokens; it does not remove them from the context. Anthropic's current documentation separates four controls: tool search reduces definitions loaded up front, programmatic tool calling removes intermediate model round trips, prompt caching reuses prefix computation, and context editing removes stale tool results. They operate at different points in the pipeline. Anthropic, Manage tool context

KV cache and cross-request prompt cache are different

A KV cache stores attention keys and values during model inference so each generated Token does not recompute all preceding Tokens. A prompt cache lets a service reuse an identical prefix across requests. The first is an inference-time data structure. The second is a service-level policy spanning calls.

Both depend on prefix identity, but their lifetimes and observable metrics differ. A client generally cannot inspect tensors held by a model worker. It can observe provider usage fields, cache-read Tokens, and time to first Token.

Anthropic currently constructs cacheable prefixes in tools → system → messages order. A tool-definition change invalidates the later system and message layers; changing tool_choice affects the message layer. The default TTL is five minutes, with a one-hour option. A new entry becomes available only after the first response begins, so a simultaneous cold batch cannot assume that all requests share the entry created by the first one. Anthropic, Prompt caching

Hit rate is not a cost model

Let ordinary input cost C_in per Token, cache writes cost C_write, and cache reads cost C_read. The input portion of one request is:

cost = uncached_tokens × C_in
     + cache_write_tokens × C_write
     + cache_read_tokens × C_read

Two traces with the same percentage hit rate can have different costs because their absolute prefix sizes differ. Hit rate also does not isolate latency: queueing, transport, service scheduling, and uncached-suffix prefill all contribute to time to first Token.

A ten-turn report therefore needs absolute cache-read Tokens, uncached Tokens, time to first Token, and cumulative cost. “80% cache hit” supplies neither the denominator nor the initial write cost.

Compaction creates information debt

A summary is not a lossless encoding of 20 KB into 2 KB. A model decides which facts survive. Information debt is the future cost created when content that appears irrelevant now becomes necessary for recovery, verification, or audit later.

The model that generated a summary should not be its only judge. Before compaction, extract factual probes such as “Was network access allowed?”, “Which files changed?”, and “What was the last tool error?” Ask them again using only the compacted context. A changed answer prevents replacement of the source trajectory.

This is a fidelity test: it measures whether task-critical facts remain recoverable. Fluent prose is not evidence of fidelity.

A controlled four-group experiment

Run the same ten-turn task under four configurations:

The model version, tool definitions, task messages, and inter-request delay remain fixed. A run outside the cache TTL is not comparable to one inside it. Group D must pass factual fidelity probes in addition to cost and latency checks.

Final-answer correctness is insufficient. If the Agent forgets a file read in turn six, repeats the tool in turn seven, and eventually answers correctly, outcome correctness passes while state retention and process efficiency fail.

A practical compaction trigger reserves future capacity:

usable = context_limit
       - reserved_model_output
       - reserved_tool_results
       - safety_margin

When input approaches usable, externalize large consumed tool results first, remove content that can be fetched again second, and summarize decision history last. A file can be reopened; an approval rationale that existed only in the conversation may not be reconstructable.

Each compaction creates a new context generation, a version identifying the source range and replacement. If behavior changes later, the runtime can compare facts across generations instead of inspecting a history already overwritten.

As of July 2026, provider APIs expose several independent controls for context pressure rather than one universal long-context switch. A larger window postpones exhaustion; it does not remove repeated prefill, irrelevant schemas, stale results, or summary errors.

The capture proves that Kimi resent prior trajectory data on its second request and that the observed Codex request reported cached input. It does not establish how either client compacts a much longer session. Cache TTL, provider-side state, and worker-local KV tensors are not directly observable in one HTTPS body. Those claims require multi-turn usage data, latency distributions, and pre/post-compaction fidelity tests.