#Reuse stable prefixes

Prompt caching reuses the stable part of a request (system prompt, tool schemas, conversation prefix) so multi-step tool loops do not re-pay full input cost every turn.

Most providers either cache automatically or need only a sticky session key. Anthropic is the exception: it needs explicit cache breakpoints (or a gateway that inserts them).

#Provider matrix

What crabcode does today for each routing path:

Anthropic is the unfriendly one. Without cache_control (or Gateway caching: auto), Claude traffic does not cache-read. That was the main gap crabcode fixed.

#What gets marked (direct Anthropic)

Hybrid of OpenCode auto + Grok Build placement:

1. Last tool — tool schemas are large and stable across a tool loop
2. Last system block — instructions / project context
3. Transcript tip — last markable content block (skips thinking / redacted_thinking)
4. Previous user tip — where the prior request ended, so turns past the 20-block lookback still hit cache

At most 4 breakpoints; the 4th slot stays free when tools or previous-user are missing so gateways can auto-mark.

Breakpoints are applied after message regrouping so adjacent tool_use / tool_result blocks stay valid.

#xAI Grok Build (cli-chat-proxy)

In addition to sticky prompt_cache_key = session_id, crabcode stamps Grok Build–style affinity headers:

Grok Build’s sampler sends that header so cli-chat-proxy emits response.doom_loop_check with tail_repetition:{n}@thinking. Crabcode treats those as confident the same way (DoomLoopRecoveryPolicy::is_confident: thinking channel, threshold 2..=64).

• Policy / window: .devrefs/references/xai-org/grok-build/crates/codegen/xai-grok-sampling-types/src/doom_loop.rs
• SSE collector: .devrefs/references/xai-org/grok-build/crates/codegen/xai-grok-sampler/src/doom_loop.rs

Doom-loop recovery matches Grok Build’s sampler, not a client tool-name babysitter. Only confident tail_repetition:{n}@thinking from response.doom_loop_check resamples: skip that generation’s tools, inject RECOVERY_REMINDER into the next request only (same <system_reminder> user-role item as ConversationItem::system_reminder — not a visible user turn). After DEFAULT_MAX_RETRIES (2) the abort is disarmed and the turn continues. Repeating reads or shell commands are not a loop — they usually mean earlier tool results were cleared.

Tool-result pruning matches Grok Build prune_conversation (.devrefs/references/xai-org/grok-build/crates/codegen/xai-chat-state/src/actor/request_builder.rs): only when estimated tokens exceed 50% of a 500k window (should_prune), last 3 user turns stay intact, older large results soft-trim at 4000/1500/1500, age ≥ 10 hard-clear. Those numbers have not changed in Grok Build; the important gate is not pruning a short session at all.

Parent-cached aux (e.g. max-steps text-only summary): keeps parent prompt_cache_key + session/conv so the conversation prefix can reuse the main turn's KV cache; assigns a fresh aux-… req id.

Subagents use the child session id on purpose. They have a different system prompt and tool set, so parent-prefix reuse would miss and can pollute sticky routing. Isolation beats a false shared key.

#Prefix stability (anti-bust)

• Empty system/user rows are dropped before the wire (they pad the sticky prefix).
• Images use hysteresis: compact only above a trigger (~6 MiB total data-urls), then reclaim to a lower target (~3 MiB) so later turns stay cache-warm instead of re-evicting every step.
• Tool schemas (built-in + enabled MCP only) ride every request, including each tool-loop step — the Responses API requires the tools array. Prompt cache is supposed to cover that prefix. Disabled MCP servers are dropped from the live manager before the next request is built, so toggling /mcp off must not leave those schemas on the wire.

#How to verify it works

Run with logs enabled and watch app.log for [prompt-cache] lines:

crabcode --emit-logs

Logging writes to app.log in the working directory when --emit-logs is set.

#Direct Anthropic

[prompt-cache] anthropic input=… output=… cache_read=… cache_creation=… total_input=… hit_pct=…

#AI Gateway / OpenAI-compatible

[prompt-cache] openai-compatible prompt=… completion=… cached_tokens=… cache_read=… cache_creation=… hit_pct=…

#OpenAI / xAI Responses

[prompt-cache] openai-responses input=… output=… cached_tokens=… hit_pct=…

#Healthy multi-step session

Notes:

• Anthropic's input_tokens is non-cached only. Total input ≈ input + cache_read + cache_creation.
• Gateway may surface cache as prompt_tokens_details.cached_tokens and/or forwarded Anthropic fields — crabcode logs both when present.
• No dashboard required: stream finals carry usage; crabcode logs them when logging is enabled.

#What we intentionally do not do

#Related

• AI Gateway automatic caching: Vercel docs
• Anthropic prompt caching: provider docs for cache_control / ephemeral