Tensor TownISSUE #103 of 120

context caching · cutting cost

NeuraVSThe Overfit Ogre

Neura saysContext (prompt) caching reuses repeated prompt prefixes to cut cost and latency.

If many requests share a long prefix (a big system prompt, a document), recomputing it every time is wasteful. Prompt/context caching stores the processed prefix so repeated calls skip that work — major cost and latency savings for chatbots and RAG with stable instructions. Structure prompts with the stable, cacheable part first and the variable part last to maximize cache hits.

Power-ups you unlock

Reuse processed repeated prompt prefixes
Big savings for stable system prompts/docs
Put stable content first, variable last
Helps chatbots and RAG

The Overfit Ogre attacks — common mistakes

Putting variable content before the stable prefix
Assuming caching with constantly-changing prompts
Ignoring provider cache rules/TTL

Boss battleExplain how prompt structure affects cache hit rate.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>[ stable system prompt + docs ] ← cache this
[ variable user question ]      ← changes
stable-first → high cache hits → cheaper/faster</pre></body></html>

▶ Open the interactive comic issue

‹ Synthetic Data · Self-Instruct · Distill Structured Outputs · Json Mode · Constrained ›