Tensor TownISSUE #116 of 120

cost · per-token economics

NeuraVSThe Overfit Ogre

Neura saysAI economics run on per-token costs — input and output tokens priced separately.

Using a hosted LLM, you pay per token, usually with output tokens costing more than input. Costs scale with prompt size (context), output length, and model tier — so a chatbot with a huge system prompt on every call gets expensive fast. Levers: smaller models where adequate, prompt caching, trimming context, capping output, and batching. Track cost per request like any other unit economic.

Power-ups you unlock

Pay per token; output usually pricier than input
Cost scales with context + output + model tier
Levers: smaller models, caching, trim context
Track cost per request

The Overfit Ogre attacks — common mistakes

Huge system prompts on every call
Ignoring output-length costs
No cost monitoring per request

Boss battleList three ways to cut per-request LLM cost.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>cost ≈ input tokens + output tokens × tier
cut: smaller model · cache prefix · trim context · cap output</pre></body></html>

▶ Open the interactive comic issue

‹ Agi · The Moving Target Latency · Ttft · Tps · Streaming ›