Tensor TownISSUE #106 of 120

computer use · operating the desktop

NeuraVSThe Overfit Ogre

Neura saysComputer use lets a model operate a desktop — clicking, typing, reading the screen.

The newest agent frontier: models that perceive a screen (via screenshots) and take UI actions — move the mouse, click, type, scroll — to complete tasks in real applications, including ones with no API. It generalizes automation beyond brittle scripts, but is slower, error-prone, and risky (it can take real actions), so it runs with confirmations, sandboxes, and human oversight for anything consequential.

Power-ups you unlock

Perceive the screen + take UI actions
Automates apps that have no API
More general than scripted automation
Slower, riskier → sandbox + oversight

The Overfit Ogre attacks — common mistakes

Granting unsupervised control of real systems
Expecting fast, flawless UI automation
No sandbox/confirmation for risky actions

Boss battleExplain why computer use needs more guardrails than text generation.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>screenshot → model decides → click/type/scroll
operates apps with no API
real actions → sandbox + confirmations</pre></body></html>

▶ Open the interactive comic issue

‹ Function Calling · Tools As Schemas Claude · Gpt · Gemini · Llama · The Lineup ›