Thesis Paper · July 2026 · Pre-registered
A small, local, text-only model operates a graphical interface — and never sees an image.
Every serious computer-use agent works the same way: screenshot, feed a large vision model, act, sleep, screenshot again. It is expensive, stateless, and blind between captures. ESP proposes the opposite organ. The model holds a labeled skeleton of the screen as world state and perceives only change — sensing structure through echoes, the way a bat flies in the dark.
The thesis, in one sentence
A small text-only language model, given a persistent labeled skeleton of the screen and a millisecond-scale stream of change events, can complete realistic interactive tasks currently assumed to require vision–language models consuming full screenshots — at a fraction of the perception cost, with faster and more reliable action verification.
How it perceives — three passes
structure from echoes
Recursive decomposition carves the screen into an adaptive region tree — big lazy cells over empty space, tiny dense ones around content. Then it probes: hit-tests, hover sweeps, cursor changes. Where the cursor flips to a hand, the system just told you something is clickable.
the skeleton learns to read
OCR runs only inside the cells the skeleton already found — faster and more accurate than reading a whole page, and every word is born attached to a node. Icons get named by their hover tooltip, once, then cached. Now the model knows exactly what the page says and where.
perception of change
The screen is diffed at millisecond scale — straight from the VNC/RDP dirty-rectangle stream where one exists. A progress bar advancing, a modal appearing, a click that silently did nothing all arrive as ten-token events against a skeleton the model already holds. Waiting becomes a sense, not a guessed timer.
One session, as the blind model experiences it
Upload a file, confirm success. Perception output is plain; the model's emitted tool calls are marked >>. It reads a filename it never typed, watches a bar to 100%, and confirms from banner text — with no image anywhere.
SKELETON (session start, abbreviated): node 7 button "Choose file" clickable node 9 text "No file selected" inert node 14 button "Upload" clickable, disabled >> click(node 7) changed(node 9, text: "No file selected" -> "report.pdf") changed(node 14, afford: disabled -> clickable) # Upload went live >> click(node 14) appeared(node 51, progressbar) changed(node 51, text: "0%" -> "47%") >> wait_for(node 51, quiet, timeout=30s) changed(node 51, text: "47%" -> "100%") vanished(node 51) appeared(node 60, banner "Upload complete") >> read(node 60) "Upload complete. Document ID 8842." >> done(success, detail="Document ID 8842")
Four falsifiable hypotheses
Fixed here before the prototype exists, so no result can quietly edit the question it answered.
A 4B–14B text model on ESP completes real multi-step web tasks within 10 points of the same model driving a full accessibility tree, and above a same-size screenshot + vision baseline.
Perception cost per completed task is at least 5× lower than a per-step screenshot loop — because deltas replace re-perception.
On tasks seeded with silently-failing controls, the echo layer's action windows raise success by at least 15 points over the same agent with change-sensing disabled.
On canvas-rendered pages with an empty accessibility tree, ESP keeps at least 70% of its structured-page success while a DOM-tree agent falls to near zero.
This is a thesis paper in the strict sense. The architecture, the four numbers above, and a fixed decision rule are all committed to print before the first line of the prototype is written. Whatever the experiments say — full success, an echo layer that ships on its own, a canvas-page niche, or honest defeat — the goalposts cannot move to meet the result. A bat does not decide where the wall is after it hears the echo. Neither will we.
ESP is the perception half of the same bet as Prism's reasoning half: mechanical scaffolding around small local models, everywhere the field assumes you need a giant one.
Architecture, related work and what is actually new, a worked session, the four hypotheses, the pre-registered experimental plan, and the failure modes we expect.
Bad Theory Labs · Lagos · hello@badtheorylabs.com