What is spatial context? The missing input for AI coding agents

Your coding agent can grep the entire repo in milliseconds, but it cannot see your screen. Spatial context is everything an AI coding agent can't get from your repo: what's on screen, what you're pointing at, and what you're saying about it — delivered as pixels plus accessibility-tree text. It's the input you already supply by hand every time you screenshot a broken layout and type a paragraph explaining where to look.

If you searched for this term, you probably found neuroscience papers about place cells or GIS literature about map neighborhoods. This page defines it for a newer problem: agents like Claude Code and Codex that have perfect recall of your codebase and zero awareness of the screen in front of you.

Spatial context, defined

A coding agent working from the repo alone can't answer three questions:

1. What are you looking at? Which app, which window, which region of which display.
2. What are you pointing at? Which of the four hundred elements on screen "this" refers to.
3. What do you mean about it? What's wrong with it, and what you expected instead.

Spatial context is the answer to all three, packaged so a language model can use it: a screenshot for appearance, accessibility-tree text for structure and coordinates, a gesture trail for reference, and a transcript for intent.

The idea is old. Richard Bolt's "Put-That-There" demo at MIT in 1980 combined pointing and speech precisely because pointing makes pronouns work — you can say "put that there" instead of describing objects and destinations in words. Forty-six years later, developers talk to coding agents all day, and "this" still doesn't resolve. Linguists call words like this, here, and there deictic: they only mean something relative to where the speaker is and what they're attending to. A repo has no here. Your screen does.

Two properties separate spatial context from neighboring categories (we'll compare them properly below). It's live — it describes the present moment, not a recording of last Tuesday. And it's shared — it establishes a common reference frame between you and the agent, the same way pointing at a whiteboard does between two engineers. It is not the agent controlling your machine, and it is not a searchable archive. It's the missing half of a conversation that currently happens over text alone.

Your coding agent is effectively blind

Here is the loop most developers run today. The agent says it fixed the layout. You look at the screen; it didn't. You press Cmd+Shift+4, drag a crop, find the file, drag it into the terminal, and then — the expensive part — write prose describing where to look: "the third card in the second row, the one with the truncated label, no, under the chart." You have become a human serializer, translating spatial information into text, which is the one format that's worst at carrying it. We measured what this costs and how to shortcut it in our guide to pasting screenshots into Claude Code.

The deeper problem isn't the friction. It's what a text-only agent systematically cannot know:

• The render, as opposed to the code. The JSX says justify-content: center; the browser shows the button clipped behind the nav. Both are true. The agent only sees the first.
• Transient states. Hover styles, toasts that vanish in three seconds, a dropdown that misaligns only while open, scroll-linked animation jank. None of it exists in any file.
• Visual properties with no textual shadow. Overlap, z-order, contrast, a font that loaded wrong, two pixels of misalignment that make a UI feel broken.
• Which instance you mean. The repo has 14 components named Card. Your screen has exactly one that's wrong.

This is why an agent will confidently declare a broken UI fixed: its evidence is the code and the passing tests, and by that evidence it is fixed. It's not lying — it's blind, and reasoning correctly from incomplete input. Giving the agent eyes changes the failure mode from "confidently wrong" to "checks its own work," which is the whole argument of Give Claude Code eyes.

The four channels of spatial context

The complete input is four channels. Each carries something the others can't.

1. Screenshots: what's actually drawn

Pixels are the ground truth of rendering — the only channel that shows overlap, clipping, color, and layout as the user experiences them. And frontier models are now genuinely good at reading them: on the ScreenSpot-Pro grounding benchmark (July 2026), Claude Opus 4.8 scores 87.9% and GPT-5.2 scores 86.3% at locating UI elements in professional, high-resolution screens. External vision parsers like OmniParser — which hit 39.6% with GPT-4o in early 2025 — are effectively obsolete for anyone using a frontier model. The bottleneck moved: models can read screens; the problem is getting the right screen, at the right moment, into the conversation.

2. Pointing: which of the 400 elements you mean

A 16-inch MacBook screen is 3456×2234 pixels of ambiguity. A gesture trail collapses it to the 200 pixels you care about. This is deixis made machine-readable — the "that" in "put that there." Research systems like Speech-to-Spatial and GesPrompt show that combining a pointing gesture with speech resolves references far more reliably than either alone, but this line of work has mostly stayed in labs and VR headsets rather than shipping on the desktop, where developers actually talk to coding agents.

3. Voice: what's wrong and what you expected

A screenshot shows the state; it doesn't say which part of the state is the bug. Narration carries intent — "this spacing should match that section," "this spinner never resolves" — and it's much faster than typing while your hands are busy pointing. The channel gets its real power from time alignment: if the words "this one" land on the same half-second as a loop of the trail around a button, the reference is resolved. Words and gesture disambiguate each other, exactly as Bolt demonstrated in 1980.

4. The accessibility tree: names, roles, and exact coordinates

The accessibility (AX) tree is the machine-readable skeleton of the screen: every element's role, name, and bounding box, plus text content — including content scrolled out of view, which no screenshot can capture. Where pixels tell an agent how it looks, the tree tells it what it is: that the blurry blue region is AXButton "Deploy" at a precise coordinate, traceable to an identifier in your code. It's also dramatically cheaper in tokens, which matters more than most people expect — the full trade-off is in accessibility tree vs screenshots, and the token math below.

Spatial context vs computer use vs AI memory vs browser automation

These four categories get conflated constantly, because they all involve "the agent and your screen." They answer different questions.

| | What the agent gets | Direction | Scope | Examples | |---|---|---|---|---| | Spatial context | A live shared reference: what you see, point at, and say, right now | You → agent | Any app on screen | SupaMaus Lite (our product), Peekaboo | | Computer use | Hands: the agent moves the cursor, clicks, and types | Agent → screen | OS-wide | Claude Code computer use (research preview) | | AI memory | Recall: a searchable, text-indexed archive of past screens and audio | Past → agent | 24/7 background recording | Screenpipe | | Browser automation | DOM inspection and control inside one browser | Agent → browser | Browser tab only | Playwright MCP, Jam.dev |

The key axis is direction. Computer use is output — the agent acting on your machine, as in Claude Code's macOS research preview (March 2026), which we tested in Claude Code computer use, reviewed. Spatial context is input — you showing the agent what you mean. They compose: an agent needs eyes before hands are much use.

AI memory tools answer a different question entirely. Screenpipe records your screen and audio around the clock into a local SQLite index — genuinely useful for "what did that error dialog say yesterday?" But recall of the past is not a shared reference in the present, and the category has been volatile: Meta acquired Limitless in December 2025 and its screen and audio capture shut down on December 19, 2025.

Browser automation is the most mature of the four — Playwright MCP gives an agent full DOM truth and control — but it ends at the edge of the tab. Your terminal, your native app, your iOS Simulator, and your second monitor don't exist. Jam.dev does excellent narrated bug reports with layered artifacts (video, console, network, transcript), also browser-only.

The token math: pixels vs accessibility trees

Screen context for an LLM has a cost problem, and it's worth doing the arithmetic, because naive screenshots are shockingly expensive.

The cautionary number: a single Playwright MCP browser_take_screenshot call was measured at 232,000 tokens — a full-resolution capture serialized without downscaling, big enough to swamp a context window in one tool call. A 2026 comparison of AI browser-automation tools puts a typical Playwright MCP session around 114K tokens. That's the price of treating pixels as free.

Done deliberately, the numbers are fine. Claude's vision pricing is roughly width × height ÷ 750, with images ideally at most ~1,568px on the long edge. A Retina MacBook frame downscaled to 1568×1014 costs about 2,100 tokens; Claude Code's computer-use preview downscales the full 3456×2234 screen to ~1372×887, around 1,600 tokens per frame. A cropped region around the element you're pointing at costs a few hundred. Meanwhile one accessibility-tree line — role, name, bounds — costs tens of tokens, and the full text of a window a few thousand.

So why not send the tree alone and skip pixels entirely? Because text-only loses too. In the OSWorld benchmark's ablations, agents given the accessibility tree plus a screenshot achieved 69.2–71.8% relative task success, versus 56.4% for annotated screenshots alone — the hybrid beats either input by a wide margin. The channels are complementary: pixels for appearance, tree for identity and exact coordinates.

Two more results sharpen the picture. GUI-Eyes (January 2026) showed a coarse-to-fine policy — look at the whole screen, then zoom into a crop — reaching 44.8% on ScreenSpot-Pro with only a 3B-parameter model: cropping to the relevant region beats brute resolution. And ActionEngine found that a structured state-machine map of a UI beat raw snapshots 95% to 66% on task success while being 11.8× cheaper. The pattern across all of it: structure plus targeted pixels wins; raw full-screen pixels lose. A well-engineered capture is built around exactly this — a downscaled annotated frame, a crop where you pointed, and the AX text for precision.

How to add spatial context to Claude Code and Codex today

There's a ladder, from zero setup to fully wired.

Rung 1: paste screenshots manually. Every frontier coding agent accepts images. Cmd+Shift+4, drag the file in, describe the location in words. Works today, costs you the serialization loop described above, and the agent still can't take the next look on its own.

Rung 2: a screenshot MCP server. Tools like Peekaboo and macos-use let the agent request screenshots and AX reads itself — the MCP spec supports images directly in tool results, so any client that speaks MCP can receive them. This closes the verification loop (the agent can check its own fix) but doesn't capture you: no pointing, no voice, no "this one, here." We compared the options in the best screenshot MCP servers.

Rung 3: full spatial context — pixels, pointing, voice, and tree together. Disclosure: this is the product we build. SupaMaus Lite is a macOS menu-bar app that runs a local, token-authenticated MCP server on 127.0.0.1:19741. Hold Control+Option and move the mouse to draw a highlight trail over anything on screen; release, and it saves an annotated screenshot with your gesture, an on-device voice transcript (WhisperKit — no audio leaves the machine), and the accessibility-tree grounding for what you circled: element role, name, and exact bounds. Spoken words like "this" and "here" are aligned to segments of the trail. Everything runs on-device with no API keys.

Hooking it up to Claude Code is one MCP registration (the app's setup pane provides the exact command with your auth token):

claude mcp add --transport http supamaus http://127.0.0.1:19741/mcp

Or run the bundled agents/install.sh, which configures both Claude Code and Codex in one step — MCP config plus an agent skill. Claude Code then surfaces captures through the /mcp__supamaus__look slash command. For Codex and other clients without HTTP MCP support there's a stdio bridge, and every capture also mirrors to ~/.supamaus/latest/ as plain files (capture.png, context.md, manifest.json), so you can paste a path into any agent's composer — the workflow we walk through in getting screenshots into Codex.

One case worth calling out: the iOS Simulator bridges your app's accessibility elements into the macOS AX tree, so pointing at a button in the Simulator resolves the actual element — including your own accessibilityIdentifier from the Swift source (shipped in v1.0.69, July 2026). For iOS developers this turns "the second cell looks wrong in dark mode" into a grounded reference an agent can act on; details in using the iOS Simulator with Claude Code.

Whichever rung you pick, the direction of travel is the same: stop describing your screen in prose and start handing the agent the screen itself.

FAQ

How do I let Claude Code see my screen?

Three ways, in increasing order of leverage: paste screenshots into the composer manually; add a screenshot MCP server so Claude Code can capture the screen itself; or run a spatial-context tool that adds pointing, voice, and accessibility-tree grounding on top of pixels. Each MCP option is one claude mcp add command.

Why does Claude say my broken UI works?

Because its evidence is the code and the test output, not the render. If the diff looks correct and the tests pass, a text-only agent has no way to observe that the button is clipped or the modal is behind the overlay — visual truth never enters its context. Give it a screenshot of the actual render and it will usually spot the problem immediately; give it a way to take screenshots and it will verify its own fixes.

Does Codex see screenshots?

Yes. Codex accepts images, and MCP tool results can carry images to any compliant client per the MCP specification. If your setup makes attaching images awkward, a file path works too: SupaMaus Lite mirrors every capture to ~/.supamaus/latest/capture.png with a companion context.md, precisely so the same capture drops into Codex, Claude Code, or any other composer.

What's the difference between computer use and spatial context?

Direction. Computer use gives the agent hands: it moves your cursor, clicks, and types — the agent acting on the screen. Spatial context gives the agent eyes on what you are doing: your screen, your pointing, your words — you informing the agent. They're complementary, and an agent with hands but no shared reference frame is the more dangerous of the two combinations.

Is a screenshot enough, or do I need the accessibility tree too?

Use both. OSWorld's ablations put the combination at 69.2–71.8% relative task success versus 56.4% for annotated screenshots alone. Pixels show how things look; the tree says what they are and exactly where — and it's cheap enough (tens of tokens per element) that leaving it out saves almost nothing.