← Blog

Give Claude Code eyes: let it see your Mac and iOS apps (not just the browser)

10 min readMilind Soni

The bug is right there on your screen — a truncated label in the iOS Simulator, a menu-bar popover that opens two pixels off. Claude Code, the agent that just wrote the code, can't see any of it, so you screenshot, describe, paste, and repeat, playing eyes for a tool that greps your entire repo in milliseconds.

"Give Claude Code eyes" is by now a small genre. Tal Rotbart used the phrase for round-trip screenshot testing of web UIs; twocentstudios used it for SwiftUI snapshot tests. Nearly everything written under this headline routes through a browser or a test harness. This post is about the rest of your screen: SwiftUI apps, Electron windows, the iOS Simulator, any native macOS window.

Giving an agent eyes takes three things: pixels (what's on screen), geometry (where things are, as coordinates and element bounds it can act on), and intent (what you meant when you pointed). Most workarounds deliver pixels. Almost none deliver the other two.

Those three together are what we call spatial context, and the rest of this post is a ladder toward getting all of it.

"The agent is effectively blind"

Developers have been saying this in exactly these words. In March 2026, anthropics/claude-code#30925 asked for the agent to view image files with its own vision:

Claude Code has vision. It can analyze screenshots, describe UIs, spot visual bugs. But it can only use this ability when I manually paste an image into the chat (Cmd+V). [...] The agent is effectively blind.

The issue is precise about the state of the art, too: "Community workarounds exist (Playwright MCP, round-trip test harnesses, Chrome DevTools MCP), but they're all browser-specific and heavyweight." It was closed a month later as not planned.

Parts of that specific complaint have since been addressed — Claude Code can read image files now, and computer use shipped as a research preview the same month. But the loop the issue describes is still how most people work on anything native: screenshot, crop, paste, type a paragraph explaining which button you mean, wait, repeat.

The workaround ladder, from Cmd+Shift+Ctrl+4 to computer use

Rung 1: manual screenshots. Cmd+Shift+Ctrl+4 copies a screen region straight to the clipboard, and Ctrl+V — not Cmd+V — pastes it into Claude Code. This works on any window, which is why it remains the default. It also costs a full context switch per round trip, the agent can never look again on its own, and the image carries zero geometry: no element names, no coordinates, no text it can grep for. We walk through the mechanics (and the Ctrl-vs-Cmd trap) in how to paste a screenshot into Claude Code. fltman's screenshot skill automates the capture half nicely — "screenshot Safari" drives macOS screencapture and hands the file path back to Claude — but the output is still just pixels.

Rung 2: Playwright MCP. This is where most "give Claude Code eyes" posts land, and inside a browser it genuinely works: the agent navigates, snapshots, and reads the DOM by itself. It is browser-only by construction, and it's heavy — a single browser_take_screenshot call was measured at 232,000 tokens, and a typical Playwright MCP session runs around 114K tokens.

Rung 3: Claude Code's built-in computer use. Since March 2026, Claude Code on macOS has a research-preview computer-use server (Pro/Max plans): enable it via /mcp, grant Accessibility and Screen Recording, and Claude can open apps, click, type, and screenshot the result — including native apps and the iOS Simulator. It's honest work, with honest limits: every screenshot is auto-downscaled (a 16-inch MacBook Pro's 3456×2234 becomes roughly 1372×887, with no setting to change it), each app needs per-session approval, your terminal is excluded from screenshots, and the model receives pixels only — there's no accessibility-tree payload, which is why the docs' advice for unreadable text is to make it bigger in the app. We cover it end to end in our computer use guide.

Computer use gives the agent hands. It doesn't solve the direction problem: when you have found the bug, you still steer those hands by typing "the third button in the toolbar, no, the other toolbar."

Where every browser tool stops

Playwright and Chrome DevTools MCP see one thing: Chromium content. Their boundary is sharp.

Electron sits right on that boundary. Because every Electron app embeds Chromium, you can relaunch it with --remote-debugging-port and drive it over CDP — juri.dev's visual feedback loop for Electron apps shows the full setup and it works well. The trade-offs: you must control how the app launches, and CDP sees the web contents, not the native window around it. There's an accessibility wrinkle too — Electron apps don't build their AX tree until an assistive client asks, though setting the AXManualAccessibility attribute forces it on.

SwiftUI and AppKit apps have no DOM at all. twocentstudios' answer was to render views through Swift Snapshot Testing so Claude could inspect the PNGs — clever, and genuinely useful for iterating on a view in isolation, but it never sees the composed, running app with real data in it.

The iOS Simulator is a native macOS window; no browser tool can reach inside it. But it has a property most people don't know about: the Simulator bridges the simulated app's accessibility elements into the macOS AX tree, so standard AX APIs can read your iOS app's elements from the Mac side — xctree demonstrates this with nothing but public APIs. That bridge is the wedge for everything below. More in Claude Code and the iOS Simulator.

Giving Claude eyes on native apps in 5 minutes

Full disclosure: we build SupaMaus Lite, a macOS menu-bar app that exists because of everything above. It runs a local MCP server on 127.0.0.1:19741 (token-authenticated, on-device, no API keys) and serves captures to Claude Code, Codex, or any MCP client:

claude mcp add --transport http supamaus http://127.0.0.1:19741/mcp \
  --header "Authorization: Bearer <token>"

The app hands you this command with your token filled in, or agents/install.sh sets up the MCP config plus an agent skill for Claude Code and Codex in one shot.

Then hold Control+Option and move your mouse over anything — a SwiftUI window, an Electron app, the Simulator. A highlight trail follows the cursor; release, and the capture is saved. What the agent gets is the three-layer package, not just pixels:

  • Pixels: an annotated screenshot with your gesture redrawn as a gradient stroke, plus an explicit pixel-to-point mapping so coordinates survive Retina scaling.
  • Geometry: the element under your trail from the macOS AX tree — role, name, bounds in global points, center — and the full window text, including content scrolled out of view, with OCR fallback for pixel-only windows.
  • Intent: whatever you said while drawing, transcribed on-device by WhisperKit.

Every capture also mirrors to ~/.supamaus/latest/ (capture.png, context.md, manifest.json), so if your agent has no MCP support you can paste a path into any composer. Requires macOS 14+ on Apple silicon; $15 one-time at launch.

Pointing beats describing

The reason geometry matters is what linguists call deixis: "this button here" means nothing without the pointing. MIT's "Put-That-There" demo combined pointing and speech in 1980; forty-six years later, gesture-plus-voice pointing at a desktop agent is still mostly research territory rather than shipping product.

The data says structure plus pixels beats pixels alone. In the OSWorld ablations, an accessibility tree combined with a screenshot reached 69.2–71.8% relative task success versus 56.4% for Set-of-Mark screenshots alone — we dig into why in accessibility tree vs screenshots.

Pointing wires your intent into that structure. When you circle a control and say "this one," SupaMaus aligns words like this and here to segments of the drawn trail, then resolves the trail against the AX tree. In the iOS Simulator that resolution goes all the way down to your own accessibilityIdentifier (shipped July 2026, v1.0.69) — so instead of the agent squinting at a 1372px screenshot, it greps your codebase for save_button_primary and lands on the exact view.

A real loop: point, speak, fix, verify

Concretely, with an iOS app running in the Simulator:

  1. The save button's label truncates with long titles. Hold Control+Option, circle the button, and say: "this button truncates when the note title is long."
  2. In Claude Code, run /mcp__supamaus__look — the slash command Claude Code exposes for the server's look prompt.
  3. The agent receives the annotated screenshot, the element (role, bounds, center, accessibilityIdentifier), the window text, and your sentence. It greps for the identifier, finds the view, and fixes the truncation.
  4. After rebuilding, the agent calls capture_context to take a fresh look at the Simulator and confirm the label renders — no hands on your keyboard for the verification pass.

For anything that moves — an animation glitch, a flow that breaks on step three — triple-tap Control to start a live recording session instead: the mic stays on, you draw regions across apps and screens while narrating, and a second triple-tap finalizes one multi-region context with a shared transcript. The agent gets a timeline manifest and pulls only the frames it needs by millisecond offset via supamaus_get_frame. That on-demand model is the token story in reverse: instead of a 232K-token screenshot dump, the agent requests two or three frames sized within Claude vision's sweet spot (≤ ~1568px long edge, tokens ≈ width×height ÷ 750).

FAQ

Can Claude Code see my screen? Not by default. You can paste screenshots (Ctrl+V), enable the built-in computer use research preview on macOS (Pro/Max, screenshots downscaled to ~1372px), or connect an MCP server like SupaMaus Lite that delivers screenshots plus element geometry and window text on demand.

Does Claude Code work with the iOS Simulator? Yes, two ways. Computer use can click through the Simulator like a user. A capture-based approach reads the Simulator's accessibility bridge instead, resolving what you point at to real elements — including your own accessibilityIdentifier — which the agent can grep for in source.

Is this the same as computer use? No, and they compose. Computer use is hands: the agent drives your machine, one approved app at a time, seeing only downscaled pixels. Spatial context is eyes plus intent: you point at the problem, and the agent receives pixels, geometry, and your words in one package. Point with one, let the other do the clicking.

Does it need API keys? No. SupaMaus Lite is a local MCP server; transcription runs on-device via WhisperKit, and captures stay on your Mac. The one optional cloud assist (gesture-segment refinement via a Cloudflare Worker, no screenshots sent) is disclosed and can be disabled.

Won't screenshots blow up my token budget? They can — one Playwright MCP screenshot call was measured at 232K tokens. Claude bills roughly width×height ÷ 750 tokens per image, so right-sized captures cost a few thousand tokens, and pulling individual frames on demand from a recording keeps a long debugging session cheap.