Claude Code computer use on macOS: what it does, where it stops, and what to add

"Isn't screen context built into Claude Code now?" Since computer use landed in the CLI, that's the first question every macOS developer asks — and the honest answer is no, because computer use solves a different problem than the one you probably have.

This is the deep guide: what actually shipped, how the perception loop works, what early testers found, and where you still need something else. Everything below is verified against the official docs as of July 2026.

What shipped

Computer use in Claude Code is a research preview on macOS, launched in March 2026. It requires a Pro or Max plan (not available on Team or Enterprise), Claude Code v2.1.85 or later, and an interactive session — it doesn't work in non-interactive -p mode, and it's not available through Bedrock, Google Cloud's Agent Platform, or Microsoft Foundry. You authenticate through claude.ai or it doesn't appear at all.

It ships as a built-in MCP server called computer-use, off by default. You enable it per project from the /mcp menu, then grant two macOS permissions the first time Claude reaches for the screen: Accessibility (lets Claude click, type, and scroll) and Screen Recording (lets Claude see the screen).

The permission model is more granular than most people expect:

• Per-app, per-session approval. Enabling the server grants nothing by itself. The first time Claude needs a specific app, a prompt lists which apps it wants, any extras like clipboard access, and how many other apps will be hidden. Approvals expire when the session ends.
• Sentinel warnings. Apps with broad reach get flagged before you approve: terminals and IDEs are marked "equivalent to shell access," Finder "can read or write any file," System Settings "can change system settings."
• Capability tiers. Browsers and trading platforms are view-only, terminals and IDEs are click-only, and everything else gets full control.

Claude also treats screen control as a last resort. The documented escalation order is: use an MCP server if one exists, use Bash if it's a shell task, use Claude in Chrome for browser work, and only then fall back to computer use.

How it works under the hood

The core loop is screenshot, act, verify. Claude captures the screen, decides on an action, synthesizes the input event through the Accessibility API, then screenshots again to check the result. Every action is a full round-trip through the model.

Three implementation details matter in practice.

Apps are hidden while Claude works. When Claude takes control, every visible app you haven't approved gets hidden, and they're restored when the turn ends. Your terminal stays visible so you can watch — but it's excluded from screenshots, so Claude never sees its own output and on-screen prompts can't feed back into the model. A machine-wide lock means only one session can drive the computer at a time (and since v2.1.195, the lock is held until the session exits, not just until the task finishes). Pressing Esc anywhere aborts immediately, and the keypress is consumed so prompt injection can't use it to dismiss dialogs.

Every screenshot is force-downscaled. A 16-inch MacBook Pro captures at its native 3456×2234 and Claude Code downscales it to roughly 1372×887 before the model ever sees it. There is no setting to change this. That's about 40% of native pixel dimensions — an 11pt menu label that renders at ~22 physical pixels on Retina reaches the model at roughly 9 pixels tall. This is why the docs' own advice for unreadable text is to increase the text size in the app, not to change display resolution. It also tracks Claude's vision guidance of keeping images under ~1568px on the long edge; at tokens ≈ width×height/750, each 1372×887 frame costs about 1,600 tokens, and a multi-step task takes many frames.

Perception is pixels, not structure. The Accessibility permission is used to send input, not to read the UI tree into the model. Nothing in the docs describes the accessibility tree reaching the model — the "make the text bigger" advice only makes sense if reading happens from the downscaled screenshot. More on why that matters in accessibility tree vs screenshots.

What early testers report

The early reports are more positive than you'd expect for a research preview, with consistent caveats.

Vanja's hands-on (March 2026) had Claude create a note in Apple Notes "first try. No errors" and navigate System Settings across multiple apps without issues. His verdict: "simple, predictable UIs work great. Complex multi-step workflows with authentication, dynamic content, or unusual UI patterns? Inconsistent." He also flags cost: because every action is a screenshot round-trip, "multi-step tasks can burn through hundreds of thousands of tokens."

AI Tool Briefing's review (March 2026) measured roughly 70–75% task completion on structured workflows, with each screenshot-analyze-act cycle taking a few seconds — a 30-click task ran 3–4 minutes. Their failure taxonomy is useful: misclicks on ambiguous or overlapping elements, dark mode and non-standard layouts tripping it up more than stock interfaces, and no undo awareness when something goes wrong. Final rating: 3.7/5, "genuinely useful, clearly unfinished."

That's a fair summary. For a first public cut of desktop control, first-try success on native app tasks is genuinely impressive, and the build-launch-click-verify loop for apps you're writing is the standout use case. The weaknesses cluster exactly where the downscaled-screenshot architecture predicts: small text, dense UI, and anything that requires knowing which element rather than roughly where.

Computer use is control; spatial context is communication

Here's the distinction that answers the question at the top of this article:

Computer use lets the agent act on your Mac. Spatial context lets you show your Mac to the agent. You need the second before the first is useful.

Computer use is an output channel — hands. It answers "how does the agent do things on my machine?" It does not answer "how do I tell the agent which thing I mean?" When you say "this dropdown clips under the toolbar," computer use gives Claude the ability to eventually click that dropdown, but it still has to find it from your prose plus a 1372px screenshot of your whole desktop. There is no channel for you to point.

That inbound channel — the pixel you're staring at, the element under your cursor, the sentence you'd say while pointing — is spatial context. It's a separate problem with a separate solution, and it's the one most "give my agent screen access" requests are actually about. If your pain is "Claude can't see what I see," you want eyes, not hands.

When to use which

| Your task | Reach for | Why | | --- | --- | --- | | Automate a GUI-only tool (no CLI, no API) | Computer use | This is exactly what it shipped for; nothing else can reach it | | Build-and-verify a native app you're writing | Computer use | The compile-launch-click-screenshot loop in one conversation is the killer feature | | UI iteration ("move this 8px, try again") | Spatial context capture | You need to point at the element and narrate intent; control comes after | | Debugging "this looks wrong right here" | Spatial context capture | The agent needs element identity and your narration, not the ability to click | | Pointing at one specific element | Spatial context (AX-grounded) | Computer use has no inbound pointing channel at all | | Browser work | Claude in Chrome or a DOM-level tool | Browsers are view-only under computer use anyway | | Grabbing a screenshot into context | A screenshot MCP server | Full control is overkill for perception-only tasks |

The pattern: computer use wins when the bottleneck is acting. When the bottleneck is explaining — which element, which state, what you meant — you need a context tool.

Using both together

Disclosure: we build SupaMaus Lite, a macOS menu-bar app that serves spatial context to any MCP client, so this section describes our own product.

The two compose cleanly because they sit on opposite sides of the loop. You hold Control+Option and trace over the broken dropdown while saying "this clips under the toolbar when I open it." SupaMaus captures the annotated screenshot at full fidelity with an explicit pixel-to-point mapping, aligns "this" to the drawn trail, and reads the macOS accessibility tree at that spot — so Claude receives the element's role, name, and exact bounds instead of guessing from a downscaled frame. In Claude Code that arrives via /mcp__supamaus__look or the standard MCP tools.

Then computer use takes over as hands: Claude patches the code, relaunches the app, clicks the dropdown, and screenshots the fix. The division of labor mirrors what the research says about perception — OSWorld's ablations found accessibility tree plus screenshot beats screenshot-style grounding alone (69.2–71.8% vs 56.4% relative task success). Computer use brings the screenshot half and the hands; an AX-grounded capture brings the structure and the pointing.

FAQ

Is computer use available in Claude Code? Yes — as a research preview on macOS for Pro and Max plans, in Claude Code v2.1.85 or later, interactive sessions only. Enable the built-in computer-use MCP server from the /mcp menu, then grant the Accessibility and Screen Recording permissions macOS asks for. It's not available on Team or Enterprise plans or through third-party providers like Bedrock.

Does computer use see the accessibility tree? Not as far as the documentation describes. The Accessibility permission is used to synthesize clicks, keystrokes, and scrolls; what the model sees is downscaled screenshots. The docs' advice to enlarge in-app text when Claude misreads it confirms perception is pixel-based. See accessibility tree vs screenshots for why the difference matters.

Why does Claude misread small text? Every screenshot is downscaled before reaching the model — a Retina 16-inch MacBook Pro goes from 3456×2234 to about 1372×887, with no setting to change it. Small UI text lands at a handful of pixels tall, which is below reliable reading size. The official workaround is increasing text size in the target app; capture tools that deliver full-resolution crops of the region you care about sidestep the problem entirely.

Can I point at an element with computer use? No. Computer use is one-directional — the agent acts on your screen, but there is no gesture, cursor, or selection channel from you back to the agent. To say "this one," you either describe the element in prose or use a spatial-context tool that captures your pointing gesture and resolves it against the accessibility tree.