← Blog

Accessibility tree vs screenshots: how AI agents should see macOS and iOS apps

10 min readMilind Soni

You tell your coding agent to fix a misaligned button in your Mac app. It takes a screenshot, squints at the pixels, edits some SwiftUI, screenshots again, and announces success — and the button is exactly as broken as before. The question under that failure is worth asking precisely: when an AI agent needs to see a macOS or iOS app, should it read pixels or structure?

Almost everything written about this assumes the web, where the DOM exists and Playwright can hand an agent a tidy accessibility snapshot of the page. Native apps play by different rules.

The accessibility tree is the closest thing a native app has to a DOM: a queryable, text-based description of every element on screen — role, name, value, and exact frame — that costs a few kilobytes instead of a few megapixels.

Three ways an agent can read a UI: pixels, AX tree, DOM

An agent observing a UI has at most three channels.

Pixels. Capture a screenshot, send it to a vision model. Universal — it works on anything that draws to the screen, including video, canvas, and custom-rendered views. Also the most expensive channel, and the only one where the model can be wrong about what it saw.

The accessibility (AX) tree. Every macOS app publishes a tree of elements to the system accessibility API — the same channel VoiceOver uses. Structured, exact, and cheap to read, but it only contains what the app chooses to expose.

The DOM. On the web this is the richest channel: selectors, computed styles, event listeners, a full DevTools protocol. It is also the channel that simply does not exist for native apps. AppKit and SwiftUI keep their view hierarchies private; there is no querySelector for a Mac app, no Elements panel, no Chrome DevTools Protocol to attach to. The only structured representation an external process can read is the AX tree.

That asymmetry drives everything else in this article. On the web you choose among three options; on macOS and iOS you choose between pixels and AX — and the interesting engineering is deciding which parts of the spatial context each channel should carry.

What the macOS accessibility tree actually contains

Query any app through the AX API in ApplicationServices and you get a tree of AXUIElement nodes. The attributes that matter for agents:

  • Role — AXButton, AXTextField, AXTable, AXMenuItem, AXStaticText. What kind of thing this is.
  • Name and description — the human-readable label: "Send reminder", "Search".
  • Value — current state: text field contents, checkbox on or off, slider position. Notably, the AXValue of a text area is the whole text, including content scrolled out of view — something no screenshot can show.
  • Frame — position and size in global screen coordinates, top-left origin, measured in points rather than pixels. On a Retina display one point is two pixels, and confusing the two puts every computed click far off target.
  • Actions — AXPress, AXShowMenu, and friends: what can be done to the element.

A realistic slice of the tree for an invoicing app looks like this:

AXWindow "Invoices — Acme Books"  frame={0, 25, 1512, 918}
└── AXSplitGroup
    ├── AXTextField "Search"  value="acme"  frame={24, 74, 220, 28}
    ├── AXButton "Send reminder"  enabled=true  frame={1301, 74, 139, 28}
    └── AXTable
        └── AXRow  selected=true
            ├── AXStaticText "INV-2041"
            └── AXStaticText "$1,240.00"

That is the whole trick: a dozen lines of text that tell an agent the search field currently contains "acme", the reminder button is enabled, and its center is at (1370, 88) — no vision model required.

The API also does hit-testing. AXUIElementCopyElementAtPosition takes a global coordinate and returns the element under it, respecting window z-order:

AXUIElementRef element = NULL;
AXError err = AXUIElementCopyElementAtPosition(
    AXUIElementCreateSystemWide(),
    1370.0f, 88.0f,   // global points, top-left origin
    &element);

This is how "the thing under the cursor" becomes a named, typed element instead of a pixel neighborhood. The one prerequisite: the reading process needs the Accessibility permission under System Settings → Privacy & Security, granted once.

The failure mode nobody talks about: "Perfect!" about a broken UI

Every screenshot-driven agent loop has the same weak link: the model grades its own homework by looking at pixels. It makes an edit, captures the screen, and pattern-matches for success — with a strong prior that its own change worked. The result is the most corrosive failure mode in agentic UI work: the agent cheerfully reports "Perfect! The alignment is fixed" over a screenshot where the button is still three points off, the toggle is still gray, or the error toast is sitting right there in the corner.

This is not carelessness; it is measurable perception error. On ScreenSpot-Pro, the benchmark for grounding targets in dense professional UIs, the best frontier model as of July 2026 — Claude Opus 4.8 at 87.9% — is state of the art and still wrong about one target in eight. Small text, low-contrast states, and near-identical icons are exactly where vision models are weakest, and exactly where UI bugs live.

Structured text removes the perception step. AXValue on the search field returns the string "acme", not a guess about what those forty pixels probably say. enabled=false is a fact, not an inference from grayish rendering. When verification is a string comparison against the AX tree instead of a vibe check against a JPEG, confident wrongness gets much harder: the check either passes or it does not.

Token cost and latency, with numbers

Screenshots are not just occasionally wrong; they are expensive on every loop iteration.

The pathological case first: one measured browser_take_screenshot call through Playwright MCP consumed 232,000 tokens, and a 2026 comparison of browser-automation tools puts a typical Playwright MCP session around 114K tokens. That is an entire context window spent on looking.

It does not have to be that bad. Claude's vision docs give the formula — tokens ≈ width × height / 750, with images ideally at most ~1,568px on the long edge. A 16-inch MacBook Pro screen at native 3456×2234 is why naive capture explodes budgets; Claude Code's built-in computer use downscales it to roughly 1372×887, which works out to about 1,600 tokens per screenshot. We looked at how popular servers handle this in our screenshot MCP roundup.

Now the AX side. The snapshot in the previous section is under half a kilobyte. A generous AX snapshot of a full window runs a few kilobytes of text — on the order of a few hundred to a thousand tokens, for a description that is exact rather than approximate. Two orders of magnitude separate the pathological screenshot from the structured read.

Latency has the same shape. macos-use, an AX-first automation tool, reads accessibility state in about 50ms — fast enough to diff the tree before and after every action. A screenshot has to be captured, scaled, encoded, and pushed through a vision encoder before the model can start thinking. Multiply either path by the dozens of observations a real task takes and the two approaches land in different cost universes.

Where screenshots still win

None of this makes the AX tree sufficient, because the tree describes what the app claims and pixels show what actually got drawn.

The AX tree cannot represent:

  • Layout and spacing quality. Frames tell you a button sits at x=1301; they do not tell you it looks cramped, misaligned with its row, or eight points from where the design says it belongs.
  • Color, contrast, and theming. Dark-mode bugs, illegible text on a tinted background, the wrong brand color — all invisible in the tree.
  • Rendering bugs. Clipped or truncated labels, blurry Retina assets, views drawn on top of each other. An element can have a perfect frame in the tree while being completely covered by another view.
  • Anything custom-drawn. Canvas, games, video, WebGL, charting surfaces — often published to accessibility as a single opaque group, or nothing at all. Pixels, with OCR as a fallback, are the only channel.

There is a deeper point here: verification of visual work is itself visual. If the task was "make it look right," no amount of structure substitutes for looking — which is why we wrote about giving Claude Code eyes in the first place. The honest framing: the AX tree verifies state, screenshots verify appearance, and most UI bugs involve both.

The right answer is both: what the research says

This is not a matter of taste; it has been measured. The OSWorld benchmark's ablations compared input representations for computer-use agents: the accessibility tree combined with a screenshot reached 69.2–71.8% relative task success, versus 56.4% for Set-of-Mark screenshots alone. Structure plus pixels beats pixels alone by double digits.

The MCP tool spec makes hybrid delivery straightforward: a single tool result can carry text and base64 images together, so one capture can return the picture and the facts in one payload.

A good hybrid capture in practice has three parts:

  1. A screenshot, downscaled to the model's sweet spot, for layout and rendering truth.
  2. The AX slice for the relevant region: roles, names, values, frames.
  3. An explicit bridge between the two — element bounds expressed in the screenshot's coordinate space, with the pixel-to-point scale stated, so the model never guesses Retina math.

Disclosure: this is what we build. SupaMaus Lite is a macOS menu-bar app exposing a local MCP server; highlight a region of any app and the agent gets the annotated screenshot, the AX elements under your gesture (role, name, bounds, center), and the window's full text over accessibility, with OCR fallback for pixel-only windows. Peekaboo is a solid open-source take on the same combination — screenshots plus AX over MCP — and macos-use is the one to watch if you want AX diffs as the primary channel.

Why web-focused guides don't transfer to native apps

Search "accessibility tree vs screenshots" today and every result assumes a browser. The advice — use ARIA roles, take Playwright accessibility snapshots, fall back to CSS selectors — quietly depends on infrastructure native apps do not have. There is no DevTools to open, no selector engine to query, no console or network tab. The AX API plus the Accessibility permission is the entire external interface, and agent tooling has to be built against it directly.

Two native-specific gotchas that web guides never mention:

Electron apps have a lazy AX tree. Electron does not construct its accessibility tree until assistive technology asks for it, so a naive AX read of an Electron app can come back nearly empty. The Electron docs document the escape hatch: third-party tools can force the tree on by setting the AXManualAccessibility attribute programmatically:

AXUIElementRef appRef = AXUIElementCreateApplication(pid);
AXUIElementSetAttributeValue(appRef,
    CFSTR("AXManualAccessibility"), kCFBooleanTrue);

The iOS Simulator bridges into the macOS tree. You do not need separate tooling for iOS work: the Simulator republishes the simulated app's accessibility elements into the macOS AX tree, so the same APIs that read Finder read your SwiftUI app — including the accessibilityIdentifiers you set for UI tests. Tools like xctree and the AXe CLI demonstrate the bridge with standard macOS AX calls, and SupaMaus Lite uses it to resolve the element you point at inside the Simulator (shipped July 2026, v1.0.69).

FAQ

Do AI agents still need screenshots if they can read the accessibility tree?

Yes. OSWorld's ablations show accessibility tree plus screenshot (69.2–71.8%) beating Set-of-Mark screenshots alone (56.4%), and the tree cannot represent layout quality, color, or rendering bugs at all. Use the tree for state and grounding, pixels for appearance.

How do I read the macOS accessibility tree programmatically?

Through the AXUIElement API in ApplicationServices — AXUIElementCopyAttributeValue for attributes, AXUIElementCopyElementAtPosition for hit-testing — after granting the reading app Accessibility permission. Xcode's Accessibility Inspector is the fastest way to explore what an app exposes before writing code.

Why is the accessibility tree empty for Electron apps?

Electron builds its AX tree lazily, only when assistive technology connects. Set the AXManualAccessibility attribute on the app's AXUIElement to force it on, as documented in the Electron accessibility docs.

Does this work for iOS apps?

In the Simulator, yes: the simulated app's accessibility elements are bridged into the macOS AX tree, so any macOS AX client — including MCP servers like SupaMaus Lite — can read roles, labels, frames, and accessibilityIdentifiers without extra instrumentation.