Put-That-There (1980) already solved your AI agent's pointing problem

I build pointing software for a living, which means I spend an unreasonable amount of time watching AI agents click on things. The failure mode that haunts me is never the dumb one. The agent screenshots the app, reasons flawlessly about the layout, names the exact button it needs — "Save changes, bottom right of the modal" — then emits coordinates that land forty pixels away, on dead background. Nothing happens. The agent looks at the next screenshot, decides the modal probably closed, and confidently declares the task complete.

The plan was right. The point was wrong. That gap has a modern name — GUI grounding — and a modern scoreboard. On ScreenSpot-Pro, which tests click accuracy on dense professional screens, the July 2026 leaderboard has Claude Opus 4.8 at 87.9%, GPT-5.2 at 86.3%, and Gemini 3 Pro at 72.7%. The best pointing system money can buy misses roughly one click in eight on the screens where clicks matter most.

The field treats this as a new problem, born with vision-language models. It is the oldest problem in interactive computing. We have been studying what it takes to point at a thing on a screen — carefully, quantitatively, with benchmarks — since before screens had windows. And the literature keeps converging on one answer:

Don't make the pointer more precise. Make the target impossible to miss.

Everyone who ever tried to point at things hit the same wall. It's worth walking the lineage, because each stop maps onto something your agent is doing — or failing to do — right now.

Small targets are logarithmically punishing

In 1954, Paul Fitts had subjects tap a stylus back and forth between brass plates for the Air Force, and found that pointing behaves like a noisy information channel (Fitts, 1954, Journal of Experimental Psychology 47, 381–391). The time to acquire a target is:

MT = a + b · log₂(2A/W)

where A is the distance to the target and W is its width. The logarithmic term is Fitts's own index of difficulty, equation (1) in the paper. What it says is brutal in a specific way: every time you halve the target's width, you pay the same constant time increment, forever. Small targets aren't linearly harder. They are logarithmically punishing, and the punishment never stops compounding. (My favorite detail: Fitts's instructions told subjects to "emphasize accuracy rather than speed." Computer-use agents got the opposite memo.)

VLM grounding rides the same curve. Accuracy collapses on exactly the targets Fitts would predict — tiny toolbar icons on 4K professional screens — and the fix that works is the one his formula suggests. GUI-Eyes (January 2026) hits 44.8% on ScreenSpot-Pro with a 3B-parameter model by grounding coarsely, cropping, and re-grounding inside the crop. Zooming doesn't make the model smarter; it makes W bigger in the model's visual field. Dogfooding our own grounding harness taught me the same lesson: for small targets, a dumb zoom-refine loop beats clever full-frame ensembling every time. Fitts would not have been surprised.

The mouse exists because it won a benchmark

Here is the part of mouse history that rarely gets told: the mouse was not adopted because it felt right. It was adopted because it won a controlled selection experiment, run under NASA contract NAS 1-3988 at Stanford Research Institute and published as "Display-Selection Techniques for Text Manipulation" (English, Engelbart & Berman, IEEE Transactions on Human Factors in Electronics, March 1967).

The setup will look familiar to anyone who has read a grounding paper: present a target on a CRT, measure how fast and how accurately each candidate device can select it. The candidates were a light pen, a Grafacon tablet, a joystick in two modes, a knee-operated lever, and "an SRI-developed device known as a 'mouse'" — a wooden box over two orthogonal potentiometer wheels, developed, in the paper's own words, "in connection with this research." For experienced users the mouse won on both axes: 1.93 seconds per selection against 2.13 for the light pen and 2.87 for the joystick, with an error rate of 9.5% against the light pen's 20.9% and the joystick's 27.8%. The authors' conclusion: "for the more experienced subjects the mouse was both faster and more accurate than any other device — including the light pen."

(The knee lever, delightfully, was the fastest device for novices — it's the only one that doesn't cost you the time of moving a hand off the keyboard. Benchmarks have been embarrassing our intuitions from the start.)

We have been benchmarking pointing since 1967. ScreenSpot is the same experiment with a new species of subject — and the lesson of 1967 is that when a pointing device loses the benchmark, you don't lecture the device. You change the hardware.

Pointing is what makes pronouns possible

By 1980 the question had moved from how fast can you point to what does pointing mean. In a DARPA-funded room at MIT's Architecture Machine Group — the lab that would become the Media Lab — Richard Bolt built the demo that every multimodal agent pitch deck is still unknowingly re-drawing: "Put-That-There" (SIGGRAPH '80), implemented by Chris Schmandt and Eric Hulteen.

The setup: a "Media Room" about sixteen feet by eleven, a wall-sized projection screen, an Eames chair, and on the user's wrist a Polhemus magnetic sensor — a 0.75-inch cube on a watchband, tracked by a nutating magnetic field. The speech recognizer was an NEC DP-100: five words per utterance, a 120-word active vocabulary. You sat in the chair, pointed at the wall, and said "create a blue square there," or pointed at a ship on a map of the Caribbean and said "put that... there."

Two things about the paper read like they were written last week. First, Bolt understood that pointing is what gives pronouns meaning. "'That' is thus defined as whatever is pointed out; effectively, it is 'ostensively defined,'" he wrote, and then, beautifully: "In our simple graphics world, what anything is, is in a subtle and interesting sense, where it is." Pronouns work in conversation because humans point; linguists call this deixis. Strip the pointing away and every "that" becomes a paragraph of description — which is precisely the tax you pay today when you type "the third card in the settings pane, no, the other one" into an agent's composer.

Second, Bolt never assumed either channel would be reliable on its own. A 120-word recognizer couldn't afford that assumption, and free-hand pointing at a wall sixteen feet away is wobbly. So the system was built around "actions in one modality amplifying, modifying, disambiguating, actions in the other." Speech tells you when to sample the pointing vector — the spoken "there," in Bolt's words, "serves as a 'voice button' for the x,y cursor action of the pointing gesture" — and pointing tells you what the words refer to. Redundant, mutually disambiguating channels. That is, verbatim, the multimodal grounding problem of 2026, solved for its constraints with a magnetic cube on a wrist, forty-six years ago.

Three detours: bigger cursors, bigger targets, stickier space

Between Bolt and the technique this essay is really about, three ideas each earned a CHI paper by attacking Fitts's W from a different angle.

Kabbash and Buxton (1995) inverted the problem with the "Prince" technique: make the cursor an area instead of a point. Fitts's law holds if you substitute the cursor's width for the target's — suddenly a one-pixel target behaves like a fat one. The catch: when two targets fall inside the area cursor at once, the technique has no answer.

McGuffin and Balakrishnan (2002) grew the targets instead: expanding targets that balloon as the cursor approaches. The striking result is that performance is governed by the target's final size even when expansion begins after 90% of the distance has been covered. You get the benefit of a big target while paying the screen real estate of a small one — until neighbors start overlapping.

Blanch, Guiard and Beaudouin-Lafon (2004) went a level more abstract with semantic pointing: decouple motor space from visual space by warping the control-display ratio, so important targets are big where your hand lives even if they're small where your eyes live. Fitts's law predicts performance from the motor-space W and A. But every target between you and your goal becomes a speed bump.

Same move, three costumes: don't sharpen the pointer, inflate the target. And all three degrade in exactly the environment where you need them — a dense, cluttered screen.

The screen becomes a Voronoi diagram

Then in 2005, Tovi Grossman and Ravin Balakrishnan published the bubble cursor (CHI 2005) and quietly closed the problem. The bubble cursor is an area cursor that continuously resizes itself "such that there is always exactly one target inside the hotspot." Never zero — you cannot point at nothing. Never two — there is nothing to disambiguate.

The geometric consequence is the part I find genuinely beautiful. In the authors' words, the bubble cursor "essentially divides up the total space in which all targets reside into regions, such that there is exactly one target inside each region, and that target is the closest target to any point within that region... In mathematical terms, this is referred to as a Voronoi diagram." Every pixel on the screen now belongs to some target. Empty space is abolished. Each target's effective width becomes its entire Voronoi cell — W inflated to the maximum extent geometry allows, without moving a single pixel of the visual layout.

And Fitts still governs it. Selection times fit Fitts's law with the effective width substituted for the actual width, at r² above 0.95 — the combined fit in their first experiment was 0.9858. In dense 2D layouts the bubble cursor averaged 0.93 seconds per selection against 1.08 for a point cursor and 1.41 for object pointing, while also cutting errors from 2.98% to 1.58%. Faster and more accurate, in the regime where every previous technique broke down. The bubble cursor didn't make pointing more precise. It abolished the possibility of missing.

Your agent is re-living all of this

Here is why I keep these PDFs within reach: every working technique in 2026-era agent grounding is one of these ideas wearing a trench coat.

Zoom-refine grounding is Fitts's W. Coarse-to-fine pipelines like GUI-Eyes work because cropping enlarges the target relative to the model's visual field. You cannot prompt your way down the index-of-difficulty curve; you can only move along it by changing the geometry.

Accessibility-tree hit-testing is the Voronoi snap. When an agent's predicted coordinate is resolved against the platform accessibility tree, the 40-pixel miss stops mattering: the coordinate collapses onto an element with a role, a name, and bounds — the nearest cell in the diagram, not a naked pixel. The OSWorld ablations put accessibility tree plus screenshot at 69.2–71.8% relative task success versus 56.4% for annotated screenshots alone. Pixels tell the agent what things look like; the tree tells it where things are — I've written a longer comparison in accessibility tree vs. screenshots.

Gesture-plus-voice deixis is Put-That-There. When a human points at a region and says "this is misaligned," the two channels disambiguate each other exactly as Bolt designed — the gesture scopes the pronoun, the words timestamp the gesture. In 2026 this mostly still lives in research prototypes like Speech-to-Spatial and GesPrompt, which is strange, because it is the highest-bandwidth grounding channel a desktop has.

Full disclosure: that last one is my bet. I build SupaMaus Lite, a macOS menu-bar app that is, honestly described, the Put-That-There stack pointed at coding agents. You hold a modifier and circle something on screen while talking; on-device Whisper transcription aligns your spoken "this" and "here" to segments of the drawn trail — Bolt's voice button, with the Polhemus cube replaced by the mouse you already hold — and the macOS accessibility tree resolves the gesture to an actual element with role, name, and bounds. The agent receives the resolved target over local MCP instead of guessing coordinates from a screenshot. The human does the pointing; the agent gets the Voronoi cell, not the pixel. More on why agents need this channel at all in why coding agents need spatial context.

The 46-year-old lesson

There's a pattern in how the industry is attacking grounding right now: bigger models, more vision pre-training, finer-grained coordinate heads. Sharpen the pointer. It works — the ScreenSpot-Pro leaderboard climbs a few points a quarter — and Fitts's logarithm quietly promises that it will keep working more and more slowly, because the hard targets are small and the punishment compounds.

The HCI lineage spent five decades on the same wall and came back with the other answer every time. The 1967 study didn't train better light-pen users; it changed the device. Bolt didn't wait for a perfect speech recognizer; he added a second channel that made perfection unnecessary. Grossman and Balakrishnan didn't build a steadier hand; they redrew the screen so that every point in it belongs to something.

Forty-six years after a man in an Eames chair said "put that there" to a wall and the wall understood him, our agents are still pointing at pixels and hoping. The research is sitting right there, and it reads like a memo addressed to us. Don't make the pointer more precise. Make the target impossible to miss.