Early tests
Finding where to click
Give the model a screenshot and an instruction, such as “click Save”.
The model can point at a button directly instead of spelling out its position as text. It still needs to understand the screen and what you asked it to do.
On 30 public screenshots, the model found 25 targets when it combined neighbouring image patches, compared with 16 when choosing a single patch. Recognising when a requested button is missing remains an open problem.
Technical details and sources
Small follow-up: rejecting missing targets. This diagnostic reuses eight examples; general target-presence detection remains unvalidated.
Public screenshot results and all predicted points
Public screenshot test protocol · Implementation review and resolution-cap correction
An action head
We compare coordinate regression, visual-patch selection, and region selection with a local offset. Multiple valid targets make a single averaged coordinate unreliable; no-target cases need their own output.
Image encoding and transformer input processing still cost time. One forward pass does not mean one operation.
A concrete reference
Microsoft’s GUI-Actor adds an attention-based action head to Qwen vision-language backbones. It aligns a task token with visual patches, then derives a point from the selected region. This is related work and a reference implementation, not an invention claimed by this lab.
Our pilot runs a direct pointer path and a paired pass with one added vocabulary projection. That isolates output-head overhead; it is not a text-coordinate or JSON comparison. A small local UI fixture tests integration and coordinate mapping; broader grounding quality needs held-out applications.
Build evidence without live account actions
The coordinate workbench creates local, explicit target annotations. It does not claim to infer targets from uploaded images. Recorded model predictions can be inspected separately from manual ground truth.