Early tests
Finding where to click
Give the model a screenshot and an instruction, such as “click Save”.
Live OCR and text matching; a simpler alternative to the GUI-Actor model used in the recordings.
The model can point at a button directly instead of spelling out its position as text. It still needs to understand the screen and what you asked it to do.
On 30 public screenshots, the model found 25 targets when it combined neighbouring image patches, compared with 16 when choosing a single patch. Recognising when a requested button is missing remains an open problem.
Where this could be useful
Return a location an app can use. Finding the point and safely carrying out the action are separate tasks.
Screens and images
- Point to an app controlFind a button without writing coordinates.
- Assistive pointingTurn a requested control into a location.
- Check changing interfacesLocate a control after a layout changes.
- Suggest an image annotationMark a starting point for a human editor.
Medical signals and scans
- Mark regions in an MRIReturn a mask that a reviewer can inspect.
- Medical-image landmarks and measurementsSuggest editable points for a specialist.
Technical details and sources
Small follow-up: rejecting missing targets. This diagnostic reuses eight examples; general target-presence detection remains unvalidated.
Public screenshot results and all predicted points
Public screenshot test protocol · Implementation review and resolution-cap correction
An action head
We compare coordinate regression, visual-patch selection, and region selection with a local offset. Multiple valid targets make a single averaged coordinate unreliable; no-target cases need their own output.
Image encoding and transformer input processing still cost time. One forward pass does not mean one operation.
A concrete reference
Microsoft’s GUI-Actor adds an attention-based action head to Qwen vision-language backbones. It aligns a task token with visual patches, then derives a point from the selected region. This is related work and a reference implementation, not an invention claimed by this lab.
Our pilot runs a direct pointer path and a paired pass with one added vocabulary projection. That isolates output-head overhead; it is not a text-coordinate or JSON comparison. A small local UI fixture tests integration and coordinate mapping; broader grounding quality needs held-out applications.
Build evidence without live account actions
The coordinate workbench creates local, explicit target annotations. It does not claim to infer targets from uploaded images. Recorded model predictions can be inspected separately from manual ground truth.
See it work
Explore 30 real screenshots · Run the small image-to-action model
A harder desktop test found 8 of 25 targets with the same model. See the new test, including infeasible requests.