Use case · Related prototype
Point to an app control
Find a button without writing coordinates.
- Screenshot + click Save
- Locate the target
- Return a point
An automation tool could locate Save, Search or another requested control in a screenshot. The surrounding application decides whether and when to perform the click.
Where the small model fits
A vision model encodes the screen and instruction. A spatial head selects a region or point directly; a no-target result is needed when the control is absent.
What would need to work
Test held-out applications, ambiguous controls and missing targets. Our recorded model predictions and live text-matching demo are different implementations, and neither establishes reliable end-to-end automation.
Try the related work
Live OCR and text matching, not the vision model in the recorded experiment.