Research results
Finding the right point
Return a location directly, then check whether it actually hits the target.
Dataset: ScreenSpot · 30 public screenshots, with a requested target and its marked location.
- Screenshot + request
- Score the image patches
- Return a point (x, y)
| How the point is chosen | Targets hit / 30 |
|---|---|
| Highest-scoring single patch | 16 |
| Combine neighbouring high-scoring patches | 25 |
Combining nearby patches helped on this small sample. The model still ran all 28 transformer layers: 0% skipped. It generated no text tokens; that does not make image understanding free.
What if the button is missing?
On two local screenshots, a confidence cutoff rejected both missing targets and kept all six present targets. It also kept one wrong point. Eight reused examples are too few to establish reliable rejection.
See the six local recorded targets
Model, subsets and limitations
This reproduces Microsoft’s pretrained GUI-Actor 2B pointer head. We did not train a new vision model. Both patch-selection methods share the same model pass, so this is a location comparison, not a measured speed advantage over JSON.
The missing-target check uses the other screenshot to choose each cutoff. It is a confidence diagnostic on a repeated interface, not a learned presence classifier. The 30 ScreenSpot cases all have visible targets.
Every public screenshot result · Sample selection · Missing-target diagnostic · Original local run