Recorded model outputs
Where would it click?
Replay a real screenshot and see the point the model chose.
Live OCR and text matching; a simpler alternative to the GUI-Actor model used in the recordings.
Back to finding where to click · Try a small live image model
These are saved GUI-Actor model outputs on 30 real ScreenSpot screenshots. No live vision model runs here. Choose a screenshot, read the task, and compare the point with the target.
Task: Loading…
Loading the recorded points…
Try the cautious review rule
On these 30 cases, this rule allowed 8 clicks: 7 correct and 1 wrong. It sent 22 to review, including 18 points that were already correct. Review is unresolved, not a successful click.
Combining nearby image patches hit 25 / 30 targets; using one patch hit 16 / 30. Both used the same model pass. Every case ran all 28 layers and generated zero text tokens.
Methods, review-rule results and limits · ScreenSpot dataset · GUI-Actor model
A harder desktop test found 8 of 25 targets with the same model. See the new test, including infeasible requests.