Fresh benchmark stress test
Click, or hold the action?
A real-screen test that includes infeasible requests.
Live OCR and text matching; a simpler alternative to the GUI-Actor model used in the recordings.
Back to finding where to click · Explore the earlier screenshot demo
Sometimes the requested button is not on the screen. We tested 50 new real screenshots: 25 with a target and 25 requests that the dataset authors label infeasible.
| Method | Correct clicks / 25 | Infeasible but clicked / 25 | Present but withheld |
|---|---|---|---|
| Always return a point | 8 | 25 | 0 |
| Hold low-confidence clicks | 2 | 0 | 21 |
The confidence cutoff allowed 2 correct clicks, 2 wrong clicks on present targets, and 0 clicks on infeasible requests. It also withheld 21 requests with a real target, including 6 points the model had already located correctly.
Holding a click is not proof that a target is missing. The output is UNCERTAIN; a withheld present request remains unfinished. This is a balanced stress test, not a production success rate.
See the first test case
Request: Click on "Hide applications"
The blue point is the model’s proposed click. The dataset labels this request infeasible. The confidence rule returned UNCERTAIN. This is a saved result, not live inference.
First request in the sealed order, not selected by its outcome. Open the original screenshot.
Does a closer look help?
For 10 requests chosen before inference, we cropped around the model’s predicted point and ran it again. Correct locations changed from 2 / 10 to 3 / 10: 1 corrected and 0 lost. This costs a second model pass; the crop never uses the correct target.
All four readout variants
| Method | Correct clicks | Wrong target | Infeasible but clicked | Present but withheld | Infeasible and withheld |
|---|---|---|---|---|---|
| Highest-scoring patch | 3 | 22 | 25 | 0 | 0 |
| Combine nearby patches | 8 | 17 | 25 | 0 | 0 |
| Combine + confidence cutoff | 2 | 2 | 0 | 21 | 25 |
| Combine + grid-adjusted cutoff | 2 | 2 | 0 | 21 | 25 |
What actually ran?
GUI-Actor-2B-Qwen2-VL processed every screenshot and instruction on a CPU. Its trained pointer head scores image patches and returns a location without generating text. All 28 language-model layers ran, along with the vision encoder. No layers were skipped.
The two cutoffs were selected using the earlier 30 ScreenSpot examples, before seeing any new OSWorld-G model outputs. The new benchmark includes Ubuntu desktop screens outside that gate-development set; model-training overlap is unknown. There is no dedicated NOT_FOUND class in this pointer head.
Inputs use the authors’ refined instructions and a maximum of 576 visual tokens. Timing was diagnostic while other experiments ran, so we make no speed claim. These 50 requests do not constitute a full benchmark evaluation.
Method, dataset, every result and limitations · Protocol fixed before inference · OSWorld-G dataset