Reference
Earlier combined results
The previous report, preserved as a reference. Start with the topic pages for shorter explanations.
Preference learning
83 calculation checks passed. The model can learn simple made-up preferences, but we have not yet shown that people prefer its suggestions.
Stopping early
600 real messages from an online chatbot. Stopping halfway gave three fewer correct answers overall, but missed seven more toxic messages.
| 600 messages | All 24 layers | Stop at 12 |
|---|---|---|
| Correct answers | 522 | 519 |
| Toxic messages missed | 19 of 81 | 26 of 81 |
| Total time | 231.8 s | 117.0 s |
The chosen rule always stops at layer 12; it does not detect readiness per message. It failed our reliability checks. Both methods use trained classifiers. The messages are archived ToxicChat data, not a live moderation trial.
Time: median of three complete 600-message passes per method on this CPU, including input preparation. The full-depth comparison returns one trained SAFE/BLOCK token. Model loading is excluded.
Finding where to click
5 of 6 visible targets found on two example screens. But when a target was missing, the model still guessed a point.
Public screenshot follow-up: 25 of 30 targets found using connected regions, compared with 16 using a single patch. See every result.
Test setup, measurements and limitations
Exploratory results, not deployment claims. These runs establish working mechanisms and expose failure cases. They do not validate arbitrary policies, human preferences or general GUI control.
Preference learning: mechanics pass, assumptions matter
The browser solver passes 83 fixtures. On 80 random bounded edits, its largest squared-distance difference from independent SciPy optimization was 1.6e-13 (tolerance 1e-7). This verifies latent geometry, not perceptual minimality.
| Synthetic test | Mean, five seeds | Observed seed range |
|---|---|---|
| Linear utility, 20 ratings | 85.3% | 83.3%–88.2% |
| Linear utility, 50 ratings | 94.7% | 92.5%–97.8% |
| Linear utility, 100 ratings | 95.7% | 92.5%–97.6% |
| Disconnected utility: linear head (balanced accuracy) | 50.3% | 49.9%–51.7% |
| Disconnected utility: quadratic head (balanced accuracy) | 85.5% | 84.2%–86.9% |
Seed ranges are descriptive, not confidence intervals. The disconnected-utility test uses a different label function from the linear test.
| Sampling at 80 labels | Held-out accuracy | Observed seed range |
|---|---|---|
| random | 93.7% | 90.6%–96.6% |
| uncertainty | 98.4% | 97.4%–99.0% |
| mixed | 98.2% | 97.5%–98.9% |
This pilot uses a 2/2/2 mixed batch with 240 candidates. It does not evaluate the browser’s 5/4/3 policy or prove label savings.
| Misspecified-model choice | Predicted score | True synthetic utility (higher is better) |
|---|---|---|
| box-optimum | 78.9% | -3.000 |
| truncated | 54.8% | -0.600 |
| reranked | 70.3% | -1.441 |
| random | 21.2% | -1.206 |
The maximum learned score is not the maximum true utility. These methods also use different candidate/generation budgets, so this is a failure illustration, not a fair efficiency benchmark.
Qwen: early exit works, the quality gate fails
Frozen Qwen2.5-0.5B-Instruct, 80 training / 32 calibration / 48 test examples. Rules vary by topic and ALLOW/BLOCK action; grammar is shared. Heads were trained at layers 6, 12, 18 and 24. This is a small synthetic policy task.
| Output path | Test accuracy | Warm CPU p50 | Warm CPU p95 |
|---|---|---|---|
| Constrained A/B/C token (zero-shot) | 25.0% | 357.8 ms | 414.1 ms |
| Trained full-depth head | 97.9% | 309.0 ms | 348.3 ms |
| Trained adaptive head | 91.7% | 79.5 ms | 90.4 ms |
All 48 adaptive requests exited after layer 6; hooks verified the remaining 18 layers did not execute. Accuracy fell by 6.25 percentage points against the trained final head, exceeding the proposed one-point tolerance. The small calibration set did not protect test quality.
The token baseline is zero-shot while the heads receive labels; its low accuracy is not evidence that heads inherently outperform tokens. This pilot did not include an equally trained token baseline; the later matched-output and chat studies provide that control. Layer 12 and 18 probes reached 48/48 on this fixture set, but choosing either after seeing test results requires a new held-out test.
Times include transformer/head execution and adaptive hooks, but exclude tokenization, loading and service overhead. CPU float32, eight threads, one seed. No equal-quality speedup, JSON speedup or rare-error guarantee is established.
Run configuration and metrics · Every test prediction · Train/calibration/test fixtures
Coordinates without text decoding
Reproduced the pretrained GUI-Actor 2B pointer path on our own interface at desktop and mobile sizes. It hit 5 of 6 labeled target boxes. Two absent-target prompts still received points: this head has no no-target class.
| Path | Warm CPU p50 | Samples |
|---|---|---|
| direct | 35555.1 ms | 8 |
| with_vocabulary | 35870.8 ms | 8 |
Both paths use the same pointer head and produced matching probabilities. “With vocabulary” adds one vocabulary projection; it is not JSON decoding or a trained coordinate-token baseline. This isolates a small piece of output overhead. All vision and transformer layers still run.
Eight prompts from two screenshots are a smoke test, not a general grounding benchmark. This uses published Microsoft weights, a shortened prompt, 936 desktop / 392 mobile visual patches, CPU float32 and eight threads. See the resolution-cap correction. Patch confidence is uncalibrated. No live clicks were executed.
Upstream GUI-Actor · Run artifact · Predictions and boxes · Annotate a target
What happens next
Broaden the data before broadening the claim. The next priority is a reliable decision exit gate and a coordinate head that can reject missing targets, followed by fair token baselines and full input-to-result timings.
Public-data follow-ups
BANKING77: every layer, seed, stopping rule and timed path · Conservative gate on a fresh reserve · ScreenSpot: every predicted point and target box.
How the classifiers work · Correction to the old image-resolution cap · Run Qwen in your browser.
Adversarial follow-ups
Separate implementation replays matched the original outputs and actual layer stopping. A second dataset and an explicit unknown-request output still failed their calibration checks. The changing-rule test also failed.
Replay and portable weights · Second dataset · Unknown-request classifier · Changing rules · Matched one-token control.
Practical workloads
600-message protocol · Chat results · Claim audit.
On ten new synthetic mazes, both learned policies reached zero goals; the shortest-path reference reached all ten. The failed moves are retained in the recorded maze replay. Full maze method and evidence.
Small follow-up experiments
Eight approaches on 100 previously inspected messages, including a gate that chooses different stopping layers per message. These exploratory results do not replace the 600-message study above. Separate adversarial review.