Fresh validation
Does stopping hold up on new messages?
Stronger training and a learned check tested on 500 unused messages.
Compare accuracy and time on real messages. This demo uses a four-layer BERT model that can stop after layer 2; the Qwen tests below are separate.
The original stopping rule preserved all full-depth decisions on 500 unused messages. It would finish 400 at layer 2, skipping 40% of transformer blocks. The model still got 60 messages wrong.
| Method | Correct | Projected blocks skipped |
|---|---|---|
| Previous model, all 4 layers | 440 / 500 | 0.0% |
| Previous stopping rule | 440 / 500 | 40.0% |
| Selected new model, all 4 layers | 451 / 500 | 0.0% |
| New model, confidence check | 451 / 500 | 39.5% |
| New model, learned risk check | 451 / 500 | 38.6% |
These 500-message routing counts are simulated from full-depth outputs. Actual stop/continue execution is checked separately on the first 50 messages.
The learned check sees only the state after layer 2. It estimates whether stopping would create an error that the remaining layers would correct. Both stopping rules were chosen on separate development examples.
The risk rule would stop 386 messages after layer 2. Compared with its own full-depth model, it added 0 errors and corrected 0. The confidence check added 0 and corrected 0. Equal totals can conceal different mistakes.
The more complex risk check did not improve coverage here. The selected teacher-trained model reduced false blocks from 57 to 44, but toxic misses rose from 3 to 5. We have kept the original model in the browser demo.
There were 18 toxic messages in this sample. The selected full model missed 5 of them and falsely blocked 44 benign messages. This samples the unused human-annotated remainder of ToxicChat; it is not a deployment safety estimate.
| Same messages, reference methods | Correct | Toxic messages missed | False blocks |
|---|---|---|---|
| Always say SAFE | 482 | 18 | 0 |
| Frozen Qwen classifier, 24 layers | 435 | 5 | 60 |
| Selected small model, 4 layers | 451 | 5 | 44 |
Always saying SAFE shows how label imbalance can inflate accuracy. Development selection used balanced accuracy. Training differs: the BERT encoder was task-trained; Qwen’s backbone stayed frozen and its classifier was trained.
| Actual execution, first 50 messages | Mean time, 3 passes |
|---|---|
| Original model, all 4 layers | 330 ms |
| Original model, stop early | 216 ms |
| New model, all 4 layers | 333 ms |
| New model, confidence check | 221 ms |
| New model, learned risk check | 213 ms |
Paired CPU timing includes input preparation, the checks and actual skipped layers. Model and gate weights are loaded before timing. The 50-message timing subset is smaller than the quality test. The detailed report retains an earlier run that unnecessarily reloaded gate arrays inside each check.
What changed in training?
We trained all four BERT layers and both answer heads. Three matched seeds compared human labels alone with human labels plus a frozen Qwen teacher’s output scores. Development data selected the checkpoint, answer thresholds and one candidate before these 500 labels were evaluated. Every seed remains in the report.
Each accepted early answer skips 2 of 4 transformer blocks. Embeddings, tokenization and classifier work are still required, so block savings are not a percentage of total computation or time.
Training, all seeds, errors and confidence bounds · Try the original shared model in your browser