Research results
Teach useful answers to appear earlier
Changing the readout is one option. Changing what the model learns is another.
Test results · Decision methods
Dataset: ToxicChat0124 · 100 previously inspected messages: 50 toxic, 50 benign. A small exploratory test, not live moderation.
- Examples with known labels
- Update small parts of Qwen
- Read an answer at chosen layers
Four short training experiments
| Training recipe | Correct / 100 | Toxic missed / 50 | Blocks skipped |
|---|---|---|---|
| Train the final answer | 73 | 19 | 0% |
| Train answers at four depths | 77 | 17 | 0% |
| Also learn from a teacher model | 80 | 14 | 0% |
| Remove the last 12 layers, then train | 66 | 22 | 50% |
All recipes use 24 layers except the truncated model, which uses 12. Before training, the final-layer classifier got 77/100 right and missed 16 toxic messages; the layer-12 classifier got 71/100 right and missed 17.
Learning from a teacher: distillation
Alongside the known labels, a second trained model supplies scores to learn from. This raised the final answer to 80/100, but did not make the earliest answers better.
| Layer | Before training | With distillation |
|---|---|---|
| 6 | 69 | 67 |
| 12 | 71 | 61 |
| 18 | 77 | 78 |
| 24 | 77 | 80 |
Correct answers out of the same 100. The main distillation result uses all 24 layers: 0% skipped.
Each recipe, limitations and failed follow-up
Task adaptation trains the final answer. Joint training also rewards correct answers at layers 6, 12 and 18. Distillation adds the teacher’s scores. Truncation physically keeps only the first 12 blocks.
These use small LoRA weight updates, not full-model retraining: 32 batches of four messages after fitting the initial classifiers. A 32-message development subset selects the checkpoint. The teacher uses the same training labels and is not an independent oracle.
A separately recorded attempt with less initial classifier training also failed: final accuracy fell from 78 to 69, while toxic misses rose from 8 to 26. These small budgets do not establish the limits of the methods.
All layers, timings and exact training · Every exploratory outcome · Review