Research results

Ways to make a decision sooner

Compare the measured speed and accuracy, then open a method for its test results.

Dataset: ToxicChat0124 · 100 previously inspected messages: 50 toxic, 50 benign. A small exploratory test, not live moderation.

Speed and accuracy

Method	Correct / 100	Time / message
Full Qwen	77	636 ms
Stop at layer 12	71	Not timed
Learned stopping	79	421 ms
Tiny model only	72	3.1 ms
Tiny model → Qwen	79	325 ms
With distillation	80	674 ms

The learned stop reduced time by about 34% against its full-model reference. The tiny-model cascade reduced it by about 50% against its own Qwen reference, but introduced one new toxic-message miss.

Same 100 messages for accuracy; training differs between methods. Warm CPU time includes input preparation: 50 timed messages for distillation, 100 for the other timed rows. The cascade’s own Qwen reference scores 78/100 at 648 ms.

Which tiny model? Google’s BERT miniature with two layers and 4.37 million trained parameters. We fine-tuned it on 384 ToxicChat messages to return SAFE or BLOCK directly, without generating text. Uncertain messages go to a separate Qwen2.5-0.5B-Instruct classifier.

Architecture, training and model revision

How each approach works

1. Stop at a fixed pointUse the first quarter, half or three quarters of Qwen. Compare the mistakes at each depth.2. Check before continuingA small checker decides whether to return a label or run more layers. Different messages can stop at different points.3. Try a tiny model firstAccept its confident answers. Send the other messages to a larger model.4. Teach the model differentlyTrain intermediate answers, learn from another model, or remove later layers.

Other datasets and stress tests · Original 600-message comparison · Try output formats in your browser