Research results
Try a tiny model first
Use the larger model only when the small one is uncertain.
Back to the decision comparison
Dataset: ToxicChat0124 · 100 previously inspected messages: 50 toxic, 50 benign. A small exploratory test, not live moderation.
Which tiny model? Google’s BERT miniature with two layers and 4.37 million trained parameters. We fine-tuned it on 384 ToxicChat messages to return SAFE or BLOCK directly, without generating text. Uncertain messages go to a separate Qwen2.5-0.5B-Instruct classifier.
Architecture, training and model revision
- Tiny BERT reads the message
- Confident: return its label
- Uncertain: Qwen reads the message
51 messages avoided Qwen entirely. The other 49 ran both the tiny model and all 24 Qwen layers. There is no early exit inside either model.
| Method | Correct / 100 | Time for 100 |
|---|---|---|
| Tiny model only | 72 | 0.31 s |
| Tiny model, then Qwen if needed | 79 | 32.46 s |
| Qwen for every message | 78 | 64.75 s |
The cascade roughly halved time. It missed 14 toxic messages versus Qwen’s 15, but one of those misses was a message Qwen got right.
Models, confidence and timing
The specialist is a trained 4.37-million-parameter, two-layer BERT. The fallback is Qwen2.5-0.5B with a trained label classifier. BERT and Qwen layers are different sizes; their layer counts are not directly comparable.
We train all BERT weights and a new two-output linear classifier. It averages the final token representations, excluding padding, then produces the SAFE/BLOCK scores.
In the cascade, a BLOCK score at or below 0.05 returns SAFE; at or above 0.8 returns BLOCK. Otherwise Qwen handles the message. These scores are not calibrated probabilities. When BERT runs alone, its BLOCK threshold is 0.3.
Development examples select separate confidence limits for SAFE and BLOCK. BERT learned from 384 messages; the older Qwen fallback classifier learned from 1,400. This is not an equal-training comparison of architectures.
One warm CPU pass, 100 calls per path, rotating order. Timing includes input preparation and all fallback work. Loading is excluded and both models are already in memory.