Research results
Process a queue together
A small throughput improvement with the same model and the same decisions.
Live q4 Qwen with vocabulary scores and a 64-token message limit; it differs from the trained Python benchmark below.
100 real ToxicChat messages, already used in the fresh-message comparison. Three timed passes repeat this workload; they do not add new accuracy examples.
When messages are waiting in a queue, group similar lengths and process several together. The same Qwen classifier still runs all 24 layers for every message.
| Messages per call | Correct / 100 | Time for 100 |
|---|---|---|
| 1 | 86 | 61.20 s |
| 4 | 86 | 55.28 s |
| 8 | 86 | 57.70 s |
Four-message batches used 9.7% less time, with identical decisions. Groups of eight helped less on average; their final pass was slower than the single-message pass.
This improves a ready queue’s throughput. It does not measure how long a live user waits for a batch to fill. The original classifier still missed 9 toxic messages and falsely blocked 5 benign ones.
Timing and execution checks
Means of three rotating passes on one CPU with four threads. Tokenization, sorting, padding, all model calls and classifier outputs are included; loading and warm-up are excluded. All 900 recorded predictions matched the reference, and all 414 model calls executed 24 layers. No layers were skipped.
Every pass, padding cost and numerical checks · Timing summary · All predictions and layer traces