Research results
Reuse the instructions
Avoid computing the same fixed instructions for every message.
Live q4 Qwen with vocabulary scores and a 64-token message limit; it differs from the trained Python benchmark below.
The same 100 real ToxicChat messages as the fresh-message comparison. This checks repeated execution, not a new accuracy sample.
Every message uses the same moderation instructions. Compute that fixed beginning once, then reuse its stored internal values while reading each new message.
| How the instructions are processed | Correct / 100 | Time for 100 |
|---|---|---|
| Read again for every message | 86 | 60.77 s |
| Reuse the fixed beginning | 86 | 37.73 s |
37.9% less time, with identical decisions on these 100 messages. All 24 layers still process each message. The saving comes from avoiding repeated work on the instructions.
Only the fixed instructions are shared. Every message receives a separate copy of that cache. Changing the instructions or model requires a new cache.
What was checked?
Three alternating timing pairs on one CPU, four threads. Times include input preparation, building the shared cache once per 100 messages, copying it, model calls and classifier outputs. Loading and warm-up are excluded.
The fixed beginning contains 53 tokens. Every request must match it exactly. The original cache remained unchanged; every model call executed all 24 layers, and all predictions matched the full-input reference. Reusing keys and values still requires attending to them; token reuse is not an equal percentage reduction in all computation.
The model still missed 9 toxic messages and falsely blocked 5 benign ones. Matching its answers here does not establish reliable moderation or universal numerical equivalence.
All passes, cache checks and numerical comparisons · Timing summary · Every decision and layer trace