Live browser demo
Reuse work inside Qwen
Measure instruction caching and batching on your device.
This runs actual Qwen 0.5B inference. Every message still uses all 24 layers. Cached paths compute the common instructions once and copy that private cache into each request. Batching processes an available queue together.
The first model download is about 800 MB, plus the runtime. For a smaller download, try batching with BERT.
Nothing has been downloaded.
| Path | Correct | Time for this set | Changed answers | Non-padding input tokens |
|---|
What is included?
Each path runs twice in reversed order. Times include tokenization, preparation, cache construction/copying where used, inference and reading the SAFE/BLOCK vocabulary scores. Loading, dataset download and warmup are excluded. Token counts include the one-time instruction calculation but exclude batch padding; they are not a total-compute estimate. No reply tokens are generated, but the full vocabulary projection still runs.
This browser uses q4 weights and bounds each message to 64 tokens. It does not reproduce the trained float32 Python classifiers or their 256-token tests. Inputs and output rules are matched across this comparison. Batching is queued throughput, not live-response latency.
The sample contains previously inspected ToxicChat messages. Changes and mistakes are reported rather than assuming caching or batching is lossless.