Development diagnostic
Which parts tolerate smaller numbers?
Locate the accuracy loss before attempting another compressed model.
Live conversion of BERT. The Qwen conversion investigated below is a separate experiment.
Smaller numbers can make a model cheaper to store and run. Our first INT8 conversion broke its answers. This test checks which parts are sensitive to that conversion.
| What uses INT8? | Correct | New errors versus float32 |
|---|---|---|
| Original float32 | 28 / 32 | 0 |
| All linear projections | 14 / 32 | 17 |
| Attention projections only | 28 / 32 | 2 |
| Feed-forward projections only | 14 / 32 | 17 |
| First 12 layers | 14 / 32 | 17 |
| Last 12 layers | 21 / 32 | 9 |
| All projections, per-channel weights | 15 / 32 | 16 |
Attention-only conversion was less damaging. It introduced 2 errors and corrected 2 others. Matching the total number correct does not mean preserving the answers.
This is a 32-message development diagnostic. All 24 layers still execute. We have not measured a speed benefit or selected this conversion for the demos.
What was inspected?
We compared the final input token’s internal representation after every layer against the same float32 model. Feed-forward conversion caused much larger changes than attention-only conversion on this sample. This narrows down the investigation; it does not prove a single cause or show that all INT8 methods fail.
The classifier, inputs and thresholds stayed fixed. Per-channel conversion was tested too. No message text is copied into these reports.