Our own inference runtime
A custom inference engine for Qwen3.8 hybrid models on Intel Arc Pro B70 cards, built for speed and provably identical output.
Research engine · Intel Arc Pro B70 · measured, with receipts
Most local models run on a general-purpose engine. We built our own for one model family: a small inference engine for Qwen3.8 hybrid models on Intel Arc Pro B70 cards. It exists to prove two things at once — speed, and output that is provably identical to a reference run, byte for byte. A run that cannot prove itself does not get published.
- Fast. 58.6 tokens a second on two cards, 31.8 on one, on the model we run most; on 2026-10-22 and 23 the record runs reached 63.6 on two cards and 36.6 on one.
- Exact. Every published output is checked against an oracle or an independent repeat run, with receipts kept.
- Resumable. A session saved mid-answer was restored in a fresh process and finished the answer byte for byte; later the same day a saved session took a new question after restore and answered exactly as a fresh run would.
- Open about gaps. Caps and pending checks are stated on the page, not hidden in footnotes.
What it is
Qwen3.8 Flash-Next is a hybrid model: 48 layers, of which 36 use gated delta net (a linear-attention design) and 12 use full attention, plus a 512-expert mixture layer and an early composite layer. General engines run it through generic kernels. This engine is designed around that exact structure: a per-layer plan, a resident cache for the experts, and snapshot banks for the state.
It is a research engine, not a shipped product. There is nothing to download as a finished program; the reproducible artifacts are the recipe pages and lab records linked below.
Measured, with the caps stated
Decode is how fast the model writes; prompt reading is how fast it ingests. All figures come from sealed, receipt-backed runs in the lab repository. Where a measurement has a limit, the limit is printed next to it.
| Setup | Decode | Prompt reading | Exactness |
|---|---|---|---|
| Flash-Next UD-IQ3_XXS, two B70 cards | 58.557–58.632 tok/s (fresh suites); requalified 58.724 / 58.676 on 2026-10-20 | 75.884–75.973 tok/s | Oracle checks the first 128 tokens; full 6,495-token outputs repeat exactly across loads |
| Flash-Next UD-IQ3_XXS, one B70 card | 31.510–31.782 tok/s; requalified 31.766 / 31.774 | 32.595–32.752 tok/s | Same discipline |
| One B70 card, binary rebuilt from source (2026-10-22) | 31.83 / 31.81 tok/s, two fresh loads | not re-measured on this row | All 12 outputs identical across loads and identical to the sealed line: the rebuild reproduces the published one-card result |
| One B70 card, save at 256 tokens then restore (2026-10-22) | restore 0.46 s + 6.78 s to finish 128 tokens, vs 36.75 s to re-read the prompt from cold | — | Merged 384-token output equals the uninterrupted run; two independent restores identical; first 128 tokens match the oracle |
| One B70 card, restore then add a new question (2026-10-22, engine v2) | restore 0.41–0.46 s; the new 127 input tokens took 5.59 s after restore, vs 16.0 s to read all 469 from cold | — | The 128-token answer equals a fresh one-shot run; saving again and resuming for 64 more tokens matches too; a repeat gives identical tokens and a byte-identical saved state |
| Two B70 cards, save, restore, add a new question (2026-10-22) | save 128.8 MB in 1.85 s; restore 2.3 s + 0.09 s to the first new token, vs 59.3 s to re-read the prompt from cold; a new 126-token question took 1.77 s after restore, vs 6.16 s from cold | — | The merged output and the draft guesses equal the uninterrupted run; after adding a question, saving again and resuming in a third process, the saved state is byte-identical on both cards (216 state blobs); 34 of 34 checks |
| Record runs (approved on LocalMaxxing) | 63.611824 tok/s (two cards, 2026-10-23; was 62.345282, 60.659520 and 58.798109 before) · 36.602512 (one card, 2026-10-23; was 35.926720 and 34.573038 before) | — | Capped scope: 128-token suites, stated on the package pages |
| Qwen3.8 27B, GGUF-comparable lane | 37.59 / 37.77 tok/s at 4K context · 37.71 / 37.70 at 32K (llama.cpp reference on the same suite: 28.49) | — | 12 of 12 requests bit-exact at both capacities; 128-token suite |
Pending, and labeled as such: the strict quality promotion for the full512 profiles is still pending, and oracle coverage beyond 128 tokens is an explicit, declared gap. Other honest limits today: greedy decoding only (no sampler), a sequential controller (no multi-user concurrency), and the Flash-Next full512 profiles are measured at 4,096 tokens — other capacities are unmeasured, not claimed. The one-card binary rebuilt from source was qualified on 2026-10-22 (row above); the two-card rebuild has not been run on the cards yet, so the two-card numbers still belong to the sealed historical binary.
Session swap: park the computation, not just the text
Ordinarily, returning to an old conversation means the model reads the whole prompt again. Because we own the engine, it can do something frontends cannot: save a session’s computed state to disk, restore it later, and continue without re-reading the prompt. The ideas behind this are described on our cache parking and cache offload pages.
The first restart-and-continue test passed on 2026-10-22. On one card, a request ran to token 256 and was saved to disk (1.10 s, 68.8 MB). Two separate fresh processes each restored it and wrote 128 more tokens. Both merged outputs equal an uninterrupted run of the same prompt, byte for byte, and the two restores are identical to each other. Restoring took 0.46 s; re-reading the same prompt from cold took 36.75 s.
Later on 2026-10-22 the next step passed too. A second engine revision can take a saved conversation, add the next question, and answer; on one card the answer equaled a fresh run of the whole conversation, byte for byte. It can also save again after a restore, and resuming from that second save matched as well. Refusals happen before anything touches the card: a tampered side file, a mismatched prefix or tokenizer, a wrong delete target.
Saved sessions are now labeled “qualified, one card”. A third build the same day writes that label, with a record of which test qualified which operation, and still reads the earlier experimental files.
Two-card sessions passed the same tests on 2026-10-22. On the two-card record engine, a session was saved with both cards' state, including the draft model's cache, and a fresh pair of processes restored it and continued; adding a question, saving again and resuming all matched a one-shot run, and the re-saved state was byte-identical on both cards. Two-card saves are labeled experimental until a further build writes the qualified label. Not yet: ctxkeeper does not call these operations. The written design contract — save, inspect, restore and delete, with compatibility checks and prefix-plus-delta rules — is what the integration is built against.
How it fits with ctxkeeper
ctxkeeper keeps a long conversation durable from the outside: notes, a searchable archive and a small working view, working with any model server. The own runtime is the other half: the checkpoint engine inside the server that can save and restore the actual computation. ctxkeeper decides what a session needs to remember; the engine provides the how for instant resume. The session-swap contract above is what the ctxkeeper integration is being built against.
Evidence and source
- The lab repositoryThe engine, the sealed run packets, the scoreboard and the receipts behind every number on this page.
- docs/own-runtime.mdThe source of truth for this page: measured state, open items and honest non-goals.
- Two-card recipe on neural.downloadReproducible package for the two-B70 setup, with its caps and missing checks listed.
- One-card recipe on neural.downloadThe same, for a single B70 card.
- Session-swap design contractThe checkpoint contract (v1.2): same-generation resume, new-turn append and restore-then-save all passed on one card on 2026-10-22.
- 2026-10-22 device window recordThe row-by-row results behind the rebuild and save/restore entries above, with evidence paths.
These are research results from one lab on one model family, not a claim of being the only fast option: vLLM and llama.cpp remain the general-purpose engines we compare against and use elsewhere.