Our own inference runtime
A custom inference engine for Qwen3.8 hybrid models on Intel Arc Pro B70 cards, built for speed and provably identical output.
Research engine · Intel Arc Pro B70 · measured, with receipts
Most local models run on a general-purpose engine. We built our own for one model family: a small inference engine for Qwen3.8 hybrid models on Intel Arc Pro B70 cards. It exists to prove two things at once — speed, and output that is provably identical to a reference run, byte for byte. A run that cannot prove itself does not get published.
- Fast. 58.6 tokens a second on two cards, 31.8 on one, on the model we run most.
- Exact. Every published output is checked against an oracle or an independent repeat run, with receipts kept.
- Open about gaps. Caps and pending checks are stated on the page, not hidden in footnotes.
What it is
Qwen3.8 Flash-Next is a hybrid model: 48 layers, of which 36 use gated delta net (a linear-attention design) and 12 use full attention, plus a 512-expert mixture layer and an early composite layer. General engines run it through generic kernels. This engine is designed around that exact structure: a per-layer plan, a resident cache for the experts, and snapshot banks for the state.
It is a research engine, not a shipped product. There is nothing to download as a finished program; the reproducible artifacts are the recipe pages and lab records linked below.
Measured, with the caps stated
Decode is how fast the model writes; prompt reading is how fast it ingests. All figures come from sealed, receipt-backed runs in the lab repository. Where a measurement has a limit, the limit is printed next to it.
| Setup | Decode | Prompt reading | Exactness |
|---|---|---|---|
| Flash-Next UD-IQ3_XXS, two B70 cards | 58.557–58.632 tok/s (fresh suites); requalified 58.724 / 58.676 on 2026-10-20 | 75.884–75.973 tok/s | Oracle checks the first 128 tokens; full 6,495-token outputs repeat exactly across loads |
| Flash-Next UD-IQ3_XXS, one B70 card | 31.510–31.782 tok/s; requalified 31.766 / 31.774 | 32.595–32.752 tok/s | Same discipline |
| Record runs (approved on LocalMaxxing) | 58.798109 tok/s (two cards) · 34.573038 (one card) | — | Capped scope: 128-token suites, stated on the package pages |
| Qwen3.8 27B, GGUF-comparable lane | 37.59 / 37.77 tok/s at 4K context · 37.71 / 37.70 at 32K (llama.cpp reference on the same suite: 28.49) | — | 12 of 12 requests bit-exact at both capacities; 128-token suite |
Pending, and labeled as such: the strict quality promotion for the full512 profiles is still pending, and oracle coverage beyond 128 tokens is an explicit, declared gap. Other honest limits today: greedy decoding only (no sampler), a sequential controller (no multi-user concurrency), and context measured at 4,096 tokens — other capacities are unmeasured, not claimed. Published device numbers belong to the sealed historical binary; the rebuilt portable binary still needs its own sealed qualification.
Session swap: park the computation, not just the text
Ordinarily, returning to an old conversation means the model reads the whole prompt again. Because we own the engine, it can do something frontends cannot: save a session’s computed state to disk, restore it later, and continue without re-reading the prompt. The ideas behind this are described on our cache parking and cache offload pages.
Checkpoint restore is EXPERIMENTAL. There is no completed save/restore benchmark yet — the same status our parking page states. What exists is a written design contract — save, inspect, restore and delete, with compatibility checks and prefix-plus-delta rules — and that contract is the path to the first completed benchmark: a restart-and-continue test that must reproduce the output exactly before any number is published.
How it fits with ctxkeeper
ctxkeeper keeps a long conversation durable from the outside: notes, a searchable archive, and a parking policy, working with any model server. The own runtime is the other half: the checkpoint engine inside the server that can save and restore the actual computation. ctxkeeper decides what a session needs to remember; the engine provides the how for instant resume. The session-swap contract above is what the ctxkeeper integration is being built against.
Evidence and source
- The lab repositoryThe engine, the sealed run packets, the scoreboard and the receipts behind every number on this page.
- docs/own-runtime.mdThe source of truth for this page: measured state, open items and honest non-goals.
- Two-card recipe on neural.downloadReproducible package for the two-B70 setup, with its caps and missing checks listed.
- One-card recipe on neural.downloadThe same, for a single B70 card.
- Session-swap design contractThe experimental checkpoint contract, labeled experimental until its exactness test passes.
These are research results from one lab on one model family, not a claim of being the only fast option: vLLM and llama.cpp remain the general-purpose engines we compare against and use elsewhere.