Longer context: the measured results

Keeper: a million-token history through a small window

The same 32K working budget, stream and 228 questions. These test remembering facts and applying rules throughout the history; the ledger and narrative totals below are separate tasks.

One model and one machine; each bar is one run. The proxy has not yet been tested under a real coding or research agent. How the proxy caught up · Question types, missed answers and original evidence · Why in-place notes are faster · Retrieval alone on the same history · How keeper works.

Completing a long task

Each approach used Qwen3.8-27B on two B70 GPUs. A synthetic ledger delivered 20 batches totaling about 121K input tokens. Updates set, add to or delete roughly 160 counters; accompanying memos were irrelevant. The final question asked for 24 current values. Each batch could be fetched only once. Large-window and revised self-editing arms used seeds 0 and 1; other 121K arms used seed 0. Model setup and task prompts.

Both file-using agents answered 24/24 in about 1.9 minutes. Their peak active contexts were 8.2K and 8.9K tokens. The equal-score large-window and summary runs took about 26 and 41 minutes: approximately 14× and 22× the elapsed time in this particular trial.

These totals combine generation, prompt processing and tool use. They are not per-token decode speeds. The “large window” arm imposed no context-management budget, but the model still filtered batches through code; it did not retain every raw memo in its conversation.

Revised self-editing scored 24/24 in 15 minutes and 21/24 in 21 minutes. Every batch was retained. On the second seed, a bug in the model’s rewritten update script stopped applying deletes from batch 17, causing three wrong totals.

The second large-window run returned no answers after 38 minutes. Raw batches and repeated copies filled its context before the final questions arrived. How this approach works and why it ran out.

A stream larger than the full window

The 478K-token ledger contained 76 batches, about 1.8 times the model’s full window. Code processed most input without placing its raw text in the model’s context.

The revised agent peaked at 10K active context; the file agents peaked at 9.7K and 7.1K. This demonstrates processing a longer stream, not attending to all 478K tokens at once.

Why files helped · Original reported task results

Bookkeeping and historical questions

Code applies the numeric changes the model submits. Correct answers still depend on interpreting the source and connecting each question to the right time and owner. The streaming tests independently checked saved states as well as final answers.

For these short documents and final questions, start by supplying the complete source. Both documents fit comfortably in one prompt. Saved states and extracted events offer intermediate records to inspect, but exact balances alone did not ensure correct answers about changing ownership.

Reading narrative text

These tasks require reading counter changes from prose. The read-mode agent updates its own table; the quoted-events agent supplies event lines with supporting quotes and the harness updates the table. The quoted agent also has a read-only archive. Question counts vary by task and seed, so each score shows its own denominator.

On the matched 480K narrative stream, read mode scored 10/10 and quoted events scored 10/10. Quoted events took 19.6 min versus 25.6 min for read mode, about 24% less elapsed time in this one-seed comparison. Both agents used a 32K working budget. These measurements used prefix caching in both methods and different agent protocols. They describe one paired task; a cold-request or replicated speed comparison has not been established.

On the matched million-token narrative stream, quoted events scored 24/24 in 47.9 min; read mode scored 23/24 in 62.2 min. This is one completed pair on the same seed and task, with a 32K working budget. It does not establish reliability across other tasks or seeds.

Re-checked on 2026-10-07 with the fixed scorer: 480K stream 10/10 in 19.4 min; million-token stream 24/24 in 47.8 min. A review had found three holes in the quoted-events checker; after the fix, both runs were repeated on the same task and gave the same scores. Repeated with a 65K server window instead of 262K to leave host memory free; the agent never used more than about 23K of context plus 4K of output. The checker still does not verify event meaning or completeness; final answers are scored. What was repeated · Lab note

Trial scores, source hashes and copied grader results.

Unless a different seed is shown, these are seed-0 trials. The dense prose-ledger attempt is void because it broke the no-files rule. Verifying supplied quotes and computing their arithmetic does not establish that the model found every relevant change.

Research record and failed attempts · Run the reading arms

Longer context slows generation

The server accepted a 262,144-token configured window; the probes supplied up to about 250K input tokens with room for an answer. Weights were FP8; the attention cache stayed at 16-bit precision.

How caching changes the wait · Original protocol, corrections and results

Record lookup accuracy

Two prompt styles, six lengths, ten questions per length and six codes per question: 708/720 codes correct. Each question sampled records spread through the context.

All 12 misses returned another record’s real code. Half were the preceding record; half had a similar record number. Errors appeared at different positions, and results did not worsen steadily: both styles were perfect at 230K.

The preset rule required at least 59/60 in both styles at a length and every shorter tested length. Only 60K met that continuous-range rule. It is a result on these similar-code prompts, not a universal reliable-context limit.

Raw data: codes among ordinary words · Raw data: wall of codes

Reusing an unchanged prefix

With drafting on, the matched cache experiment produced identical output token sequences in 99/99 cases: eleven prompts, nine variants each, up to 30K input and 48 output tokens. Both servers used 832-token reading chunks. Two additional cold-versus-cached checks at 120K also matched.

The separate standard gate compared the shipped 4,096-token read configuration with the 832-token exact-cache configuration. Both returned 12/12 reference answers on the first and second passes. Their roughly 1% decode-rate difference does not establish a speed gain.

The cached 200K questions started in about 2 seconds; the original separate cold run took 114 seconds. Fixed small chunks raised cold-read latency: 30K went from about 9.8 to 11.4 seconds (16% longer), and 120K from about 55 to 69.6 seconds (about 27% longer).

How the cache works · Cache experiments · 99-case data · Standard decode gate data

Remembering what was dropped

The 119K narrative task with twelve extra questions about earlier text. Moving dropped batches to a searchable archive matched summarising on correctness, with less time and writing.

Quoted events with an archive

These two trials used different harness revisions. Seed 0 is the rerun with the final questions pinned in context; its earlier attempt lost those questions. The question counts shown below come from each trial’s grader.

What this shows and how the archive works · How to reproduce

Past the window: questions by distance

Histories of 240K to 1,000,000 tokens behind a 16K or 32K live window, with questions planted at known distances back. A keeper that manages the context from outside the agent answered 39/39 and 46/46 on two 240K histories that open with a rule and a method. Over 1,000,000 tokens it answered 226/228 in 83 minutes and, with in-place notes, 228/228 in 50 minutes; on a second 1,000,000-token history the in-place form answered 210/222 in 50 minutes. Run as a proxy with in-place notes, for an agent that manages nothing, it answered 226/228 in 50 minutes (one stream). All are finished, graded runs. With in-place notes the two 240K histories took about 10 minutes each in place of 16, with every answer right. Retrieval alone kept up at 240K (39/39 and 45/46) where the rule names what it covers, never found a rule that names nothing (0/16, against 16/16 for the keeper), and fell behind late in the 1,000,000-token history: 189 of the 203 questions asked along the way, in a stopped run. On a harder history the keeper with retrieval added answered 47/50 and 46/50, against 42/50 with notes only (one seed, exploratory), and 46/49 on a second seed where the run without retrieval did not finish. An earlier 1,000,000-token run by another method was stopped before its final check and is not a verified score.

On the plain histories at 32K over two seeds, archive and search answered 28/30 planted questions and all 35 final ones; window-drop answered 0/5 beyond about 25 batches back, and compaction reported six old values as current on one seed and none on the other. Summarising at a threshold scored 23/29 on one seed, below window-drop’s 25/29. Before the keeper, no method kept a rule stated once at the start without a pinned place for it.

Full tables, findings and limits · Lab note · Raw evidence

Original research

• Research summary (Markdown)Completed findings and pending work in the recorded research snapshot.
• Window, recall and one-step decisions (Markdown)Preregistered probes, corrections and results.
• Exact prefix cache tests (Markdown)99-case comparison and the separate decode-rate gate.
• How the cache works (Markdown)Detailed engine analysis, state sizes and reuse rules.
• Self-editing and ledger comparisons (Markdown)Task rules, failure analysis and revised-agent design.
• CPU cleanup census (Markdown)Token accounting and potential retrieval costs.
• Where the time went (Markdown)Saved-run timing reconstruction and prefix reuse.
• Paper and related research (Markdown)CLM, alternatives and proposed disk-cache experiments.
• Run instructions and scripts

Website measurement record and aggregation notes · Latest research scripts · How to reproduce