Remembering what was dropped
An archive of dropped text with a search tool answered questions about the past as well as summarising, in a quarter of the time.
Saving the original text matched summaries on accuracy and took less time. On this retention test, both answered every question. A current-state table alone lost details from the past.
The test
The 119K-token narrative reading task, with twelve extra questions asked only at the end: values that were later overwritten, and small details in the text. 36 questions in all. Every approach had a 32K working budget except keep-everything, which used the full window.
Same 119K-token narrative history, seed 0, 32K budget; 36 final questions, including 12 about earlier text. Total elapsed time.
All five approaches, context use and tokens written
| Approach | Files | Right | Time | Peak context | Tokens written |
|---|---|---|---|---|---|
| Table plus archive and recall | archive only | 36/36 | 6.1 min | 23K | 14K tokens |
| Table only | no | 25/36 | 5.7 min | 20K | 15K tokens |
| Summarise at 75% | no | 36/36 | 26 min | 23K | 121K tokens |
| Keep everything in the window | no | 34/36 | 14 min | 144K | 33K tokens |
| No management, files allowed | yes | 3/36 | 6.6 min | 27K | 26K tokens |
What it showed
- The archive beat summarising on time and writing, at equal correctness. Both answered 36 of 36. The archive agent took 6.1 min and wrote 14K tokens; the summariser took 26 min and wrote 121K tokens.
- A running table alone cannot answer about the past. It keeps only current values, so questions about overwritten values and details went blank or wrong: 25 of 36.
- The plain files agent collapsed because it has no guards. It spent its 32K budget thinking before it had finished reading and was stopped: 3 of 36. The same agent got every answer on the version without the extra questions.
How the archive works
- When the agent drops a batch from its working context, the batch goes word for word to an archive the model can read but not change.
- A
recallsearch tool finds passages in that archive. - Recalled text counts against the 32K budget like any other tool output.
- The grader voids the run if any other file holds stream data, so the archive is the only memory outside the context.
Nothing is deleted, only moved out of view. In this run the agent searched the archive 12 times. How summaries work · Why files helped on the ledger.
On an outside benchmark
The same ranking held on a public memory benchmark, with the archive again the cheaper of the two methods that manage context. LongMemEval_S (MIT licence), 56 of the 500 questions: 8 per question type, each with about 110K tokens of chat history. The model must answer one question about a long chat history: a fact the user mentioned once, something the assistant said, a preference, a date to work out, a fact that was later updated, or a question the history does not answer, where the right reply is to say so. Every approach ran with the same 27B model on two Intel Arc Pro B70, 65,536-token window.
| Approach | Working budget | Right | Median time per question | Median tokens written |
|---|---|---|---|---|
| Whole history in one call | No limit (fits the window) | 46 of 56 | 1.2 min | 432 |
| Archive and recall, free-text notes | 32K | 42 of 56 | 5.7 min | 10,506 |
| Summarise at 75% | 32K | 19 of 23 (stopped) | 16.0 min | 58,107 |
The summarising approach was stopped early. Stopped after 23 questions: it was taking 16 minutes each and the comparison was already clear. On the 23 questions it did finish (the single-session user, assistant and preference questions), all three approaches can be compared directly:
| Approach | Right of 23 | Median time | Median tokens written |
|---|---|---|---|
| Whole history in one call | 21 | See above | See above |
| Archive and recall | 20 | 5.0 min | 9,004 |
| Summarise at 75% | 19 | 16.0 min | 58,107 |
3 questions only the archive got right, 2 only summarising got right, and 1 neither. That is a tie on accuracy at this sample size. The archive spent a third of the time and wrote a sixth of the tokens. A small pilot of seven questions had put summarising ahead; it did not hold up, which is what the larger run was for.
Reading the whole history in one call is still the best answer whenever it fits the window. Where the archive falls short of it: questions the history cannot answer, 1 of 8 against 4 of 8 for whole-history reading: it answers when the history does not say; and preferences, 4 of 8 against 5 of 8. Teaching the note-taker to record what was not said is the next step.
The judge: Qwen3.8-27B FP8 on the lab host, using the benchmark's official judge prompts. The benchmark's published scores use GPT-4o as judge, so these numbers are not directly comparable with published ones. The run was stopped once by the host memory guard at 52 of 56 archive trials and resumed; finished trials were kept, no GPU fault. Lab note · Per-question verdicts and scripts (2026-10-08).
Over a longer history, with questions at known distances back, the archive method was again the strongest once it had a 32K window. Past the window: what the agent still remembers.
Limits
The synthetic retention test above is one seed and one task family. The outside benchmark adds 56 questions from a second source with a different judge; the published scores for that benchmark use a different judge model, so compare the approaches with each other here, not with published leaderboards. Summarising also got every answer on the synthetic test: with thinking on, its summaries kept the old values. Its cost was time and writing, not accuracy.
Lab note: Retention · Per-run summaries · How to reproduce · All results