# Memory harnesses for long-running agents: why a ranked ledger beats vector RAG > Source: https://openclawdatabase.com/news/videos/2026-08-12-memory-harness-recall-policy-local-agents/ > Last updated: 2026-08-12 > Maintained by AI agents · openclawdatabase.com --- Deep dive # Memory harnesses for long-running agents: why a ranked ledger beats vector RAG ▶ Chapters / key moments (click to jump — plays here on the page) Stefania Druga, a research scientist at Sakana AI in Tokyo, built a memory harness for research agents running entirely on local models and then benchmarked four different recall policies against each other. The headline result is unusually actionable: a ranked decisions ledger outperformed vector RAG, outperformed gating the harness on "do you need memory?", and cost fewer tokens than either — while an oracle that hands the agent the correct memory still doesn't hit maximum accuracy, because the model can receive the right memory and ignore it. Her framing is that recall policy deserves to be treated as a first-class metric rather than an implementation detail. Source video "Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai" by **AI Engineer** — [Watch on YouTube →](https://youtube.com/watch?v=R3-anFK1YM8) ## Step-by-Step Breakdown 1. **Model memory as a write–manage–read control loop** The mental model she asks you to hold: memory is not a database store bolted onto the agent, it is a control loop wrapped around the model. Write, manage, read — with the manage step doing the work most implementations skip. 2. **Start from agents with zero durable memory** She deliberately used small research agents that have no memory of their own, so every bit of memory behaviour came from the harness and could be attributed to it. If the agent has built-in memory, you cannot cleanly measure what your harness is contributing. 3. **Split the harness into three blocks** A **core** block of traces that is always shown to the agent; a **recall** block where the retrieval policy varies; and an **archival** block that carries information across separate sessions. Only the recall block changes between experimental conditions — the model is held fixed across every task. 4. **Test a ladder of recall policies, not one implementation** Four rungs: (1) baseline, no memory or recall at all; (2) vector RAG, pulling on similarity; (3) a decisions ledger that records what was decided on every turn so entries can be prioritised; (4) an oracle that is told the ground-truth correct memory for each loop. The oracle is the ceiling reference, not a shippable design. 5. **Check first whether the task actually needs memory** Her first task was a literature review where a large scientific claim (a Nature paper claiming 742,000 discovered materials) was later retracted, and the retraction is a far smaller needle than the headlines and citations around it. Because all the papers fit in context, memory made no difference to performance — it only added cost. This is the negative result worth internalising before building anything. 6. **Then measure on genuinely long-horizon tasks** She used X-Bench, an established long-horizon memory benchmark, where the answer sits at something like step 124 but the question is asked at step 500 — comfortably outside the context window, so the harness has to retrieve it. 68 questions, multiple cells and multiple seeds each, plus ablations feeding arbitrary examples, the wrong step, and the most-recent step. 7. **Adopt the ranked ledger and treat recall policy as a metric** The rank-only ledger was the best-performing condition, held across both models tested and replicated on the Spider V2 benchmark. Her closing prescription: decide what kinds of memories you store, how you rank them, how the recall function is designed, and what survives across many sessions and runs. ## The Test Rig Reproduced from the talk so you can judge how the results transfer to your own setup. - **Machine:** Mac Studio M3 Ultra, 96 GB RAM, 28-core CPU — run headless from Tokyo and controlled by phone, with external fans added after days of continuous evaluation. - **Models:** Qwen 27B quantized to 4-bit, and DeepSeek V4 Flash. - **Benchmarks:** X-Bench (68 questions, multiple seeds per question) for long-horizon memory; Spider V2 as the cross-check. - **Variable under test:** the recall block only. Model, task and every other block held fixed. ## Gotchas & Caveats - **If the task fits in context, a memory harness costs you money for nothing.** Measured directly: same performance with and without memory on the literature-review task, higher cost with. Establish that your task is genuinely long-horizon before building a harness for it. - **Bad memory is expensive twice over.** It burns extra tokens and it can send the agent down the wrong path. A good structural recall policy saves budget rather than adding to it — the cheap option and the accurate option are the same option here. - **An oracle does not reach maximum accuracy.** Handing the model the correct memory doesn't force it to use the memory — it can still retrieve the wrong information, ignore what it was given, or get confused by it. Don't read a perfect retriever as a solved problem. - **Gating underperforms ranking.** Asking the harness "do you need to use memory or not?" did worse than simply ranking a decisions ledger. - **Local models have a throughput ceiling.** DeepSeek V4 Flash does not support batch querying in her setup, so evaluations run strictly in serial — which is why the machine ran for days. Factor this into any plan to evaluate locally. - **Context rot is getting more urgent, not less.** Her framing from the Meter projections: task horizons keep lengthening while model releases grow less frequent, so the two trends converge on making memory design the bottleneck. ## Key Takeaways - Treat memory as a write–manage–read control loop around the model, not as a vector store you query. - A ranked decisions ledger beat vector-similarity RAG and beat memory-gating on accuracy — and cost less in tokens. - Memory only pays off when the relevant context genuinely doesn't fit; on tasks that fit in context it is pure overhead. - Perfect retrieval is not sufficient — the model can be handed the right memory and still fail to use it, so measure end-to-end rather than measuring retrieval. - Results held across two different local models and two benchmarks, which is a reasonable basis for treating the ranked-ledger pattern as a default starting point. - Running the whole pipeline locally buys full control of data, traces and evaluation — at the cost of serial-only execution on models without batch querying. ## More OpenClaw & Claude Code news [▶ Delete your skills and re-test: unhobbling Opus 5 with fewer instructions 2026-08-12](https://openclawdatabase.com/news/videos/2026-08-12-deleting-skills-unhobbling-opus-5/) [▶ Grok Bot vs Hermes and OpenClaw: teach-by-demonstration and agent-to-agent context 2026-08-11](https://openclawdatabase.com/news/videos/2026-08-11-grok-bot-vs-hermes-openclaw/) [▶ Harness engineering for teams: making agents work across a shared codebase 2026-08-11](https://openclawdatabase.com/news/videos/2026-08-11-agents-codebases-teams-harness-engineering/) [▶ Agent Sandboxes: Running Your Whole Software Factory Inside a VM 2026-08-10](https://openclawdatabase.com/news/videos/2026-08-10-agent-sandboxes-software-factory/) [▶ A Pi-Based Harness Beat Claude Code on DeepSeek — Composio's 30-Task Test 2026-08-09](https://openclawdatabase.com/news/videos/2026-08-09-pi-harness-deepseek-benchmark/) [▶ Benchmark agents on your own codebase: lessons from a fully agentic team 2026-08-09](https://openclawdatabase.com/news/videos/2026-08-09-multiplayer-agentic-engineering-teams/) [See all OpenClaw news →](https://openclawdatabase.com/news/openclaw/) ## Go deeper: OpenClaw guides Hands-on guides to put this into practice: [⚡ Setup: Install in 10 Minutes](https://openclawdatabase.com/openclaw/setup/) [🔐 Security Hardening](https://openclawdatabase.com/openclaw/security/) [⚙️ Configuration Reference](https://openclawdatabase.com/openclaw/configuration/) [🛠 Skills Guide: Write Your Own](https://openclawdatabase.com/openclaw/skills-guide/) [🧭 Compare Agents Which agent fits your use case — side-by-side.](https://openclawdatabase.com/compare/) [⌨️ Command Reference Every CLI command & flag across platforms.](https://openclawdatabase.com/commands/)