Mining agent traces: LangChain's loop for improving agents with data

Key Takeaways

• The loop is four steps, and shipping is step one. Ship the agent so it operates in a real environment; collect every trace it produces (tool calls, output messages, API and CLI calls); mine that trace data; then run experiments to check whether a new prompt, tool, orchestration or loop actually improved things. Without step one there is no data to mine.
• Point an agent at your traces — that is the cheapest thing you can do. LangChain centralizes traces into a tracing project, either per agent or across all of them, then sends agents to read the traces other agents produced. The questions are targeted: find interactions where users got upset or were clearly happy; does the agent get noticeably worse after the first compaction, or the second; what happens if I run this task on a different model instead.
• Two constraints bite at scale. Reading traces costs roughly input-token price × number of traces × average trace size, which gets expensive fast. And a long coding-agent session won't fit in another agent's context at all — so the trace has to be treated as an external object you query into, not context you feed in wholesale.
• Harness engineering first, fine-tuning second, harness engineering again. Harness work returns feedback in about two minutes, so it is almost always the right first move. Only once you hit the ceiling — where more prompt tweaking stops buying anything — does fine-tuning on your domain-specific task make sense, and then you go back to harness work on top of the tuned model. Many teams never need the middle step.
• Densify the feedback signal. A benchmark that returns only pass or fail gives an agent almost nothing to act on: it did a pile of work and learned one bit. Traces are the substrate that carries dense feedback, and agents are good at reading them and deciding what to try next.
• Distillation has a concrete recipe. Take the good traces and good examples from a stronger model's runs, prepare them as a dataset, and fine-tune a small model to mimic that behavior. In work with Harvey on a legal benchmark, an open model matched a frontier model's trace-judging ability at roughly one to two orders of magnitude lower cost, reached through harness engineering informed by reading the traces.
• At high inference volume, the economics flip from tokens to hardware. Running your own cluster gives unlimited inference for a fixed cost you can spin down — worth computing against per-million-token pricing before assuming tokens are cheaper.
• Memory can't be append-only. Agents working with you across months and years can't just append everything to one large file and search it. The talk points at sleep-time compute — reading back the whole trace history and updating agent state — as where this goes next.