What survived two years of rebuilding an in-product agent: the plan in the harness, a finish gate, big results as files, and an 8,000-character memory file

How the agent evolved

1. Late 2023: prototype on early GPT-3.5-class models; arithmetic was quickly moved out of the model into harness code.
2. Launch as a "copilot": a router over a set of skills, a decision tree, with rails.
3. August 2025 rewrite: iterative tool calling and a to-do tool so the agent could plan.
4. February 2026, "Alex 2.0": a new default model and a fuller planning flow and interface.
5. June 2026: specialist agents merged into one agent that loads skills and tools on demand, plus long-term memory.

The speaker's point: they did not train any of these models. Everything they rebuilt was harness code, and the loop, the plan, the stopping rule and the memory are their own.

Decisions that survived

• The plan moved into the harness, one piece at a time. With a plain to-do tool the model spent most of its calls ticking items off and lost focus after 20 or more iterations. They added an in-progress marker, then rendered the plan as a message on every model call so the rules sit next to the data, then dropped the in-progress marker (anything unfinished, in ranked order, is the next thing to do). They are now testing with no to-do tool at all: the harness keeps the state and the completion check.
• Finishing is a gate the harness enforces. "A text reply with no tool call means done" broke on models that announce a step in one turn and call the tool in the next. Now there is an explicit finish call that bounces back if plan items are still open, up to three times, a warning one step before the iteration limit, and a forced finish. Users can bring their own model provider, so the design guards against models they have not tested.
• Treat huge tool results like files. Store the result and keep a pointer plus a one-line description in context. For big JSON trace data they truncate inside the JSON and let the agent use jq to fetch what it needs, because agents are good at writing code.
• Checkpoint the trajectory. When the agent's own history passes a window, the harness makes the model write a checkpoint and drops what it replaced. History has a fixed token budget.
• Long-term memory is one small file. A markdown file per user, capped at 8,000 characters, loaded on every full request and updated by a single background job after the session ends. It beat vector retrieval, episodic and semantic stores and temporal knowledge graphs for them. It is small, exact, easy for a human to audit, and cannot mix users' memories because only one job writes it.
• One agent beat ten specialists. They had a specialist per page of the product, then rolled back to one agent that loads skill instructions and tools on demand.
• Default to "fast and good enough". The newest model is not always best for your task; they spend the time on the harness. They are exploring auto-routing per task and small post-trained models, and note some surfaces (such as a filter-with-AI box) need a single model call with no loop.

Testing

• Trace everything: the product's live agent traces feed regression datasets, so production sessions become the test cases.
• A whole session can be stored as a dataset and a specific turn replayed through the agent.
• Harness changes are A/B tested with experiments (in Arize's own platform, the speaker's employer).

Caveats

• A single company's experience with one agent; none of the numbers (the 8,000-character cap, three bounces) is a recommendation to copy blindly. Test your own.
• The talk promotes Arize's tracing and evaluation product; the design lessons stand independently of it.
• No commands or configuration files are shown. The reproducible parts are the patterns above.