Testing an agent when no benchmark exists: plant the answer, make the agent find it, and grade with code, not an LLM judge

The method

• Start from the answer. On the earlier retrieval project the team took a page, asked a model to write a question about it, and had the agent search for that page. Code compares the page found with the page used, so no opinion is involved. Evaluations went from one a week (a meeting where experts compared five chatbots on a big screen and often disagreed) to dozens a day.
• Remove complexity. Publication-grade tricks didn't help on their task and often broke something else. With fast tests, focusing on the embedding model, chunking and the BM25 weighting gave steady gains.
• Apply the same inversion to contracts. The 2026 agent has four steps: spot a clause, find supporting documents, write an amendment, and classify whether to keep it. For the spotting step he describes five ways to build a test with a known result.

Five ways to manufacture a test

1. Past decisions. Diff a signed contract against the original and expect the same flags. It failed for them, because the team keeps only the last version of a contract.
2. Rules. Take a rule (for example a liability cap above $1 million), write a clause that breaks it, generate a contract around it, and expect that flag. They produced 275 tests this way, from rules supplied by experts and documentation plus their own.
3. Break a good contract. Remove a clause from a signed contract and expect the agent to say something is missing, mixed with unchanged contracts that should raise nothing. He calls this the most powerful approach; the agent missed most removed clauses, which made it a hard benchmark.
4. Invariance. Change what should not matter, such as the supplier name, the address or the clause order, or rerun the same contract, and expect the same flags.
5. Templates. Experts already use contract templates with blanks. Fill each blank from a list of values, some of which break rules, so every generated contract has known flags. Feedback from experts took about four weeks, while the script returns an evaluation in five minutes.

Gotchas & Caveats

• He avoids LLM judges where he can: the noise from a judge was comparable to the noise in the agent, and finding a judge prompt that worked took nearly as much effort as the original problem.
• Some designs have to be changed so they can be tested, even if that moves away from the first design.
• Generated tests are a starting point, not the go or no-go for shipping.
• Expert feedback still matters, but he uses it to write better tests and prompts, not to patch the agent directly.
• No code is shown, and the results are the speaker's account of one company's projects. He also mentions trajectory checks and text-to-SQL as other places where the known answer can be planted.

Key Takeaways

• If you can plant the answer, you can grade with code and run the test hundreds of times a week.
• Use invariance checks (swap a name, shuffle the order) to catch brittle behaviour for free.
• Turn expert corrections into new test cases rather than one-off fixes.