Testing agents with Harbor: a task is instructions plus a Docker environment, a verifier writes the reward, an oracle proves the setup, and trials give pass@k

The model, as the speaker describes it

• A task is a Markdown instruction file (what you would say to the agent), an environment (typically a Dockerfile, with the data the task needs, such as a SQLite file or a git repository at a commit), and a tests directory of verifiers: scripts that run after the agent finishes and check the intended side effects.
• Rewards. Verifiers produce a reward, often 0 or 1, written to a reward file. Because verifiers look only at what the agent leaves behind, you can hold the task and verifier constant while changing the agent, its prompts, skills or tools. He compares them to implementation-independent integration tests.
• Anti-cheating. Under the default local Docker setup, the verifiers and solution are mounted into the container only after the agent finishes, so the agent cannot read them.
• Datasets, trials, jobs. Many tasks form a dataset. Each task runs a configurable number of independent trials, each in a fresh container, to account for stochastic agents. The whole run is a job, and optional ATIF (agent trajectory interchange format) JSON records the agent's steps for debugging.
• Oracles. You can drop an oracle, a script that solves the task, in place of the agent. It lets you debug the data, environment and verifiers before spending tokens on a model.
• Scale and metrics. Trials are independent, so they parallelise on sandbox providers; his team uses Daytona with about 200 concurrent workers. Aggregate metrics such as pass@k (did the agent pass within k attempts) can be computed per task or across the dataset.

Checked against Harbor's README

From the Harbor repository, read 2026-10-27.

• Install: uv tool install harbor or pip install harbor.
• Run: harbor run -d "<dataset@version>" -m "<model>" -a "<agent>". The README's example is harbor run --dataset [email protected] --agent claude-code --model anthropic/claude-opus-5-5 --n-concurrent 4, which runs locally in Docker. harbor run --help lists the options and harbor datasets list shows third-party benchmarks such as SWE-Bench and Aider Polyglot.
• Cloud sandboxes: pass --env, for example --env daytona (its example uses --n-concurrent 100). The README lists Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake and others, and does not mention E2B, which the speaker names.
• Not confirmed: the README does not cover the task file layout, the reward file, oracles, trials or ATIF, and Harbor's docs landing page did not either, so those are the speaker's description.

How Arize uses it

• The Phoenix server runs on a single node, so the whole thing goes in a Docker container with a SQLite file of realistic data. They test three surfaces: a coding agent using the Phoenix CLI, a coding agent using its MCP server, and Phoenix's own server-side agent.
• A realistic task: "find hallucinated LLM calls from the last week of my traces and add them to a data set." The verifier calls a REST endpoint or queries the SQLite file to check the dataset row was written.
• Tests that don't depend on implementation let them consolidate skills across the CLI, MCP and server-side routes without rewriting tests when a tool changes.
• Phoenix maps Harbor datasets, trials and rewards onto its own datasets, experiments and evaluations, and converts ATIF into OpenInference traces. He says Harbor also has its own local UI and a hosted service, so Phoenix is optional.

Gotchas & Caveats

• He is a maintainer of Phoenix and OpenInference, so the Phoenix integration is a product pitch.
• This works best when your whole system fits in a container. For a distributed system with S3 and other services he says it is probably harder and he has not tried it; mocking those services is the pain Harbor is meant to avoid.
• Verifier quality is the limit: a weak verifier gives a confident wrong score.

Key Takeaways

• Write verifiers that check side effects, not the agent's steps, so they survive changes to the agent.
• Run an oracle first to prove the environment and verifiers work before paying for model runs.
• Run several trials per task and report pass@k, because single runs hide the variance.