A Pi-Based Harness Beat Claude Code on DeepSeek — Composio's 30-Task Test

The Composio benchmark

Same model (DeepSeek V4 Flash), same 30 agentic tasks, only the harness changes. "oh-my-pi" is a fork of Pi by Can Boluk that adds power tools on the Pi foundation. Figures as reported in the video.

Three separate winners across three metrics. The Pi-lineage harness completed the most tasks, at roughly half Claude Code's cost per success — but took more than twice as long. Claude Code was the fastest by a wide margin and the most expensive by a wide margin.

Why the system prompt is the explanation

The argument for why a minimal harness wins with open models: most coding agents ship very large system prompts — Claude Code's was reportedly around 10,000 tokens of instructions and best practices injected before you type anything. Those prompts are tuned for the lab's own model. Plug DeepSeek, Kimi or GLM into that harness and the model is reading thousands of tokens of guidance written for a different model — some of it helpful, some actively confusing, all of it billed on every request.

Pi takes the opposite bet: a system prompt of a couple hundred tokens that essentially just names the available tools and gets out of the way. The supporting evidence offered is that Anthropic themselves cut Claude Code's system prompt by roughly 80%, with their own testing showing no performance loss — which suggests much of it was dead weight. A near-empty prompt is also naturally model-agnostic: there is nothing in it for a non-Claude model to fight with.

Key Takeaways

• The harness is part of the model, not a neutral pipe. A three-task swing out of 30 — about 10 percentage points — came from changing only the harness. That is the kind of gain people wait months and pay thousands for from a new model generation.
• No harness won on all three metrics. Pick by what you're optimising: throughput of completed tasks, wall-clock speed, or cost per success. For background work where latency doesn't matter, "done correctly and cheap" beats "fast and wrong."
• Big system prompts cost you twice — once in money on every request, and again in confusing a model the prompt wasn't written for.
• Pi's surface is deliberately small: a TypeScript codebase with four core tools (read, write, edit, bash), no onboarding wizard, no login wall, and everything else opt-in.
• Model switching mid-session preserves context — run a cheap model for the grind and escalate to a heavier one for the hard part without starting over.
• Conversation-as-a-tree is the underrated feature. Every message has a parent, so when the agent goes down a wrong path you branch from before the mess instead of living with poisoned context.

Commands & Code Mentioned

/model      # open the model picker — 15+ providers, switch mid-session, context preserved
/tree       # view the conversation as a tree and branch from any earlier message
/fork       # make a separate copy of the session
/clone      # duplicate the current session
/compact    # summarise older context
pi install <npm-package|git-repo>   # add sub-agents, plan mode, to-do trackers

Configuration lives in models.json (wire up any OpenAI- or Anthropic-compatible endpoint, including local Ollama / LM Studio / vLLM) and a .pi folder holding extensions (TypeScript files hooking into the agent lifecycle), skills (folders of on-demand instructions) and prompts (which become slash commands). Everything also runs headless via print mode, a JSON event stream, an RPC mode and an SDK.