Building an Automated Video Editing Pipeline with Claude Code

The architecture

Claude Code is the brain, but it does almost none of the heavy lifting. Every expensive operation is delegated to a purpose-built tool, which is what makes the system cheap to run and easy to swap parts out of later.

• Inputs — raw footage, a brand kit (fonts, colours, logos), and style references from creators whose look you want to emulate.
• Whisper (local, free) — word-level transcription with timestamps. This is the key architectural choice: the transcript is the timeline, so every edit decision is a text operation rather than a video operation.
• FFmpeg (free) — performs the actual cuts, trims and joins against an edit decision list.
• Hyperframes (free, installed from GitHub) — captions, graphics and animations, rendered as HTML/React and animated on a page.
• Higgsfield MCP (paid) — generated B-roll, using Seedance 2. This is the only per-video cash cost.
• Output — a local HTML preview page pairing the timestamped transcript with the cut video, then a 1080p MP4 render.

His framing of the human role: creative director, not operator. Set the vision, supply the context and constraints, hand over the tools, then review output — the same shape as managing a person.

Step-by-Step Breakdown

1. Draw the system before you prompt

   He describes the pipeline to Claude in conversation first and has it emit Excalidraw JSON (via a local skill that knows the Excalidraw format), pastes that into Excalidraw, and gets a clean architecture diagram. That diagram then gets screenshotted into the empty repo as the highest-fidelity way to communicate intent.

2. Open an empty folder and describe the build, then ask for plan mode

   Working in a brand-new repo with nothing in it — no git init, no scaffolding. The opening prompt names the inputs (raw footage, B-roll, graphics, brand kit), names Claude Code as the orchestrator, names each extension (FFmpeg, Whisper, Higgsfield MCP, Hyperframes), and then ends with the instruction that does the real work: understand this first, then enter plan mode and ask me questions to clarify your understanding so we know we're building the right thing.

3. Answer the clarifying questions properly

   Plan mode came back with the decisions that actually determine the output: how aggressive the automated cuts should be (he chose cleanup plus tangent removal), what the pipeline ships (long-form only — no vertical shorts, which keeps resolution and edit-style logic out of scope), how Claude should spend on the only paid step (propose before spending, don't run wild), the state of the brand kit, and whether to build a local HTML preview page.

4. Install Hyperframes at project level, not globally

   Cloned from its GitHub repo and symlinked, so upstream updates propagate automatically. Installed into the project rather than the machine. The installer writes skills into both an agents folder and a .claude folder, so the same skill set is readable by Codex and Claude Code — plus a skill lock file.

5. Interrupt the session to register the new MCP server

   He hits Escape mid-plan, states "I've installed Hyperframes MCP, can I restart the session so you can see it," and resumes. Claude picked the new server up without a full restart, which he notes is a change from previous behaviour where connecting an MCP meant killing and restarting the session. Interrupting to inject a fact and resuming is presented as a normal, safe move.

6. Let it build phase one — the "spine"

   Roughly 20 minutes of real time. Claude scoped itself to the cut engine and orchestration once it understood Hyperframes already owned captions and graphics, and deliberately deferred phase two (screen-recording track) and phase three (B-roll) — a spine-first sequencing where the edit decision list is built to model multiple sources from the start. It wrote the cut logic as JavaScript files, wrote tests, and reported 85 passing.

7. Feed it real footage immediately

   A ten-minute talking-head recording from the previous week's video goes into the raw/ folder. This is where the system broke — see the errors section below. Transcription of ten minutes of audio ran as a background job long enough for him to go get a coffee.

8. Review in the preview page and give abstract feedback

   The preview shows the timestamped transcript beside the video, with kept and cut segments visible. He leaves notes at the level of rules, not instances — for example "removed the silence before this but not the throat clearing", and "sloppy cut between this and the next segment; make cuts clean without any cross-speaking between segments, or flag for me to review". The point is that a rule updates the skill and applies to every future run; a per-clip correction patches one cut and teaches the system nothing.

Common Errors & Fixes Covered

Cost model

Worth reproducing because it is one of the few honest per-output cost breakdowns for an agent-driven creative pipeline:

• Claude — the $100/month plan, orchestrating with Opus and dropping to Sonnet for execution passes to conserve tokens.
• Whisper, FFmpeg, Hyperframes — free.
• Higgsfield B-roll — roughly a couple of dollars per generated clip; ten clips puts a single video in the $20–30 range.
• Target — under $50 per finished video in generation costs.

He also notes a difference in how the two ecosystems bill: Claude usage costs on every run, whereas Codex usage is included in the OpenAI plan — which is why his company puts everyone on individual Claude plans, at the cost of not being able to share skills and projects across the team. See our cost calculator to model your own usage.

Gotchas & Caveats

• Opus is overkill for most of this. The heavy lifting is done by Whisper, FFmpeg and Hyperframes — the model is only routing. Orchestrate with the strong model, execute with the cheap one.
• Don't expect a one-shot perfect render. His stated goal for the first attempt was "pretty good", improved within the session. Treating the first output as a draft is the workable expectation.
• Build modularly so tools are swappable. He picks Hyperframes as the best generative-animation option as of early August 2026 and explicitly designs so it can be replaced when something better lands.
• Front-load context instead of correcting later. His claim: the only time plan mode produces a wrong plan is when it wasn't given enough context. Time spent on constraints and vision up front is repaid in the build.
• Feedback should be a rule, not a patch. Phrase corrections so they apply to every future instance of the situation, so the agent updates its skill rather than fixing one clip.
• Record so the cut engine can succeed. If you fluff a line, stop, pause, and say it again cleanly. Mid-sentence self-corrections are the hardest thing for any cut engine — or human editor — to clean up, and they rarely sound right afterwards.
• Long-running steps are normal. Transcription of a ten-minute recording ran in the background for around ten minutes. Plan to have something else going rather than watching the terminal.
• The stack is not Claude-specific. He notes repeatedly that Codex, Cursor, OpenCode or Hermes could sit in the orchestrator slot — nothing in the architecture depends on Claude Code specifically.

Key Takeaways

• Make the transcript the timeline. Word-level Whisper timestamps turn every edit decision into a text operation, which is what makes an LLM a plausible editor at all.
• Use the agent as an orchestrator over deterministic tools, not as the thing doing the work. FFmpeg cuts, Whisper transcribes, the model decides.
• Passing tests proved nothing here. Real footage found two bugs — a transcript-vs-audio misconception and a hard-coded dB threshold — that 85 green tests missed.
• Spine-first sequencing paid off: phase one shipped a working cut engine with an edit decision list that already modelled multiple sources, so screen recording and B-roll can be added without a rewrite.
• Plan mode with an explicit "ask me clarifying questions first" instruction surfaced the decisions that actually shaped the build — cut aggressiveness, output formats, and spend policy on the one paid step.