One CLI, One Skill, One Sandbox: How Hugging Face Automated a Whole Job

Step-by-Step Breakdown

1. Write down the manual workflow first, step by step

   Before any automation, the human process was explicit: for each new paper, find its GitHub URL if one exists; read the repo README; check whether anything new is worth publishing on the Hub; if the artifacts are already there, check whether the model card or dataset card and metadata tags are complete and open a pull request if not; if the artifacts aren't there, open a GitHub issue asking the authors to publish them; then follow up. Everything built afterwards is a mechanical translation of this list — the automation didn't invent a process, it copied one that already worked.

2. Version 1 — choose a workflow over an agent, on purpose

   He explicitly frames the choice using Anthropic's Building Effective Agents post: a workflow uses LLM calls inside a predefined path — more predictable, more controllable, less flexible — while an autonomous agent is an LLM in a loop calling tools until it's done. In 2024 the advice in that post was to avoid building agents unless you need one, start with a single LLM API call, and avoid frameworks. He followed it: each step of the manual workflow became an LLM API call, with no agent framework at all.

3. Deploy version 1 as a nightly cron on GitHub Actions

   The deployment is deliberately unglamorous — a Python script on a schedule. He runs it once every night, using GitHub Actions as the cron runner, and points to the "free cron jobs with GitHub Actions" pattern as the best entry point because the free tier is generous and the UI makes managing many scheduled jobs easy. Every night it reads hundreds of arXiv papers and opens GitHub issues or Hub pull requests.

4. Add tracing before you need it

   Observability is LangFuse, used specifically for tracing: what the LLM is doing, what went in, what came out, which prompts ran, what it cost, and latency. This is the piece most home-built pipelines skip and then regret — he treats it as standard equipment.

5. Notice the second bottleneck: the replies

   Automating issue creation just moved the work. Hundreds of new issues a night generate a flood of GitHub notifications, and answering them is its own job — "kind of like going through your mailbox". That follow-up became the target of version 2.

6. Version 2 — switch to a fully autonomous agent, and say why

   For follow-ups he went the other way: an LLM in a loop with tools and skills. His stated reason is that the guidance changed — an Anthropic workshop on the Claude Agent SDK argued models had become good enough that agents may now beat workflows, effectively reversing the earlier "avoid agents" advice. He also cites a Cursor talk where 12,000 lines of sophisticated custom workflow code were replaced by a roughly 200-line skill, and says his experience matches: thousands of lines of custom code collapse into an agent with a CLI and a skill.

7. Build the agent on the Claude Agent SDK with Bash as the tool

   The agent is the Claude Agent SDK (he rates the Python SDK highly). Its main tool is simply Bash — the terminal — because the actual work is running Hugging Face CLI commands. That's paired with the Hugging Face CLI skill, and he says that is genuinely all it needs. The agent comments on GitHub as follow-up and posts results into a Hugging Face Slack channel.

8. Swap the model without rewriting the agent

   He started on Claude models and, as of that week, was running GLM 5.2 via Hugging Face inference providers — a service that wraps providers like Together AI, Fireworks and Cerebras behind a unified, OpenAI-compatible and Anthropic-compatible interface. That compatibility is what makes the swap a config change rather than a rewrite. He cites GLM 5.2 beating Opus 4.8 on a post-training bench while being cheaper as the reason it was worth switching.

9. Deploy with one container per agent loop

   Deployment is Modal, specifically its batch processing feature: spin up a large number of containers in parallel, where each container is one agent loop processing exactly one GitHub issue. He highlights fast startup as the reason it suits background and overnight agent work. This one-unit-of-work-per-container shape is the transferable part — it bounds blast radius and makes concurrency trivial.

10. Invoke it as a skill, and let agents invoke agents

    Although Modal supports cron, he still triggers the follow-up run manually — via a skill he wrote in Cursor for processing issues on Modal. Running it invokes Composer 2.5 as the driving agent, which in turn invokes all the per-issue agents. He notes this is "the loop that people are talking about" — an agent whose job is to fan out to other agents — and it ends by posting the results to Slack.

11. Take the slop problem seriously

    Asked how to stop an agent spamming the internet, he points to Hamel Husain's LLM Evals FAQ as the thing to read, and closes the talk with "don't forget about evaluation" as one of three conclusions. His empirical result: out of thousands of issues, two negative replies.

12. Reuse the same pipeline for a different surface

    The same workflow drives a Daily Papers account on X that posts interesting papers and artifacts every four hours, with Gemini choosing the best visual to attach. It passed 90,000 followers with no ongoing involvement — a good illustration that once the pipeline exists, a new output channel is mostly a new final step.

Gotchas & Caveats

• He does not disclose that the issues are agent-written, reasoning that people would close them faster and that the content is identical to what he wrote manually. That is a judgement call worth making consciously rather than by default — many projects and platforms have explicit policies on automated issue filing, and this is the part of the talk to disagree with before copying.
• Automating one step relocates the work rather than removing it. Automated issue creation produced an unmanageable notification queue; plan for the second-order load before you turn the first stage on.
• The workflow-versus-agent answer changed with model quality, not with taste. He built deterministic in 2024 on then-current advice and autonomous later on newer advice — both were correct at the time, which is a caution against treating either as a permanent rule.
• Evaluation is the guard against slop at volume, and it's the one component he flags as commonly missing. An agent opening thousands of issues nightly with no eval loop is a reputational incident waiting to happen.

Key Takeaways

• The end-state architecture is deliberately tiny: the Claude Agent SDK, Bash as the tool, one domain CLI, one skill wrapping that CLI, and a sandbox. Most of the complexity people add around agents is replaceable by a good CLI.
• Transcribe the human workflow into explicit steps before automating — both versions of this system are direct translations of a written-down manual process.
• GitHub Actions is a legitimate free cron runner for nightly agent jobs, and Modal batch is a good fit when you want one container per unit of work.
• Choosing an OpenAI/Anthropic-compatible provider layer is what turns "switch models" into a config change — he moved from Claude to GLM 5.2 without restructuring the agent.
• Add tracing (LangFuse here) at the start: prompts, inputs, outputs, cost and latency are what let you debug an agent you never watch run.
• A skill that fans out to many agents is a practical orchestration pattern — one driving agent, N single-purpose loops, results collected into a channel a human reads.