Skip to content

The Token Bill Was Never the Bottleneck. Somebody Has to Say Yes.

AutoCraft is an open harness that triages a GitHub backlog with Haiku, Sonnet and Opus, writes the fixes and merges some of them itself. The interesting part is not the model chain. It is the single file allowed to write, the audit table beside it, and the approvals an agent cannot grant itself.

AI-assisted draft. Reviewed and edited by the Phosphene team before publication.

On this page
A dark wooden desk with one small steel strongbox, a neat line of blank white index cards leading to its single keyhole, and an open ledger with empty pages beside it

ArtCraft started as AI-built replacements for Photoshop and Lightroom, and the project has grown past that: by Brandon Thomas's count at the AI Engineer CODE Summit on 12 November, roughly 4,000 developers work on it, more than 300 applications are in development, and the apps see about 1.7 million downloads a day. A codebase that size does not have a backlog problem in the usual sense. It generates issues faster than any maintainer can read them, and the reading is the part that actually costs.

Thomas used his talk, titled "Let's Open Source the World", to present AutoCraft, an open harness that works that backlog with a chain of Claude models and merges some of the results without a human in the loop for that specific step. The code is on GitHub under MIT or Apache-2.0.

AutoCraft mirrors every issue and pull request into a local SQLite database and runs stages over the mirror: deduplication, classification, priority, a decision stage where an operator answers scope questions, review, fix-writing, verification, a merge queue, and a second review of each pull request after it lands. Different stages get different models. Haiku does the cheap first pass on priority, Sonnet writes patches for clear bug reports, Opus reviews heads and settles disagreements.

That pipeline is the easy part to describe and the boring part to copy. Two design decisions in the repository are worth more attention.

The budget is a maintainer's attention

The README states the premise outright: the pipeline is built around maintainers' judgement, "which is the limited resource, not around token cost."

Once you take that seriously, the rest follows. A duplicate-detection pass that costs a few cents to run is worth running on everything, because it saves a human from reading the same bug report eleven times. A fix that would take a maintainer twenty minutes to write is worth a Sonnet call even if that call is expensive by comparison, because twenty minutes of a maintainer is the thing you cannot buy more of. And a decision the maintainer has to make anyway should arrive as a summary with a recommendation and drafted answers, not as a raw item with a link.

Most agent deployments I see get this backwards. They are built to minimise tokens, because tokens are the line item on the invoice, and then a human ends up reading raw model output for three hours a day. The invoice was never the constraint. The queue was.

AutoCraft also enforces the budget in the other direction. Every write to GitHub needs the repository's approval. Several stages also have daily limits: 40 review posts, 20 fix-up pushes, 10 fix PRs, 10 merges through the queue, 20 comments. Those caps are not universal: the cifix step has no daily limit and instead caps attempts per head and per pull request. There is a global pipeline switch, a per-repository switch, and a newly tracked repository starts with its pipeline disabled. Tokens are cheap enough to leave running. Attention is not, so the system defaults to doing less.

One door, and a test that keeps it the only door

The part I would copy first is the write boundary. Only src/github/write.ts may write to GitHub. Every write lands in an audit table. And a test fails if any other file in the repository writes to GitHub, which turns a convention into a build error.

Around that door sit three more locks. Every GitHub write needs an operator's approval for that specific repository, split by action type: labels, comments, fixing, pushing, merging. Approvals are granted on the dashboard with GitHub sign-in, they are audited, and the CLI can read them but cannot set them. That keeps the approval record outside the pipeline's own tool surface. It is not containment, and the documentation is careful not to claim it is: a process running as the same account could call gh with the stored credentials and never touch the harness at all. Roles are separate (operator can approve, maintainer can read and comment, viewer reads), and contributors' own agents reach the inbox through an MCP server using that person's role.

The remaining boundary is the one people skip, and the documentation is unusually precise about where it sits. Untrusted code runs in a separate build environment as a different OS user under rootless Podman, never in the same place as the GitHub and Claude credentials.

Model calls are the harder case, because a model that judges a patch needs both the code and a credential. AutoCraft's answer is to run most calls with no tools at all, and to register a sandbox mode for the handful that need a checkout. Inside bubblewrap those calls see the claude binary, /usr, the TLS certificates, the checkout mounted read-only at /repo, and the used account's credentials file, read-only. Home is empty, the environment is clean, and CLAUDE.md, AGENTS.md and every .claude/ path are stripped from the checkout so repository text cannot load as project instructions. Claude Code is held to Read, Grep and Glob, confined to the working directory and the checkout, with no MCP servers, skills or slash commands and a fixed turn limit. A backstop then scrubs token-shaped strings out of anything a reply carries and refuses a patch that holds a credential.

So the credential and the code do share the sandbox. What is removed is the model's ability to wander: three read-only tools, one read-only checkout, no way to read the rest of the account's config directory, and no repository-supplied instructions to follow. That is a narrower claim than isolation, and a more useful one to copy, because it names the actual attack surface instead of asserting a wall.

The arrangement answers a general problem. If you point an agent at creative production, the analogue of GitHub write access is the asset library: covers, template catalogs, exported masters, the files your site actually serves. One module that writes there. An audit row per write. A test that fails when a second path appears. And approval that lives somewhere the agent cannot reach.

Two reviews, one of them hostile

The review stage has an unusual constraint: Opus reviews every wanted pull request at its current head, reading the repository, and writes nothing to GitHub until the final review is posted. No drip of inline comments the author has to reconcile later. One verdict, one fix list.

Then, before a pull request the rules would merge goes into a batch, Opus reviews its head commit a second time, "adversarially and independently of the first review". A defect it verifies converts the merge into a fix-up or an explicit decision. The merge queue itself runs in integration batches of up to 50 pull requests, with upstream CI checking the batch rather than each branch in isolation.

A second pass by a fresh context is cheap and it catches a specific class of failure: the reviewer that already decided the change was fine and is now looking for confirmation. The same model can do both jobs if you keep the runs separate.

For interface changes there is a third check that anyone shipping design work should notice. With the screenshots feature on, a UI pull request is rendered at the base commit and at the head commit, a model compares the two images, and a clear regression holds the merge. Not a pixel diff with a threshold nobody tunes. A visual comparison that has to name what broke.

Human input stays authoritative

One line in the stage table does more for operator trust than any amount of prompt engineering: a label a person set is never changed.

Every automation that writes back into a system eventually overwrites a human decision, usually because the merge logic treated "current value" and "our value" as the same thing. Deciding up front that human-set state is sticky removes an entire category of argument. If an agent disagrees with a label, it can open a conversation about it. It cannot silently flip it.

What survives the port

You are unlikely to run a 4,000-developer project's backlog. The structure still maps onto smaller creative operations, because the failure modes are the same shape. An agent fleet writing into shared assets needs one write door and an audit trail, or nobody can answer which run produced the broken cover. It needs approvals stored where the agents cannot grant them. It needs a completion test that is not the agent's own claim of success. And it needs to be designed around the hours of human attention it consumes, not around the cost of the calls.

Thomas's framing for the whole thing was "an autonomous software factory", and the repository's own description is more modest: a model harness for maintaining repositories with a large backlog. The honest version is the second one. The models do the reading and the drafting. The factory floor still needs a person at the door, and the design work is making that person's presence count for something instead of making it constant.

Sources

Share this guide
X LinkedIn