The main job of a software factory is fixing the software factory
Ryle ·
When I started letting coding agents write most of the changes in my hobby projects, I thought of the agent setup as overhead. You build the pipeline, tune it for a while, and then it gets out of the way and you build the things you actually wanted. Five months in, that hasn't happened, and I no longer expect it to. The thing the agents work on most is the system that runs the agents. The applications I set out to build still get built, but more and more they look like a byproduct of making the factory able to build them.
I've come around to thinking that's the right way to run it. This post is why.
The numbers
I sorted every non-bot commit on the integration branch of my main repo by the files it touched. "Factory" means the agent CLI, CI and build tooling, deploy manifest generation, the LLM gateway, the agent runtime and orchestration, and the agent instructions. Everything else, meaning the services, data pipelines, UI and apps, counts as product. I left out notes, docs, lockfiles and generated deploy manifests, because agents commit notes alongside almost every change and they'd swamp the count.
| Month | Commits | Mostly factory | Partly factory | No factory files |
|---|---|---|---|---|
| Month 1 (partial) | 1,488 | 52% | 17% | 31% |
| Month 2 | 2,366 | 49% | 19% | 32% |
| Month 3 | 2,290 | 43% | 22% | 35% |
| Month 4 | 1,388 | 38% | 17% | 45% |
| Month 5 (26 days) | 2,608 | 46% | 16% | 37% |
| Total | 10,140 | 46% | 18% | 35% |
"Mostly factory" means more than half of the commit's files are factory files. The line is fuzzy. Some of what I counted as product is the chat UI I drive the agents from, and if you count the agents' notes as factory work the mostly-factory share goes up to 55%. So I'd call it about half and not read much into the exact figure.
What surprised me is the trend, or the lack of one. Setup costs are supposed to decay. Month 4, when I was mostly building features, was the low point at 38%, and the next month went straight back up to 46%.
Why it doesn't go away
Every new kind of application is a new kind of change the factory hasn't seen before. When I added a phone app, the agents could write the app code fine. What they couldn't do was build it, sign it, run it in a simulator, get the test results back, or know whether they were allowed to push a release to testers. All of that was factory work, and it came in a burst of PRs: simulator UI tests as a gate before any release, uploading the test result bundle only for failed runs so it didn't eat storage, and an explicit rule for when an agent may cut a release without asking me. Android went the same way, and so did a desktop app, a WASM package that every UI build needed, and static sites published to object storage.
The same thing happens at a smaller scale with every first: the first service that needs a new kind of secret, the first app that talks to a box on my home network, two PRs from different agents that pick the same migration number. Each one turns up a missing capability or a bug in the pipeline, and the fix goes into the factory, because that's where the next agent will need it.
So I've started thinking of the applications partly as test fixtures. A new app is the most thorough integration test the factory ever gets, and it's much cheaper than trying to imagine the gaps ahead of time.
Fix the class, not the instance
When an agent makes a mistake there are two possible fixes. One is the change itself: fix the bug, merge, move on. The other is the change to the factory so that no agent makes that mistake again. The second is almost always worth more, because it applies to every later change in every project.
I learned this by doing the first one too many times. An agent doing a rename sweep edited comments inside SQL migrations that had already been applied, the checksums stopped matching and the new pods wouldn't start. Over nine days there were three separate PRs restoring the original bytes of applied migrations, each a quick fix of one instance. The fix for the class is a check that refuses any edit to an applied migration, and it would have taken about as long to write as the first restore.
The cases where I did fix the class are the ones that stuck:
- Agents kept adding separate test binaries when the rule was inline tests. A line in the instructions didn't stop it. A tool hook that blocks the action did, and I wrote about the cost side of that in the CI post.
- Agents reported things as working because a command exited 0. That became a definition of done as the verified deployed result, with a separate verification step that reads what's actually running, which I went into in the fleet post.
- Agents wrote tests that couldn't fail. That became a lexical scan in CI, described in its own post.
- A pause I'd set got undone by a session that had the command to undo it. Pauses became flags on the server that agents can't flip.
The instructions are part of the factory too. My agent instructions are about 4,000 words now, and their source has had 70 commits in the two months since I moved it into its own directory. Nearly every rule in them is there because something went wrong: a GPU bill, a pause someone reversed, a PR that merged and never deployed. They get regenerated, merged, published and checked like any other change. They also need pruning, because a rule that no longer matches the code costs tokens on every turn and misleads every agent that reads it.
The agents fix the factory too
What makes this sustainable is that agents write most of the factory fixes as well. A failed post-deploy check opens a ticket, an agent picks it up, and the fix is often a change to the pipeline that failed. The factory is its own biggest customer. The loop goes: build an app, hit a factory bug, have an agent fix the factory, carry on with the app. Each pass leaves the factory able to do something it couldn't before, and the next app benefits without anyone thinking about it.
There's a trap in this. Factory work is satisfying, and it can absorb everything. The repeated review loop I described in the fleet post was factory work that made the factory worse: every extra round felt responsible, and together they cost more than the changes they were reviewing. Automated follow-ups turn into their own incident if nothing bounds them. So factory changes need the same discipline as product changes: a real problem that happened, a verified result, and some way to tell whether it paid off.
That last part is where I'm weakest. I can show that about half the commits go to the factory. I can't cleanly show that each factory fix cut down repeat failures. My guess is that most of them did, judging by how rarely the same incident shows up twice in my notes once the class is fixed, but that's a guess and not a measurement.
What I'd tell someone starting
Expect about half of the agent work to go into the pipeline that runs the agents, permanently, and treat that as the main output and not a tax on it. When an agent makes a mistake, ask what change to the factory would stop the next agent from making it, and file that ticket too. Expect each new kind of app to break the factory in a new way, and pick projects partly for the gaps they'll expose. Keep the instructions under version control and review them like code, including deleting rules that have gone stale. Hold factory changes to the same verified-in-deployment standard as everything else, because every agent trusts the factory, so a factory fix that doesn't work does more damage than a broken app.
The apps are still why I do any of this. They're also the cheapest way I know to find out what the factory can't do yet.