The plan, described in the post about the pipeline, had three roles. Claude is the architect: it splits the work into cards and writes the tests. Bonsai, a local model, writes the code. Jev checks it. After two milestones I looked at who actually did what, and it was not what the plan said.

The first fall

I counted the Rust lines added to the client. Claude wrote about 14,800 lines of product code, Bonsai about 3,300, so 18%. In the last step done the old way (reply, edit, react, delete), Claude wrote 94% of the code.

The cards were the craziest part. Before giving a card to Bonsai, Claude wrote the task, the stubs, the frozen tests, and then implemented the card itself in a scratch copy, to prove that the tests can be passed, and threw that copy away. For one card of that step this was about 230 lines of preparation (task 29, stubs 44, tests 139, proof about 20) for a 20-line function. So each function was written three times: by Claude as pseudocode in the task, by Claude in the scratch copy, and by Bonsai on the card. And everything around it was more than ten times bigger than the function itself.

Bonsai wrote so little because it worked blind. One request per attempt, no tools: it could not look up a method or run the compiler, and it learned about its mistakes only from the next attempt (the Bonsai post has the details).

Jev was simply in the wrong place. It stood last, after the compiler, clippy, the project rules and the frozen tests. These checks are so strict that the code that reached Jev was almost always fine already: Jev said “yes” in 201 of 204 cases. On top of that, while we were “calibrating” it, every question Jev got was also put to Claude as a judge, and Claude agreed with Jev on all 204. Using Jev at this point had no effect at all.

And all of this ran inside one Claude Code session. Its context grew, and every next step cost more tokens than the previous one.

So formally the system had three parts, but only Claude worked. And it did not do 100% of the work, it did 300%: for itself, for Bonsai and for Jev. My limits ran out on Tuesday morning and would reset only two days later.

The fix

I decided to learn from these mistakes and rebuild the pipeline:

  1. Move Jev from judging code to managing the process. We even found a few nice-looking examples: opencode-jev-orchestrator, jev-review, taste-lint and oko.
  2. Give Bonsai an agent loop with tools.
  3. Limit Claude to describing the work: a strict description of each step, no repeated “proofs” that the code can be written. Bonsai takes the work on itself.
  4. The same for tests: Claude only describes the cases as tables, and Bonsai writes the tests.
  5. Mutation testing checks the quality of the tests.
  6. No Claude Code. The pipeline is a separate tool that calls Claude through claude -p when it is really needed.

Jev was supposed to drive all of this: switch between Bonsai and Claude, stop Bonsai when it went into a loop, and escalate to Claude in the cases where Bonsai could not cope, which I hoped would be rare. It looked like a relatively simple and nice tool that would solve all my problems.

But something went wrong.

Jev never started driving. Every decision it was meant to make was recorded “in shadow”: the program acted on a fixed rule, and Jev’s answer was only written down next to it. Claude proposed this for the loop detector following the example of the calibration, and then a phrase I had dropped made it the rule for everything. I said we should start from “seeds and statistics”. What I meant was this: we do not know the right thresholds (when to stop a process, when to switch), so we pick reasonable values, collect statistics as we go, and tune the values later. Claude read it differently: there is no data to trust Jev or Bonsai, so all their work has to be duplicated.

And that was only the beginning. Then Claude started overengineering: it imagined possible problems and put a protection against each of them, then a protection against the protection, and so on. Invariants, a journal of actions with recovery after a crash, a “gray zone” for Jev’s answers, a veto on the number of Claude calls. It measured everything, and kept changing the code based on these measurements, adding something here and removing something there. The longer it worked, the deeper it dug. At some point it had written 15,000 lines of pipeline, against 19,400 lines of the client itself, and still had not reached the end. Even the plan to have Bonsai fix the pipeline’s own bugs failed: its first fix card escalated, and from then on Claude wrote the pipeline alone.

At some point I could not stand it anymore and told it to stop and just run the pipeline, to see how it works.

The second fall

It worked monstrously badly.

Claude picked a simple mechanical task, the theme: go through the code, find every hard-coded colour and size, turn them into variables, and move control over them into a separate file. That is all, quite banal.

But the first thing the pipeline did was generate a specification for it. I do not know what it was doing there. It generated and generated, the tokens were eaten, I watched with sadness as the limits got closer, and from the outside nothing happened at all. The first specification was thrown away whole because three of its tables had no input column. The second one could not be committed, because Claude had put its own report (“Wrote crates/ui/tests/theme.rs. Decisions: …”) at the top of a test file instead of code. And the tests themselves, which in the new scheme Claude was not supposed to write, came to 1,571 lines for moving constants around.

In the end, under my pressure, the task finally went to Bonsai, and the pipeline broke at once, because it had a bug. Then another one. Then another one:

  • The first two cards escalated with zero attempts in four seconds: the pipeline tried to restore a file from git that did not exist yet.
  • The escalation went to Claude, whose login had expired. The pipeline called it 19 times in 20 seconds.
  • Then it asked me through an issue to decide what to do about a “veto: 40 Claude calls in this step”.

Bonsai never got a single token.

So the limits and my patience ran out a second time, and in a sense the situation was worse than before: Exerakt had not got a single new line of code. I stopped this whole mess and decided to throw it all away and start from a clean sheet.

Afterword

Claude still cannot think on its own. Taking something as a model and repeating it in another form is what it does well. But inventing something new it cannot do yet (and maybe never will). The client worked out because it had a model: every feature was checked side by side with Element. The orchestrator had none, and it was assembled from pieces: the shadow mode repeated the calibration of the first pipeline, and the scheme was stitched together from the answers of four subagents, each designing its own part.

When Claude tries to create something new, the best it gets is Frankenstein’s monster: a creature put together from pieces of old notes and scraps of documents, and often this creature does not want to come alive, even from a lightning strike.

So the new parts, like the orchestrator, I will write fully under my control. Claude will honestly follow my instructions and will not try to shape things its own way, because there is no model here except the one that forms in my head. And I will try to make sure that we start from the happy path and add features only when they are needed. Here my plain human unwillingness to go deep into details may be the barrier on the way to overengineering.

The next post is the postmortem of this pipeline and the start from a clean sheet.