When I set up the pipeline, I was worried that Bonsai would write poor code. An independent review said it plans well and codes poorly in agentic tasks, so the whole scheme was built to compensate: fixed signatures, frozen tests, and a task small enough to fit in one or two files.

In hindsight I can see that I made a big mistake here. I ran Bonsai not as an agent but purely as a text generator: one request per attempt, no tools, the answer is the full content of the files it changes. Now the mistake looks obvious, but in the first iteration I did not give this difference much weight. The numbers below are about that setup.

The numbers

By now Bonsai has gone through 28 cards in 34 runs. All 28 were accepted in the end, 12 of them on the first attempt of the first run. Six cards went back to Claude at least once.

More interesting is where the failed attempts failed. Every attempt goes through the same checks in the same order: the files it touched, compilation and clippy, the project’s rules, the tests, and Jev. Over all runs:

  • 27 failed attempts stopped at compilation and clippy;
  • 4 at formatting;
  • 3 at Jev;
  • none at the tests.

When Bonsai’s code compiled, the logic was almost always right.

What went wrong

About two thirds of the compilation failures were code that does not compile at all. The most typical one is a method that does not exist, like this check of an id’s first character:

if input.first() != Some('@') {

Slices have first(), strings do not. The same happened with is_ascii_digits and with a normalized method on one of our own types. The rest were the places where Rust is hard: moving a value out from behind a reference, a temporary that does not live long enough, a function that returns a value where Ok(value) is expected. The remaining third were clippy’s style remarks, such as two nested ifs that can be one.

Claude and Jev classify every failure, and 30 of 34 were classified as trivial: a typo, a wrong method name, a missing import.

The most expensive failure was of a different kind, and it was in the previous post. A card asked for date calculations and named the algorithm without writing it down. Bonsai spent its whole budget twice trying to derive the formulas and did not produce any code. Where it had to know something rather than write something, it had no way to find out.

My conclusion after all these runs was this: Bonsai has no problem with how to write code, but there is a lot it does not know. And in the setup I chose it had no way to check: it could not look up a signature in a dependency or run the compiler before answering. It learned about a missing method only from the next attempt’s error list.

What changed

Bonsai’s prompt did not change: it is the same as on the first run. What changed were the cards. After the first step Claude got two rules for writing them: a known algorithm is written out, not named, and every signature is checked against the project’s rules before the card is cut. In the runs since 19 October the first attempt compiles in 6 of 9 runs, against 7 of 25 before. Maybe the better written cards are the reason, but the cards themselves are different and the numbers are small, so I cannot say that for sure.

The orchestrator also runs clippy --fix before the clippy check now, so clippy’s style remarks that it can fix by itself would no longer cost an attempt. Since it was added, it has not had anything to fix.

All of this cost about four hours of Bonsai’s time for 28 cards, most of it spent on reasoning tokens.

Running Bonsai as an agent, with tools and the compiler inside the loop, is still ahead. The next post is about Jev and how much its answers can be trusted.