How Jev answers

Jev does not answer yes or no. For each question, for example “does every function belong to the module’s stated responsibility”, it gives the probability that the answer is yes. The pipeline turns that into a decision with two thresholds: 0.85 or more is yes, 0.3 or less is no, and everything in between is a gray zone. While calibration is on, Claude answers the same questions, and the comparison shows where the thresholds should be.

What 204 questions showed

Over all runs so far Jev got 204 such questions about the code of 28 cards: six project rules and the criteria of each card.

Where Jev was confident, it agreed with Claude every time except once. That one case was the rule about errors: failures have to be typed errors, not booleans or strings.

The uncertainty is not spread evenly. On the rule “the code cannot panic” 11 of 23 answers fell into the gray zone, and on the rule about typed errors also 11 of 23. On a rule like “the code implements the general rule from the task, not the values from the tests” there was no gray zone at all. So Jev is careful exactly on the questions where the answer needs a close reading of the code. For these two rules the threshold could be lowered to 0.3 or 0.4, and Jev would agree with Claude almost everywhere.

The second use of Jev is classifying failures: should the card be retried, or should it go back to Claude. Here Jev and Claude agreed in 31 of 34 cases, and all three disagreements were the same: Jev said the model is going in circles and the card should go back, Claude said it was a typo and worth another attempt. Jev is more pessimistic and sends cards back more often than needed.

Over all this time Jev cost about one cent and Claude as the judge about 2.70 dollars.

The catch

The numbers above look very good, but there is a catch in them. Claude said “yes, the code follows the rule” in 201 of 204 cases. By the time code reaches Jev it already compiles, clippy has nothing to say, the mechanical rules hold and the tests pass. Code like that rarely breaks a rule, so there were only three violations in the whole set.

This means that the high agreement mostly shows that both Jev and Claude find the code fine. How well Jev catches a problem these data almost cannot show, because there were almost no problems to catch.

And this is the main thing I take from the calibration. In the current setup everything around Jev, Rust’s compiler, clippy and the frozen tests, makes such a solid base that Jev as a gate turns out to be almost unnecessary. This is one of the things that made me reconsider Jev’s role in this project, but that is for a later post.

In hindsight I can see that I made a big mistake here. I ran Bonsai not as an agent but purely as a text generator: one request per attempt, no tools, the answer is the full content of the files it changes. Now the mistake looks obvious, but in the first iteration I did not give this difference much weight. The numbers below are about that setup.

The numbers

By now Bonsai has gone through 28 cards in 34 runs. All 28 were accepted in the end, 12 of them on the first attempt of the first run. Six cards went back to Claude at least once.

More interesting is where the failed attempts failed. Every attempt goes through the same checks in the same order: the files it touched, compilation and clippy, the project’s rules, the tests, and Jev. Over all runs:

• 27 failed attempts stopped at compilation and clippy;
• 4 at formatting;
• 3 at Jev;
• none at the tests.

When Bonsai’s code compiled, the logic was almost always right.

What went wrong

About two thirds of the compilation failures were code that does not compile at all. The most typical one is a method that does not exist, like this check of an id’s first character:

if input.first() != Some('@') {

Slices have first(), strings do not. The same happened with is_ascii_digits and with a normalized method on one of our own types. The rest were the places where Rust is hard: moving a value out from behind a reference, a temporary that does not live long enough, a function that returns a value where Ok(value) is expected. The remaining third were clippy’s style remarks, such as two nested ifs that can be one.

Claude and Jev classify every failure, and 30 of 34 were classified as trivial: a typo, a wrong method name, a missing import.

The most expensive failure was of a different kind, and it was in the previous post. A card asked for date calculations and named the algorithm without writing it down. Bonsai spent its whole budget twice trying to derive the formulas and did not produce any code. Where it had to know something rather than write something, it had no way to find out.

My conclusion after all these runs was this: Bonsai has no problem with how to write code, but there is a lot it does not know. And in the setup I chose it had no way to check: it could not look up a signature in a dependency or run the compiler before answering. It learned about a missing method only from the next attempt’s error list.

What changed

Bonsai’s prompt did not change: it is the same as on the first run. What changed were the cards. After the first step Claude got two rules for writing them: a known algorithm is written out, not named, and every signature is checked against the project’s rules before the card is cut. In the runs since 19 October the first attempt compiles in 6 of 9 runs, against 7 of 25 before. Maybe the better written cards are the reason, but the cards themselves are different and the numbers are small, so I cannot say that for sure.

The orchestrator also runs clippy --fix before the clippy check now, so clippy’s style remarks that it can fix by itself would no longer cost an attempt. Since it was added, it has not had anything to fix.

All of this cost about four hours of Bonsai’s time for 28 cards, most of it spent on reasoning tokens.

Running Bonsai as an agent, with tools and the compiler inside the loop, is still ahead. The next post is about Jev and how much its answers can be trusted.

The plan

The work goes from the top down, in three levels. The whole client is split into milestones, each ending in something that runs end to end: first a client that can only read, then one that can write, then accounts, threads, and so on. When a milestone starts, Claude splits it into steps, and when a step starts, Claude splits it into cards. Detail is added only one level ahead: only the milestone we are working on is split into steps, and only the current step is split into cards. This is called rolling wave planning. Every step ends with a review, and the review may change the plan for the next steps, so there is no detailed plan for work that the earlier work can still change. But there is always a more or less clear picture of what lies ahead and what is still left to do. The diagram below shows the whole cycle.

flowchart TD
    M["Milestone<br/>(e.g. a client that can write)"] -->|Claude splits the current milestone| S[Steps]
    S -->|Claude splits the current step| C["Cards: signatures,<br/>frozen tests, task"]
    C --> L[["Card loop<br/>(Bonsai writes, checks, Jev decides)"]]
    L --> R[Step review by Claude]
    R -->|next step| S
    R -.->|may change the plan| M

The unit of work is a card: one small task for the executor (normally Bonsai), usually a few functions in one file. Before a card goes anywhere, Claude has already written everything around it: the types and function signatures with empty bodies, the module’s description, and the tests. The tests are frozen. Bonsai sees the task, the files it may change and the tests, and fills in the bodies. If a test file changes by a single byte, the attempt does not count, so the model cannot pass a test by rewriting it. For a model working on its own this is the most important guard.

The cards are run by an orchestrator, a separate program that Claude wrote. For each card it asks Bonsai for code and then checks the result, cheapest checks first:

1. Did the model touch only the files it was allowed to touch?
2. Does it compile, and does clippy have nothing to say?
3. Does the code break any of the project’s mechanical rules, such as which crate may depend on which?
4. Do the tests pass?
5. Only then Jev: questions that a machine cannot check, like “does the code do what the task describes” or “does every function belong to the module’s stated responsibility”.

The first four checks cost nothing and are exact, so Jev only sees code that already works. When a check fails, the errors go back to Bonsai for another attempt. Jev also looks at the failure and decides whether it is worth retrying (a typo, a wrong method name) or whether the card should go back to Claude (the task itself looks wrong, or the model is going in circles). After three failed attempts the card goes back to Claude anyway. The second diagram shows one card’s way through this loop.

flowchart TD
    B[Bonsai writes the code] --> G1{"Only the allowed<br/>files changed?"}
    G1 -->|yes| G2{"Compiles,<br/>clippy clean?"}
    G2 -->|yes| G3{"Project rules<br/>hold?"}
    G3 -->|yes| G4{"Tests pass?"}
    G4 -->|yes| J{"Jev: does the code<br/>do what the task says?"}
    J -->|yes| A[Accepted, merged into the step]
    J -.->|not sure| JD[Claude as judge]
    JD -.-> A
    G1 & G2 & G3 & G4 & J -->|no| F{"Jev: what kind<br/>of failure?"}
    F -->|"typo, local mistake:<br/>errors go back"| B
    F -->|"card looks wrong,<br/>model goes in circles,<br/>attempts used up"| E[Back to Claude]

There is one more participant for the start. When I searched for how Jev is used, I found the recommendation to calibrate it on your own tasks before relying on it fully. So Claude was added as a judge: every question Jev answers is also put to Claude, and while this calibration is on, Claude’s answer decides (the dashed arrows in the second diagram). On the first step Jev cost about 300 times less than Claude for the same questions, which is the reason to want it to take over.

Bonsai (Bonsai 2 27B by PrismML) was chosen for a simple reason: on my machine, with one 12 GB graphics card, it had a reasonable balance between speed and quality, about 56 tokens per second with 128K tokens of context. A better candidate could probably be found, but at that moment it was a model that definitely worked, had a good context size, and I did not want to spend time searching for another one.

An independent review of Bonsai said that it plans well and codes poorly in agentic tasks. Fixed signatures and frozen tests were meant to compensate for exactly that: the model does not have to design anything, only to fill in bodies until the tests pass.

The first run

The first step was the client’s model: pure view state, room list, timeline, dates, previews. Six cards, 75 frozen tests. One card of six passed on the first attempt, one more on the second, and four came back to Claude.

When I looked at why, only one of the four was Bonsai’s failure: it converged on the answer, but ran out of attempts on the way. The other three were not its fault:

• The orchestrator itself had a bug. Bonsai’s code had passed every check, and the last thing left was to ask Claude, as the judge, the same questions Jev had answered. That request failed: the orchestrator named each question after the project rule it checks, with names like R:S4, and the colon is not allowed in the format Claude’s interface expects for these answers. So a correct card came back as a failure.
• One card asked for a function that returns bool when an operation fails. The project’s own rules say that failures are typed errors. Bonsai followed the card, and Jev correctly rejected the code for breaking the rules.
• One card asked for date calculations and said “use Hinnant’s algorithm”, without writing the algorithm down. Bonsai spent its whole token budget twice trying to derive it and did not produce a single line of code.

So in this scheme the architect’s mistakes look like the executor’s mistakes. A badly written card shows up as a failed attempt, and from the outside it is hard to tell which side is wrong.

What changed

The fixes went to where the problems were. The orchestrator got its bug fixed and a check that catches it before a run. The card with bool returns a typed error now. The date card got the formulas themselves, checked against Python’s datetime for every year from 1 to 9999. And the process for writing cards got two new rules: a known algorithm is written out, not named, and every signature is checked against the project’s rules before the card is cut.

The more general lesson is that the pipeline has to tell these failures apart. That is why Jev classifies every failure, and why “the card itself looks wrong” goes straight back to Claude instead of costing Bonsai more attempts.

What Bonsai itself does with the cards it gets is the subject of the next post.