The previous posts mentioned that every question Jev answers in the pipeline is also put to Claude, and that Claude decides while this calibration is on. Here is what that comparison showed.
How Jev answers
Jev does not answer yes or no. For each question, for example “does every function belong to the module’s stated responsibility”, it gives the probability that the answer is yes. The pipeline turns that into a decision with two thresholds: 0.85 or more is yes, 0.3 or less is no, and everything in between is a gray zone. While calibration is on, Claude answers the same questions, and the comparison shows where the thresholds should be.
What 204 questions showed
Over all runs so far Jev got 204 such questions about the code of 28 cards: six project rules and the criteria of each card.
Where Jev was confident, it agreed with Claude every time except once. That one case was the rule about errors: failures have to be typed errors, not booleans or strings.
The uncertainty is not spread evenly. On the rule “the code cannot panic” 11 of 23 answers fell into the gray zone, and on the rule about typed errors also 11 of 23. On a rule like “the code implements the general rule from the task, not the values from the tests” there was no gray zone at all. So Jev is careful exactly on the questions where the answer needs a close reading of the code. For these two rules the threshold could be lowered to 0.3 or 0.4, and Jev would agree with Claude almost everywhere.
The second use of Jev is classifying failures: should the card be retried, or should it go back to Claude. Here Jev and Claude agreed in 31 of 34 cases, and all three disagreements were the same: Jev said the model is going in circles and the card should go back, Claude said it was a typo and worth another attempt. Jev is more pessimistic and sends cards back more often than needed.
Over all this time Jev cost about one cent and Claude as the judge about 2.70 dollars.
The catch
The numbers above look very good, but there is a catch in them. Claude said “yes, the code follows the rule” in 201 of 204 cases. By the time code reaches Jev it already compiles, clippy has nothing to say, the mechanical rules hold and the tests pass. Code like that rarely breaks a rule, so there were only three violations in the whole set.
This means that the high agreement mostly shows that both Jev and Claude find the code fine. How well Jev catches a problem these data almost cannot show, because there were almost no problems to catch.
And this is the main thing I take from the calibration. In the current setup everything around Jev, Rust’s compiler, clippy and the frozen tests, makes such a solid base that Jev as a gate turns out to be almost unnecessary. This is one of the things that made me reconsider Jev’s role in this project, but that is for a later post.