Element as the reference

For the look and the features I like Element a lot, so it is no surprise that I chose it as the reference for where the project should go. This also answered a question the pipeline cannot answer by itself. Jev and Claude check whether a card is done, but nobody checks where the whole project is going. For now I want to reproduce the look and feel of Element, so every feature of Exerakt is finished by comparing Exerakt and Element side by side: the same account, the same room, the two windows next to each other.

What was built

The second milestone was a client that reads Matrix the way Element shows it. It took six steps:

1. Turning the HTML of formatted messages into the client’s own representation.
2. Drawing it: quotes, code, lists, headings, tables, links, mentions, and spoilers, with selection and copy working across all of them.
3. Showing replies, edits, reactions, deleted messages, and messages that could not be decrypted, with the same wording as Element.
4. Read receipts and the line that marks where the user stopped reading.
5. Pictures and colour emoji.
6. Local time, a status line for the connection, and a final check of the whole milestone against Element.

Side by side

A card is checked by its tests, but nothing tests how the whole client looks. So every step ended with a comparison, and since the process is automatic, Claude did it as well:

1. A local Conduit server ran in Docker. A small program created rooms on it and filled them with the kinds of messages the step was about: formatted text, replies and reactions, pictures.
2. Claude started both clients, logged in to the same account on this server, on a separate workspace of an invisible (headless) Hyprland output, so nothing appeared on my screen. It captured Exerakt with grim, a screenshot tool for Wayland, and Element through its developer tools.
3. Claude put the two screenshots side by side and compared them. What could be fixed within the step was fixed there; everything else became a task for a later step.

Below are three of these comparisons from the second milestone, Exerakt on the left and Element on the right. They are not final screenshots: they show the clients the way Claude saw them at the moment of each check. Two of the differences in them, local time and colour emoji, were done later within the same milestone; the others are still ahead.

Works:

• bold, italic, inline code, and strikethrough;
• code blocks, numbered and nested lists, headings, quotes, tables, and rules;
• coloured text, spoilers, links, and mentions, all in the same places as in Element.

Not yet:

• syntax highlighting and line numbers in code blocks;
• dimmer quotes, and a mention shown as the user’s name with an avatar;
• local time: at that moment Exerakt showed UTC, local time came later in the same milestone.

Works:

• the replied-to message above the reply;
• the edited mark;
• reactions with their counts under the message;
• messages grouped by sender the same way.

Not yet:

• colour emoji, which came later in the milestone;
• the replied-to sender with an avatar and a coloured bar;
• icons before deleted and undecryptable messages.

Works:

• the read marker line after the same message;
• the same times;
• the badge of the other user who read the room, on the same message.

Not yet: avatars and coloured sender names, here and everywhere else.

Other people’s code

As I said at the start, drawing the messages itself went fairly smoothly. The problems came from two other places: the servers, and one of the libraries.

The servers

Matrix servers that speak the same protocol do not always send the same things. But first, a few technical details about two features where this showed up.

The first one is read receipts. When you read a message, your client tells the server, and the clients of other people in the room show it, usually as small avatars next to the last message each person has read. The second one is the read marker: a line in a room that shows where you stopped reading last time, so that when you come back you see where the new messages begin. The server keeps its position, so the line is in the same place on all your devices.

A Matrix client gets news from the server through one long-lived connection, and both of these should come through it. With Synapse they did. With Conduit they did not come at all, so Exerakt never learned where the line was or who had read what. I found this only because Claude checked the feature against both local servers. Exerakt now asks the server for the line separately every time a room opens, which Conduit does answer. Other people’s read receipts on Conduit are still missing.

The emoji

The library problem was with emoji. They came out in black and white. To draw a character, parley, the text layout library, looks for a font that has it, and it looked through the regular fonts first. Some of them have black-and-white emoji, so they were picked before the colour emoji font. The fix was to find the emoji in the text in Exerakt itself and send them to the colour emoji font directly. As a side effect, the first emoji the program shows now takes 0.7 milliseconds instead of 97 (yes, almost 140 times faster), because the slow part had been the search through the system’s list of fonts.

Additional lessons

It turned out that Bonsai is not the only one who does not know everything. When you reply to a message in Matrix, the client also puts a copy of the original message into the reply, for older clients that cannot show replies. A modern client has to remove that copy before showing the reply. In the first step of the milestone Claude cut a card asking Bonsai to do exactly that, and Bonsai did it. Only then it turned out that matrix-sdk-ui, the part of the Matrix SDK that builds the list of messages, already removes the copy, so the card was withdrawn.

The mistake was Claude’s, the architect’s, like the ones in the first run, and it led to a new rule for writing cards: before cutting a card, read what the dependency already does with the data.

The next post is about the third milestone, writing messages, and about how the plan for the pipeline met reality.

Why Wayland only

With Wayland the reader may ask: why this way? Why not go the standard route, choose a good, time-tested cross-platform library and use it?

My answer has two parts. The first one is philosophical. I believe that with LLMs it is possible (or will be in the near future) to build specialised solutions efficiently. Do you want it for macOS or Windows? You create a project like mine, feed it the code of Exerakt, and after some time you have your own version of the application, built and tuned for your environment, in the form you need. The word “soft” in software stops being just an adjective and becomes a literal description of what you do: you change the code to fit the problem, instead of changing the problem to fit the “right” framework.

The second part is technical. A cross-platform library is not only a convenient and free way to get an application that runs everywhere. It is also an extra layer of abstraction that makes you write code not the way you want. It keeps you from using native things the way they were meant to be used. It brings its own complex bugs that often need non-trivial workarounds, and it makes you give up features simply because the framework does not support them yet.

So I chose to specialise for one protocol. And the other decisions went the same way. Wherever I could, I avoided big ready-made solutions. They are good, great, time-tested (no sarcasm), but they would make me bend the code the way the framework needs, not the way I want. I did not want to run, every time, into a feature that “cannot be done the way you want because of this limitation of the framework” and then kick myself, thinking “why did I make this choice? Nobody forced me to use Qt/GTK/your favourite framework”.

Drawing everything myself

The client draws its whole interface by itself, and this gives simple answers to problems that are hard with a toolkit. For example, selecting text across many messages, the problem I had with Fractal: in Exerakt the whole timeline is one text document, so a selection simply goes across messages. HTML is laid out my own way, and themes have no limits beyond what I write.

The stack is small: winit for the window and input, softbuffer to put pixels on the screen, tiny-skia to draw, and parley to lay out text. Everything is drawn on the CPU first, and the GPU comes in only if the CPU turns out to be too slow.

These crates are young, so before going further I wrote down the criteria the first milestone had to check. If one of them failed, one crate would be replaced and the others would stay:

• selecting text across several messages and copying it;
• smooth scrolling of a timeline of 2000 messages at 1440p, on the CPU.

The first milestone

The first milestone was a client that can read: log in, restore the session, show the rooms and their history, and send a plain message. It took five steps:

1. The client’s model: the data the screen shows, such as rooms and their messages.
2. The layer that talks to Matrix: login, the session, and sync through matrix-sdk.
3. The window, drawing, and text.
4. The views: the room list, the timeline, and the message field.
5. Connecting all of it into a working client and testing it end to end against local Matrix servers (Conduit and Synapse).

Selection across messages works. A frame of the timeline with real data at 1440p is drawn in 3 to 4 milliseconds, less than a quarter of the time a frame has at 60 Hz. So the stack stays.

Against Element

Then I compared it with Element Desktop on the same account with 14 rooms, both with a stored session:

Of course the client is still very far from its final shape, and it does not have a huge number of features that can grow its size and slow its start. Even so, the numbers are very encouraging, and they support my confidence that I should keep going in this direction.

The next post is about the second milestone: reading Matrix messages the way Element shows them.

In hindsight I can see that I made a big mistake here. I ran Bonsai not as an agent but purely as a text generator: one request per attempt, no tools, the answer is the full content of the files it changes. Now the mistake looks obvious, but in the first iteration I did not give this difference much weight. The numbers below are about that setup.

The numbers

By now Bonsai has gone through 28 cards in 34 runs. All 28 were accepted in the end, 12 of them on the first attempt of the first run. Six cards went back to Claude at least once.

More interesting is where the failed attempts failed. Every attempt goes through the same checks in the same order: the files it touched, compilation and clippy, the project’s rules, the tests, and Jev. Over all runs:

• 27 failed attempts stopped at compilation and clippy;
• 4 at formatting;
• 3 at Jev;
• none at the tests.

When Bonsai’s code compiled, the logic was almost always right.

What went wrong

About two thirds of the compilation failures were code that does not compile at all. The most typical one is a method that does not exist, like this check of an id’s first character:

if input.first() != Some('@') {

Slices have first(), strings do not. The same happened with is_ascii_digits and with a normalized method on one of our own types. The rest were the places where Rust is hard: moving a value out from behind a reference, a temporary that does not live long enough, a function that returns a value where Ok(value) is expected. The remaining third were clippy’s style remarks, such as two nested ifs that can be one.

Claude and Jev classify every failure, and 30 of 34 were classified as trivial: a typo, a wrong method name, a missing import.

The most expensive failure was of a different kind, and it was in the previous post. A card asked for date calculations and named the algorithm without writing it down. Bonsai spent its whole budget twice trying to derive the formulas and did not produce any code. Where it had to know something rather than write something, it had no way to find out.

My conclusion after all these runs was this: Bonsai has no problem with how to write code, but there is a lot it does not know. And in the setup I chose it had no way to check: it could not look up a signature in a dependency or run the compiler before answering. It learned about a missing method only from the next attempt’s error list.

What changed

Bonsai’s prompt did not change: it is the same as on the first run. What changed were the cards. After the first step Claude got two rules for writing them: a known algorithm is written out, not named, and every signature is checked against the project’s rules before the card is cut. In the runs since 19 October the first attempt compiles in 6 of 9 runs, against 7 of 25 before. Maybe the better written cards are the reason, but the cards themselves are different and the numbers are small, so I cannot say that for sure.

The orchestrator also runs clippy --fix before the clippy check now, so clippy’s style remarks that it can fix by itself would no longer cost an attempt. Since it was added, it has not had anything to fix.

All of this cost about four hours of Bonsai’s time for 28 cards, most of it spent on reasoning tokens.

Running Bonsai as an agent, with tools and the compiler inside the loop, is still ahead. The next post is about Jev and how much its answers can be trusted.

The plan

The work goes from the top down, in three levels. The whole client is split into milestones, each ending in something that runs end to end: first a client that can only read, then one that can write, then accounts, threads, and so on. When a milestone starts, Claude splits it into steps, and when a step starts, Claude splits it into cards. Detail is added only one level ahead: only the milestone we are working on is split into steps, and only the current step is split into cards. This is called rolling wave planning. Every step ends with a review, and the review may change the plan for the next steps, so there is no detailed plan for work that the earlier work can still change. But there is always a more or less clear picture of what lies ahead and what is still left to do. The diagram below shows the whole cycle.

flowchart TD
    M["Milestone<br/>(e.g. a client that can write)"] -->|Claude splits the current milestone| S[Steps]
    S -->|Claude splits the current step| C["Cards: signatures,<br/>frozen tests, task"]
    C --> L[["Card loop<br/>(Bonsai writes, checks, Jev decides)"]]
    L --> R[Step review by Claude]
    R -->|next step| S
    R -.->|may change the plan| M

The unit of work is a card: one small task for the executor (normally Bonsai), usually a few functions in one file. Before a card goes anywhere, Claude has already written everything around it: the types and function signatures with empty bodies, the module’s description, and the tests. The tests are frozen. Bonsai sees the task, the files it may change and the tests, and fills in the bodies. If a test file changes by a single byte, the attempt does not count, so the model cannot pass a test by rewriting it. For a model working on its own this is the most important guard.

The cards are run by an orchestrator, a separate program that Claude wrote. For each card it asks Bonsai for code and then checks the result, cheapest checks first:

1. Did the model touch only the files it was allowed to touch?
2. Does it compile, and does clippy have nothing to say?
3. Does the code break any of the project’s mechanical rules, such as which crate may depend on which?
4. Do the tests pass?
5. Only then Jev: questions that a machine cannot check, like “does the code do what the task describes” or “does every function belong to the module’s stated responsibility”.

The first four checks cost nothing and are exact, so Jev only sees code that already works. When a check fails, the errors go back to Bonsai for another attempt. Jev also looks at the failure and decides whether it is worth retrying (a typo, a wrong method name) or whether the card should go back to Claude (the task itself looks wrong, or the model is going in circles). After three failed attempts the card goes back to Claude anyway. The second diagram shows one card’s way through this loop.

flowchart TD
    B[Bonsai writes the code] --> G1{"Only the allowed<br/>files changed?"}
    G1 -->|yes| G2{"Compiles,<br/>clippy clean?"}
    G2 -->|yes| G3{"Project rules<br/>hold?"}
    G3 -->|yes| G4{"Tests pass?"}
    G4 -->|yes| J{"Jev: does the code<br/>do what the task says?"}
    J -->|yes| A[Accepted, merged into the step]
    J -.->|not sure| JD[Claude as judge]
    JD -.-> A
    G1 & G2 & G3 & G4 & J -->|no| F{"Jev: what kind<br/>of failure?"}
    F -->|"typo, local mistake:<br/>errors go back"| B
    F -->|"card looks wrong,<br/>model goes in circles,<br/>attempts used up"| E[Back to Claude]

There is one more participant for the start. When I searched for how Jev is used, I found the recommendation to calibrate it on your own tasks before relying on it fully. So Claude was added as a judge: every question Jev answers is also put to Claude, and while this calibration is on, Claude’s answer decides (the dashed arrows in the second diagram). On the first step Jev cost about 300 times less than Claude for the same questions, which is the reason to want it to take over.

Bonsai (Bonsai 2 27B by PrismML) was chosen for a simple reason: on my machine, with one 12 GB graphics card, it had a reasonable balance between speed and quality, about 56 tokens per second with 128K tokens of context. A better candidate could probably be found, but at that moment it was a model that definitely worked, had a good context size, and I did not want to spend time searching for another one.

An independent review of Bonsai said that it plans well and codes poorly in agentic tasks. Fixed signatures and frozen tests were meant to compensate for exactly that: the model does not have to design anything, only to fill in bodies until the tests pass.

The first run

The first step was the client’s model: pure view state, room list, timeline, dates, previews. Six cards, 75 frozen tests. One card of six passed on the first attempt, one more on the second, and four came back to Claude.

When I looked at why, only one of the four was Bonsai’s failure: it converged on the answer, but ran out of attempts on the way. The other three were not its fault:

• The orchestrator itself had a bug. Bonsai’s code had passed every check, and the last thing left was to ask Claude, as the judge, the same questions Jev had answered. That request failed: the orchestrator named each question after the project rule it checks, with names like R:S4, and the colon is not allowed in the format Claude’s interface expects for these answers. So a correct card came back as a failure.
• One card asked for a function that returns bool when an operation fails. The project’s own rules say that failures are typed errors. Bonsai followed the card, and Jev correctly rejected the code for breaking the rules.
• One card asked for date calculations and said “use Hinnant’s algorithm”, without writing the algorithm down. Bonsai spent its whole token budget twice trying to derive it and did not produce a single line of code.

So in this scheme the architect’s mistakes look like the executor’s mistakes. A badly written card shows up as a failed attempt, and from the outside it is hard to tell which side is wrong.

What changed

The fixes went to where the problems were. The orchestrator got its bug fixed and a check that catches it before a run. The card with bool returns a typed error now. The date card got the formulas themselves, checked against Python’s datetime for every year from 1 to 9999. And the process for writing cards got two new rules: a known algorithm is written out, not named, and every signature is checked against the project’s rules before the card is cut.

The more general lesson is that the pipeline has to tell these failures apart. That is why Jev classifies every failure, and why “the card itself looks wrong” goes straight back to Claude instead of costing Bonsai more attempts.

What Bonsai itself does with the cards it gets is the subject of the next post.

I like trying new things. At work I use AI all the time, but as an assistant: I am in the loop for every feature. I agree with the agent on what it is going to write, correct it as it goes, and keep its work in line with the bigger picture of the project. This works very well. What I had never tried is the opposite: let the model work on its own from start to finish, choose the solutions, find its own mistakes and improve its own code.

At the same time I wanted to use my own experience. A model works better when its work is structured: there is a problem statement, a breakdown into tasks, steps that end with tests and a review of the design. Not “make it good”, but a process that looks like the classic way of building software, the one I know works. There is nothing new here. It is only a formalisation of what already exists, written down precisely enough to be run without me.

And I did not want to spend thousands of dollars on it. So the third thing I wanted to see is how the models work together when each one does the job that fits its price:

• Claude is the organiser and planner: it designs the interfaces, writes the tests and splits the work into small tasks.
• Jev, a small and very cheap decision model, is the orchestrator: it checks the result against the task and decides what happens next.
• Bonsai, a 27B model running locally on my GPU, does all the grunt work.

I had heard about this kind of setup, but I had never used it in practice.

The project

Then the question was what to build. I did not want another to-do application or another proof of concept. I have made too many pet projects like that, and they led nowhere. This time it had to be something I really need and will use every day.

I chose a desktop client for Matrix. Matrix is an open, federated chat protocol: anyone can run a server (I run my own), servers talk to each other the way email servers do, rooms can be end-to-end encrypted, and any client works with any server.

This may look like a strange choice, because there are already many Matrix desktop clients. But each of them has something that does not work for me:

• Element is the closest to what I need, but it runs only on Electron, which is heavy, and on my Hyprland desktop the left Alt key opens its menu, while I switch keyboard layouts with left Alt.
• FluffyChat is poor in features: it turns message HTML into its own widgets, so formatting comes out badly, and it has no threads.
• With the others there were other unpleasant things: no custom themes in Nheko, no text selection with the mouse in Fractal.

These are all good clients, and I have no big complaints about them. I could live with their shortcomings or send pull requests. The closest to my wishes was Tesseract, but it develops fast enough that keeping a fork would be expensive, and I did not want to carry its Windows and macOS code.

So two things came together: the wish to try a process where models write the code, and the wish to have a good client made for my needs, one I can change in any direction without depending on the plans, means or taste of its maintainers. That is how Exerakt was born: a Matrix desktop client for Wayland, written in Rust.

How it is going is the subject of the next posts.