Element as the reference

For the look and the features I like Element a lot, so it is no surprise that I chose it as the reference for where the project should go. This also answered a question the pipeline cannot answer by itself. Jev and Claude check whether a card is done, but nobody checks where the whole project is going. For now I want to reproduce the look and feel of Element, so every feature of Exerakt is finished by comparing Exerakt and Element side by side: the same account, the same room, the two windows next to each other.

What was built

The second milestone was a client that reads Matrix the way Element shows it. It took six steps:

1. Turning the HTML of formatted messages into the client’s own representation.
2. Drawing it: quotes, code, lists, headings, tables, links, mentions, and spoilers, with selection and copy working across all of them.
3. Showing replies, edits, reactions, deleted messages, and messages that could not be decrypted, with the same wording as Element.
4. Read receipts and the line that marks where the user stopped reading.
5. Pictures and colour emoji.
6. Local time, a status line for the connection, and a final check of the whole milestone against Element.

Side by side

A card is checked by its tests, but nothing tests how the whole client looks. So every step ended with a comparison, and since the process is automatic, Claude did it as well:

1. A local Conduit server ran in Docker. A small program created rooms on it and filled them with the kinds of messages the step was about: formatted text, replies and reactions, pictures.
2. Claude started both clients, logged in to the same account on this server, on a separate workspace of an invisible (headless) Hyprland output, so nothing appeared on my screen. It captured Exerakt with grim, a screenshot tool for Wayland, and Element through its developer tools.
3. Claude put the two screenshots side by side and compared them. What could be fixed within the step was fixed there; everything else became a task for a later step.

Below are three of these comparisons from the second milestone, Exerakt on the left and Element on the right. They are not final screenshots: they show the clients the way Claude saw them at the moment of each check. Two of the differences in them, local time and colour emoji, were done later within the same milestone; the others are still ahead.

Works:

• bold, italic, inline code, and strikethrough;
• code blocks, numbered and nested lists, headings, quotes, tables, and rules;
• coloured text, spoilers, links, and mentions, all in the same places as in Element.

Not yet:

• syntax highlighting and line numbers in code blocks;
• dimmer quotes, and a mention shown as the user’s name with an avatar;
• local time: at that moment Exerakt showed UTC, local time came later in the same milestone.

Works:

• the replied-to message above the reply;
• the edited mark;
• reactions with their counts under the message;
• messages grouped by sender the same way.

Not yet:

• colour emoji, which came later in the milestone;
• the replied-to sender with an avatar and a coloured bar;
• icons before deleted and undecryptable messages.

Works:

• the read marker line after the same message;
• the same times;
• the badge of the other user who read the room, on the same message.

Not yet: avatars and coloured sender names, here and everywhere else.

Other people’s code

As I said at the start, drawing the messages itself went fairly smoothly. The problems came from two other places: the servers, and one of the libraries.

The servers

Matrix servers that speak the same protocol do not always send the same things. But first, a few technical details about two features where this showed up.

The first one is read receipts. When you read a message, your client tells the server, and the clients of other people in the room show it, usually as small avatars next to the last message each person has read. The second one is the read marker: a line in a room that shows where you stopped reading last time, so that when you come back you see where the new messages begin. The server keeps its position, so the line is in the same place on all your devices.

A Matrix client gets news from the server through one long-lived connection, and both of these should come through it. With Synapse they did. With Conduit they did not come at all, so Exerakt never learned where the line was or who had read what. I found this only because Claude checked the feature against both local servers. Exerakt now asks the server for the line separately every time a room opens, which Conduit does answer. Other people’s read receipts on Conduit are still missing.

The emoji

The library problem was with emoji. They came out in black and white. To draw a character, parley, the text layout library, looks for a font that has it, and it looked through the regular fonts first. Some of them have black-and-white emoji, so they were picked before the colour emoji font. The fix was to find the emoji in the text in Exerakt itself and send them to the colour emoji font directly. As a side effect, the first emoji the program shows now takes 0.7 milliseconds instead of 97 (yes, almost 140 times faster), because the slow part had been the search through the system’s list of fonts.

Additional lessons

It turned out that Bonsai is not the only one who does not know everything. When you reply to a message in Matrix, the client also puts a copy of the original message into the reply, for older clients that cannot show replies. A modern client has to remove that copy before showing the reply. In the first step of the milestone Claude cut a card asking Bonsai to do exactly that, and Bonsai did it. Only then it turned out that matrix-sdk-ui, the part of the Matrix SDK that builds the list of messages, already removes the copy, so the card was withdrawn.

The mistake was Claude’s, the architect’s, like the ones in the first run, and it led to a new rule for writing cards: before cutting a card, read what the dependency already does with the data.

The next post is about the third milestone, writing messages, and about how the plan for the pipeline met reality.

Why Wayland only

With Wayland the reader may ask: why this way? Why not go the standard route, choose a good, time-tested cross-platform library and use it?

My answer has two parts. The first one is philosophical. I believe that with LLMs it is possible (or will be in the near future) to build specialised solutions efficiently. Do you want it for macOS or Windows? You create a project like mine, feed it the code of Exerakt, and after some time you have your own version of the application, built and tuned for your environment, in the form you need. The word “soft” in software stops being just an adjective and becomes a literal description of what you do: you change the code to fit the problem, instead of changing the problem to fit the “right” framework.

The second part is technical. A cross-platform library is not only a convenient and free way to get an application that runs everywhere. It is also an extra layer of abstraction that makes you write code not the way you want. It keeps you from using native things the way they were meant to be used. It brings its own complex bugs that often need non-trivial workarounds, and it makes you give up features simply because the framework does not support them yet.

So I chose to specialise for one protocol. And the other decisions went the same way. Wherever I could, I avoided big ready-made solutions. They are good, great, time-tested (no sarcasm), but they would make me bend the code the way the framework needs, not the way I want. I did not want to run, every time, into a feature that “cannot be done the way you want because of this limitation of the framework” and then kick myself, thinking “why did I make this choice? Nobody forced me to use Qt/GTK/your favourite framework”.

Drawing everything myself

The client draws its whole interface by itself, and this gives simple answers to problems that are hard with a toolkit. For example, selecting text across many messages, the problem I had with Fractal: in Exerakt the whole timeline is one text document, so a selection simply goes across messages. HTML is laid out my own way, and themes have no limits beyond what I write.

The stack is small: winit for the window and input, softbuffer to put pixels on the screen, tiny-skia to draw, and parley to lay out text. Everything is drawn on the CPU first, and the GPU comes in only if the CPU turns out to be too slow.

These crates are young, so before going further I wrote down the criteria the first milestone had to check. If one of them failed, one crate would be replaced and the others would stay:

• selecting text across several messages and copying it;
• smooth scrolling of a timeline of 2000 messages at 1440p, on the CPU.

The first milestone

The first milestone was a client that can read: log in, restore the session, show the rooms and their history, and send a plain message. It took five steps:

1. The client’s model: the data the screen shows, such as rooms and their messages.
2. The layer that talks to Matrix: login, the session, and sync through matrix-sdk.
3. The window, drawing, and text.
4. The views: the room list, the timeline, and the message field.
5. Connecting all of it into a working client and testing it end to end against local Matrix servers (Conduit and Synapse).

Selection across messages works. A frame of the timeline with real data at 1440p is drawn in 3 to 4 milliseconds, less than a quarter of the time a frame has at 60 Hz. So the stack stays.

Against Element

Then I compared it with Element Desktop on the same account with 14 rooms, both with a stored session:

Of course the client is still very far from its final shape, and it does not have a huge number of features that can grow its size and slow its start. Even so, the numbers are very encouraging, and they support my confidence that I should keep going in this direction.

The next post is about the second milestone: reading Matrix messages the way Element shows them.

I like trying new things. At work I use AI all the time, but as an assistant: I am in the loop for every feature. I agree with the agent on what it is going to write, correct it as it goes, and keep its work in line with the bigger picture of the project. This works very well. What I had never tried is the opposite: let the model work on its own from start to finish, choose the solutions, find its own mistakes and improve its own code.

At the same time I wanted to use my own experience. A model works better when its work is structured: there is a problem statement, a breakdown into tasks, steps that end with tests and a review of the design. Not “make it good”, but a process that looks like the classic way of building software, the one I know works. There is nothing new here. It is only a formalisation of what already exists, written down precisely enough to be run without me.

And I did not want to spend thousands of dollars on it. So the third thing I wanted to see is how the models work together when each one does the job that fits its price:

• Claude is the organiser and planner: it designs the interfaces, writes the tests and splits the work into small tasks.
• Jev, a small and very cheap decision model, is the orchestrator: it checks the result against the task and decides what happens next.
• Bonsai, a 27B model running locally on my GPU, does all the grunt work.

I had heard about this kind of setup, but I had never used it in practice.

The project

Then the question was what to build. I did not want another to-do application or another proof of concept. I have made too many pet projects like that, and they led nowhere. This time it had to be something I really need and will use every day.

I chose a desktop client for Matrix. Matrix is an open, federated chat protocol: anyone can run a server (I run my own), servers talk to each other the way email servers do, rooms can be end-to-end encrypted, and any client works with any server.

This may look like a strange choice, because there are already many Matrix desktop clients. But each of them has something that does not work for me:

• Element is the closest to what I need, but it runs only on Electron, which is heavy, and on my Hyprland desktop the left Alt key opens its menu, while I switch keyboard layouts with left Alt.
• FluffyChat is poor in features: it turns message HTML into its own widgets, so formatting comes out badly, and it has no threads.
• With the others there were other unpleasant things: no custom themes in Nheko, no text selection with the mouse in Fractal.

These are all good clients, and I have no big complaints about them. I could live with their shortcomings or send pull requests. The closest to my wishes was Tesseract, but it develops fast enough that keeping a fork would be expensive, and I did not want to carry its Windows and macOS code.

So two things came together: the wish to try a process where models write the code, and the wish to have a good client made for my needs, one I can change in any direction without depending on the plans, means or taste of its maintainers. That is how Exerakt was born: a Matrix desktop client for Wayland, written in Rust.

How it is going is the subject of the next posts.