Berlin.rb #3,

Rails architecture in the age of AI

By Andrzej Krzywda

Chapters

Summary

This summary was written with AI from the recording, then checked by Berlin.rb and Andrzej Krzywda. Spotted a mistake? Write to hello@berlinrb.org.

Andrzej Krzywda has run Arkency, a consultancy for large legacy Rails applications, for almost twenty years. He argues that architecture still matters with AI, because agents don't deal well with unstructured codebases. Arkency models the business as domains that take commands and publish events, process managers that react to those events, and read models built from them, a vocabulary agents handle well. With that structure and AI, CHI Cargo, a cargo handler at Frankfurt Airport, shipped a new repackaging operation in around two weeks. He expects people to review less code, so he proposes a harness: ArchSpec to enforce architecture rules and track known violations, and Mutant to check test coverage, so agents can refactor large codebases faster.

Six events, and a calmer view 0:21

Andrzej is a programmer, an architect and an entrepreneur, and this year he didn't know what to expect, from the business side or the programmer side. So over the last two months he went to six conferences and meetups, mostly in the Rails community, to talk to as many people as possible. He still doesn't know much, he says, but he feels calmer. Things are not as bad as the most pessimistic people claim, even if the job market is not great in some segments. He expects strong growth in the economy and in innovation, pointing to the discoveries now happening in mathematics, the foundation of the industry. But things are changing, so everyone should prepare for a change.

The moods differed. EuRuKo in Brno felt more pessimistic, while an Omarchy meetup was upbeat: you can tweak your OS and do everything with Linux. Rails World shook his thinking the most. There DHH told the audience he was rewriting everything in Rust. Andrzej calls it a good wake-up call, and its nicest side effect, which DHH probably didn't intend, was that the Ruby community started working on Ruby's performance. Andrzej says Spinel and Roundhouse already work very well for Rails applications, and that within the next year Rails apps should probably reach a performance that, if not close to Rust, is comparable to it. A .NET meetup in Wrocław turned out to be an AI meetup, which made him ask whether we will still need language meetups in five years. He thinks we probably will, because Ruby people share values and a mindset. A conference for IT entrepreneurs in Poland had the gloomiest mood, though the charts and numbers were not so bad. Finally, at an AI Tinkerers meetup he met people who build AI. A lot happens in Wrocław, he says, because Callstack invests heavily in AI.

The wall has moved 5:31

Arkency has always focused on bigger, older Rails applications. Andrzej's message is that architecture still matters, because agents don't deal well with unstructured codebases. Arkency's own flavour combines domain-driven design, CQRS and events. They are not the only possible techniques, but they work for Arkency.

Rails projects used to hit a wall around their second or third year, when the codebase got so complex that new features slowed down, and that was when a CTO or CEO would call Arkency. AI moves the wall further out, but it also helps teams hit the new wall faster. Past it, even an agent adds a feature and breaks two existing ones. Codebases also grow much faster. Arkency looks at dozens of Rails codebases a year. A million-line one used to turn up as a potential client about once a year, and this year it sees codebases that reach that size very quickly.

AI makes developers faster, but not instant. Andrzej points out that programmers used to be the agents, prompted by managers and the C-level. Now there are faster agents, so developers need new roles. When he asked who still writes code by hand every day, nobody in the room did. A friend told him he writes code by hand at weekends, as recreation. Developers are still programmers, Andrzej says, but the definition is shifting.

Mapping the business into code 10:33

Andrzej believes domain-driven design prepared Arkency well for AI, because DDD was always about mapping the business to the code. If the code reflects the departments and processes of the business, requirements use the same language as the codebase and are easy to follow. Modern DDD is about domains and events. The classic book from 2004 had nothing about domain events, and now they are a big part of it.

A domain is roughly a department, such as sales, marketing, pricing or taxes, and becomes a Ruby module or namespace, ideally decoupled from the others. That is hard in Rails. Active Record and "don't repeat yourself" lead to a normalised database where every piece of information lives in one place, but in a business a client means something different to marketing, sales and invoicing. Reflecting the business in a Rails codebase is a big change that used to take years. In it, the domains may share a database, but they don't look at each other's tables. They communicate by announcing what has happened.

Each domain has an input and an output: commands go in, such as asking marketing to capture a lead or sales to sell a product to a customer, and events come out, such as "lead captured", "sale happened" and "invoice issued". Business processes connect the domains. The simplest one says that when marketing captures a lead, sales should try to convert it. A more complicated one might first check the CRM to see whether the lead was a client before. DDD used to create a shared vocabulary for programmers, the business and the code. Now agents work on the code too, and in Arkency's projects non-technical people push code, which Andrzej calls risky. The more structured the codebase, and the more rules the agents are taught, the less likely their changes are to break production.

Domains, process managers and read models 16:35

Put more formally: domains take commands and publish events. Process managers take events from different domains and, once a certain combination has happened, issue a command to a domain. Read models are where CQRS comes in. Splitting a system into reads and writes feels odd at first, but the business already does it: it asks for reports. A read model is built from events, such as a list of invoices, clients or sales, and the user looks at it to make a decision and click something, which is a new command.

Read models are the perfect place for Active Record, Andrzej says. They are about creating rows, deleting rows and changing columns, with no need for callbacks and little need for validations, since validation happens in the domains. The hard part for many is that the data is denormalised, with a table for each kind of report. Read models act as caches that solve invalidation by always invalidating: every new event produces a new version.

In Ruby 19:18

Andrzej shows the building blocks in code:

Commands and events combine like an algebra. Andrzej doesn't claim this is the only or best architecture, but agents handle it very well, because its vocabulary is concise English that the programmers, the business and the code share.

A new operation at CHI Cargo 22:38

Arkency's client CHI Cargo runs a cargo business at Frankfurt Airport, moving shipments from planes through its warehouse to trucks, and back. About three years ago it started a domain-driven transformation of its Rails application, about ten years old by his recollection, using Arkency's Rails Event Store. Such transformations never reach a state where every department is cleanly separated, Andrzej says. They stay hybrid: events published from service objects, aggregates only where needed, read models that sometimes query other data.

Then came a new requirement: repackaging shipments, so that pieces of one shipment go onto different pallets, called load units, depending on whether they go to Hungary, France or Poland. Time was short, so programmers acted as product owners. Two of them went to the warehouse, because the warehouse workers are the real domain experts, not the people in the offices. 26:09

After those conversations, the programmers came back with a diagram of the change, made of the familiar building blocks and created by AI. The structure is always the same: an event comes in, a process manager issues a command, an aggregate handles it, and an event results. The key was an existing event, "piece checked in", the most important event in the business, which means a piece has passed every airport procedure and is ready to go. The new operation ends in that same event, so every process that follows it keeps working. The agents wrote the code, because they have a skill for each building block. The architecture has also become a business framework: the business asks how things are done, and can ask the agent too. The feature shipped in around two weeks, which Andrzej says was "crazily impossible" before AI and before the transformation. 30:05

The harness 30:17

Most of the room still reads the code before accepting a pull request. From his conversations, Andrzej expects that in some months fewer people will. Reviews are the bottleneck, and that can't last. The answer is this year's buzzword, the harness: tests, checks and guardrails that let an agent review a pull request.

The push for AI often comes from the C-level. When Arkency warns that less review means bugs in production, clients often answer that if the team is five times faster, they can accept some bugs, as long as one can be spotted and reverted within two hours. For developers it is binary, Andrzej says: it works or it doesn't. Many businesses think in thresholds, though not all of them.

Architecture rules with ArchSpec 32:43

ArchSpec, by Carmine Paolino, came up in Stephen Margheim's talk at the previous Berlin.rb. It lets you write architecture rules as code: module A never calls module B, a read model never calls a domain, a domain never calls a process manager directly. Shopify's Packwerk is a similar tool.

Both track technical debt. Add a rule and the tool accepts it but reports, say, 57 existing violations, which you store in a file. CI then rejects any new violation, which these days the agent rather than you would introduce, while staying quiet about the known ones. Combined with an agent, this becomes a way to pay the debt down. Fix the first violation together with the agent in small steps, working out where the data should come from instead, then ask it to write a skill for this kind of fix. The agent will get the second one wrong, but after the second, third or fourth time it does better and better. Once you have made four fixes together, the agent can probably handle the remaining 53 on its own, if the skill explains the fix well. 34:04

This needs very good test coverage, because without it the agents are blind: they fix something without knowing whether it works, and they are not precise enough to be trusted. Andrzej thinks there was never a better time to aim for 100% coverage, because generating tests is now "instant and free". 36:06

Mutation testing with Mutant 36:45

Andrzej calls mutation testing the only way to check whether coverage really is 100%. It tests the tests: it changes the code, for example a + 1 into a + 0, runs the tests, and checks that one fails. Given a module, it mutates its syntax tree and, as Andrzej puts it, finds "all the operators that were not tested". Andrzej uses and recommends Mutant, which costs $30 a month but has the largest number of mutations. Once the tests catch these, you can refactor more confidently and fix the ArchSpec violations. ArchSpec and Mutant together are tools for fixing an existing codebase.

Refactoring at AI scale 38:02

Everyone defines "software factory" differently. For Andrzej it is tooling and a harness that let an agent produce working code from an initial prompt or spec. More important to him is the other side: cleaning up the mess the agents create, which they still do, and the mess humans created before them. Refactoring is now possible at AI scale. Arkency's business was always improving legacy applications one careful step at a time, and that always took time. Now it is much faster, though not instant: get a unit to 100% coverage, introduce ArchSpec, define the rules, and tell the agent to fix it.

New roles 39:41

At Arkency's clients, non-technical or less technical people now push code, and Arkency's job is to let them ship. At the conference for IT entrepreneurs, one observation was that C-level people had never been as hands-on: they use agents themselves and are impatient, because they see what is possible. Managers and product owners want the same.

Andrzej sees developers wearing two hats. The forward deployed engineer translates business pain into technology. They talk to people who just do their work, discover pains the business doesn't always see, and create features, like the programmers who went to the warehouse. This person is close to a consultant and can be non-technical. The platform engineer builds and improves the harness, cares about architecture and refactoring, introduces ArchSpec and Mutant, and keeps the platform working even while non-technical people push code. 40:53

Agents in remote VMs 42:29

Andrzej presents the rest as newer trends he hasn't tested as thoroughly. His favourite agent harness is Amp, which can run each conversation about a repository in its own isolated environment in the cloud. He doesn't worry if an agent installs something bad there, because in his setup there are no secrets on it to leak. Using it for three months has been the biggest shift in his work this year: he can start 20 or even 50 agents, setup costs nothing, and his laptop is no longer the limit. His prediction, which he says three months ago he would have rejected as "the local development guy", is that local development may not survive the next year or two. His laptop is now for presentations, the browser and talking to Amp, and his code no longer runs on it.

Learning to manage agents 44:40

Developers are becoming architects, platform engineers and also managers of agents, though most never were managers. With agents you don't need to be diplomatic, but you do have to learn to tell others what to do. Andrzej borrows a technique for training new managers: shadowing. The new manager follows an experienced one into every meeting and call, then explains afterwards what happened, which is the fastest way to learn the skill. He shadowed people who work well with agents. At EuRuKo in Brno, Carmine Paolino, the creator of RubyLLM, showed him his setup for twenty minutes, and Andrzej learned more than in the six months before. 45:33

Residual architecture 47:42

Architects also have to prepare for the unknown. Arkency's book club read Barry O'Reilly's Residues, which promotes residual architecture, an idea operating at a different level from DDD. You start with a basic architecture, Active Record or DDD, and come up with many stressors from outside: what could hit the system, what would happen, and which parts become attractors. The aim is to cover every type of problem rather than every possible scenario, at least as an intellectual exercise in how you would fix each one.

The book's example is a network of electric vehicle charging stations, with stressors such as a car accident; stressors are often physical. For software, Andrzej asks: if it ever stops being safe to keep your website up, do you have one button to shut everything down, agreed with management, or will you run around in a panic in the middle of a Sunday night? A table of stressors against components shows which scenarios are most catastrophic, and which components fail together even when they aren't coupled in code, like a users microservice and a CRM. 50:15

Coming up with stressors is hard at first, but AI is very good at it, and programmers are good at risk analysis. The aim is to cover every type of problem, not every scenario. Even something unprecedented is useful, because it reveals a path through the system that more likely events could also take. Stressors also make good conversation points with the business.

Takeaways 51:46

From the Q&A 54:35

Background

Short explanations of ideas the talk builds on, written by Berlin.rb.

Mentioned in the talk

Given at Berlin.rb #3 at Le Wagon Berlin - Coding Bootcamp. Want to speak too? Submit a talk.