Berlin.rb #2,

Let Your Software Write Its Own Code: Runtime Code Generation in Production

By Rafał Cymerys

Chapters

Summary

This summary was written with AI from the recording, then checked by Berlin.rb and Rafał Cymerys. Spotted a mistake? Write to hello@berlinrb.org.

Rafał Cymerys, CTO of Upside, puts an AI agent inside an application and lets it write and run its own code. In his demo, a cat management system built with RubyLLM, one import agent takes a CSV with renamed or merged columns, an Excel file, a zip of CSVs and a PDF of trading cards. It writes Ruby scripts to explore and convert each one, asking the user when unsure. The same pattern pulls breed data from APIs, matches it to the cats, and builds reports and charts on request. His lessons from production: pick widely used stacks, give agents generic tools, always sandbox the code, and test prompts with a pipeline instead of tuning them by hand.

Software only does what we built it for 0:58

Rafał opens with Clippy, the assistant in Microsoft Word that could not help with anything outside its script, such as making every second paragraph blue. Real software has the same limit. Take Excel's CSV import. An application needs fixed data structures and validations to work and scale, so it can only accept what its developers thought of when they built it. The user, meanwhile, has data in whatever format another system gave them, and doesn't want to learn how the import works.

The usual answers are expensive. Teams spend months on matching screens with drag and drop and heuristics, a whole ETL industry sits between systems to move data from A to B, or users get a template to fill in, which works until someone fills it in their own way. The result is a support queue full of "my import doesn't work" in a consumer app, and Jira tasks for the important customer in an enterprise one.

An agent that writes its own code 5:19

A traditional import parses the file, validates it, and throws an error if that fails. Some teams put an LLM behind the parser to match columns, but that is still limiting: every new format, such as a TSV or an older Excel file, is still a new story in the backlog.

Rafał's alternative is an agent (a model, memory and tools, looping until the job is done) with one more tool: it can write code, run it, and iterate on it. The code runs in a sandbox, something the agent can break, because neither what the AI generates nor what the user uploads can be trusted, and the upload can influence the AI. The agent and the sandbox exchange data through the file system, which he finds a simple concept that goes surprisingly far.

The import demo 8:51

The demo app is a cat management system, the kind of cloud tool a vet practice might move to from a 1990s ERP whose exports are often incomplete or in odd formats. Its table holds each cat, its breed, its owner and their address, its favourite food and its last vet visit. One import agent handles every file:

The demo ran on Claude Haiku 5.5, "a pretty dumb model" in Rafał's words, but one that handles these scripts well. A year earlier, he says, GPT-4.1 often couldn't write a script that ran the first time, so the agent needed many rounds of fixing its own code. Opus solved the PDF faster, in fewer iterations, so the right model depends on the problem.

How the importer is built 20:22

The agent is built with RubyLLM. Its prompt says, in short: import this file, here is a way to look into it, write your own scripts and figure it out. All we want is a CSV that fits our schema exactly. It also tells the model that it can't run forever and should converge, rather than crunching through the whole file with LLM calls. The tools are:

Beyond imports

The same pattern, an agent that must return tabular data of a known shape and can write code to get there, covers more of what the clinic needs.

Breeds from APIs 22:19

To check whether a cat is over- or underweight, the vet needs a catalogue of breeds with their origin and weight range, and the user tells the app where to get it. Given an XML API Rafał made up, the agent wrote code to fetch and parse it. Given only the docs page of The Cat API, it downloaded the docs, found the endpoints and called them, then stopped because it needed an API key. Rafał pointed it to a demo key from the docs. It flagged that weights came in both metric and imperial units (he wanted kilograms) and that some breeds had no origin, a real flaw in that API's data, and asked how to handle them. It ended with 100 breeds. 23:47

Matching cats to breeds 25:47

Another agent matched the breed names in the cat data to the imported breeds. Most matched directly, but the API has no "Mixed Breed". The agent can't finish until every cat has a breed, so it asked, and Rafał told it to use Ragdoll. It finished with a summary of what it matched to what.

Reports and charts on request 26:42

Playing the user who wants a report but can't say exactly what, Rafał asked for the average number of kilograms that cats of each breed are over the breed's standard weight. The agent asked how to define standard weight, then returned the number of overweight cats in each breed and how many kilograms over they are on average. In his experience a reporting feature brings a flood of requests, because every user wants something different, and teams end up exporting everything to Power BI. Here the application writes the report itself.

Asked for a pie chart of the 10 zip codes with the most cats, the agent asked how to break a tie, exported the data, computed statistics, and drew the chart as SVG, calculating the slices itself. In Python it would reach for Matplotlib, but in Ruby it writes the chart by hand, which is good enough. Rafał's general point: reduce a problem to a coding problem, and let the coding model solve it at runtime, without having to think through every request users might make.

Key learnings from production 31:12

From the Q&A 35:20

Background

Short explanations of ideas the talk builds on, written by Berlin.rb.

Mentioned in the talk

Given at Berlin.rb #2 at Techtelmechtel. Want to speak too? Submit a talk.