041

The lab built for agents

Inside 041’s research workflow: agents, experiments and the infrastructure connecting them.

A glowing green branching network spreading across a dark hemisphere, with the 041 mark in the corner.

We are building 041 towards fully automated research. As AI accelerates research, a lab’s ability to put it to work becomes increasingly important.

Agents already do all the implementation work in our lab. We essentially never edit code ourselves. They launch training jobs, follow experiments and produce the plots we need to understand what happened. Our attention increasingly goes to the questions, the evidence and the next direction.

This changes what we need from our tools. Everything needs to work through CLIs or APIs. We have been rebuilding the infrastructure around the research loop: discuss an idea with an agent, let it implement and test several variations, examine the results together, then pursue the most promising directions.

Our role is increasingly to direct the research. Agents handle the implementation and coordination that used to require our constant attention.

Experiment tracking that agents can use

Experiment tracking was the first piece we built.

I had liked Neptune, and Weights & Biases was an established alternative. But we no longer needed the dashboards, whereas agents needed the relevant information.

Metrics tracking is the foundation for well-orchestrated research. OpenAI agreed to acquire Neptune in a deal reportedly worth under $400 million in stock.

For us, experiment tracking also serves as the lab’s memory. Agents need to learn from earlier experiments, but they cannot carry every run and every measurement in their context. They need to retrieve the right evidence when a question arises.

We built Metrics around that workflow. An agent can find relevant runs, select the measurements and training steps it needs, and query the results with SQL. It might compare validation loss across an architecture sweep, then inspect a particular diagnostic around the point where one run diverged. We can record many metrics without making the agent read them all.

Three relevant fragments are selected from a large archive of experiments and brought into a small context window.
Record broadly. Retrieve precisely.

Other systems can provide this information too. The difference is how directly an agent can get the exact data it needs, with fewer intermediate steps, less irrelevant output and less context spent on retrieval. More of its context stays available for the research itself.

When we want to inspect the results ourselves, an agent can hand us a link to the exact comparison. More often, I simply ask an agent to put the specific plot I need directly in the chat.

ML research rethought for agents

Next came the framework. We grew tired of agents writing more code than the experiment needed. The extra code consumed tokens and created more opportunities for errors.

We built S-expGPU so an agent can describe an experiment in one Lisp file: the data, model, objective, optimizer and run. The compiler owns the training machinery. Lisp lets us build compact abstractions around the ideas we are testing. Macros capture repeated structure, leaving each variant focused on what actually changes.

Equivalent PyTorch and S-expGPU implementations of a quadratic layer and a gated residual layer. A four-line Lisp macro generates the parameter declarations and model function for both.(full size, opens in new tab)

Lisp syntax is tighter than Python. Macros enable code that outputs code.

The compiler checks tensor shapes on a laptop, without a GPU or training data. When something is wrong, the agent gets a structured diagnostic pointing to the source and explaining the mismatch. More mistakes are caught before they become failed GPU jobs.

Startup time matters, as research involves many short attempts: check an idea, fix a mistake, run a small test, change something. Time spent starting processes and compiling models adds up. Interestingly, also the training performance has been better than with PyTorch. In our published benchmark suite, S-expGPU has lower training-step times than torch.compile across all 14 measured workloads on NVIDIA A100 GPUs.

Soon after we brought S-expGPU into our workflow, we made our biggest research progress yet. A Friday-evening idea became big results by Sunday. We then investigated the architecture across code, Sudoku and chess. We describe those results in From reasoning towards superintelligence.

It changed how I work, too. Smaller implementations let an agent keep more related variants in its context at once. I can explore several ideas together, compare them and carry what we learn from one into the others.

We have also separated the Lisp frontend from the execution backend. S-expGPU already runs on NVIDIA GPUs and Apple GPUs through Metal. AMD support is nearly ready. Optimization is still ongoing, though our benchmarks already show it outperforming compiled PyTorch. This is a substantial area of investment across the industry: Modular raised $250 million in 2025 to build its unified compute layer. Our focus is the research workflow we need, with control over the path from an agent’s experiment to the code running on a GPU.

Compute that keeps up

The experiments need to be run somewhere. We want to specify the hardware an experiment needs and find the cheapest available machine that fits. Access to multiple clouds is critical, as today GPU availability can be a challenge in itself.

SkyPilot addresses this problem and has raised over $20 million. We used its open-source software, but in our workflow the time agents spent waiting for API responses became a bottleneck. We reached a point where building around our own requirements made more sense than continuing to patch it.

The result is rrun. An agent sends a folder, a command and hardware requirements. rrun selects a machine across the connected clouds, runs the job, streams the logs and removes the machine when it is no longer needed.

An experiment is routed to one available GPU among three compute pools.

rrun finds the cheapest fitting machine across multiple clouds.

Fast iteration is a core requirement. Job, status and log requests return in a fraction of a second. Machines can stay warm between attempts, allowing the next job to start in two or three seconds. These details matter when an agent is working through a sequence of experiments. Every unnecessary wait delays the next useful observation.

Making cheap compute usable

Spot instances can stretch a training budget considerably. The tradeoff is that you may lose the machine mid-run, but this is fine if the training resumes reliably. S-expGPU supplies the resumable run, rrun supplies the next machine, and Artifacts makes the checkpoint available.

The storage economics matter. A training run might write hundreds of checkpoints but read one back only when it needs to resume. Sending every checkpoint to a central bucket would be needlessly costly. Artifacts writes checkpoints to a bucket near the GPUs. When a run moves to another cloud, it reads the latest checkpoint once, then writes subsequent checkpoints near its new machine.

Three compute locations write checkpoints to nearby storage. One checkpoint transfers to the next machine each time training resumes elsewhere.
Write locally. Resume anywhere.

Our example case saves a 20 GB checkpoint every half hour for a week. Sending every checkpoint across clouds can cost roughly $700 in transfer fees. Keeping writes local reduces cross-cloud transfers to one checkpoint per move, roughly $2 per resume at the example’s assumed rates.

What this enables

Each of these layers could support a company of its own. At 041, we are building them for the foundation for something bigger. The lab is built to get to the frontier AI, and win. We win through faster progress with fewer resources. S-expGPU is the perfect example of it, how it’s allowing us implementing more related ideas on parallel. Any model’s lead disappears quickly. Sustained speed matters more than any single result.

← ALL POSTS