Nov 18, 2026

Spending 100 Days Automating My Job

100 days, two engineering bottlenecks automated, and roughly 20 engineering hours saved every week. An inside look at building self-healing invoice ingestion and automated code review at Loop, the challenges behind them, and what it takes to automate engineering work at scale.

I was surprised by how quickly I settled into life at Loop. Within my first week, I was laughing with my new teammates as we broke a record three (3!) glasses trying to make one cup of coffee, chatting on the office couches in the late afternoon, and putting up my first PRs (pull requests, which are how we make changes to the codebase). A hundred days in, I figured it was time to not just reminisce about our tennis and happy hour outings, but also what I’ve been working on.

On-call took longer to settle into. My first shift overwhelmed me. Failures kept landing faster than I could clear them, and I’ve taken several more shifts since. Most of what I built over these 100 days started as an attempt to make that list looming over me shorter. 

My first project: self-healing ingestion

One of Loop's core offerings is freight audit and pay. When a carrier bills one of our customers, we check the invoice to make sure the customer isn’t overpaying. Before we can audit anything, though, we have to ingest and normalize the invoice. Ingestion can fail for all kinds of reasons, from missing fields to misspelled addresses (more detail here). Reacting to these failures make up the bulk of on-call. 

Automating them away was my first assignment, and I thought I’d been handed the easiest project in the world. My plan was simple: toss each failure at the smartest frontier model. If the model couldn’t fix it, show it a few examples of fixes that worked. If it succeeded, have it write down its method and follow that playbook the next time the problem came up. This plan was too naive, for two reasons.

The first is volume. Loop processes tens of thousands of invoices a day, and our team sees hundreds of failures a week. One agent per failure means hundreds of agents, each starting from zero. The second is that it doesn’t acknowledge the race condition that happens when two failures share a root cause and two competing fixes go up for it.

Instead, we built a system of agents that works on root causes instead of individual failures. First, failed workflows are grouped deterministically into rough issues. A persistent orchestrator, which owns the health of the whole system, investigates each issue’s root cause and assigns each root cause to a short-lived fixer: an instance of Mack, our internal code agent. The fixer puts up a PR, redrives the failed workflows once the fix deploys, and closes the issue if they succeed. If they don’t, it hands the issue to an engineer. Because fixers are disposable, the orchestrator’s context stays intact no matter how many come and go.

End to end: a carrier starts billing a surcharge we’ve never seen, and every invoice from them that day fails on an unknown charge code (ingestion failure example). The tracker groups those failures into one issue. The orchestrator checks the grouping and hands it to a fixer, which opens a PR adding the charge code to our config. If the PR is routine, our automated code reviewer approves it (more on that below). The fixer then redrives the failed invoices, confirms they ingest, and closes the issue. In an ideal scenario, just one engineer needs to be in the loop.

The system handles the failures with shapes we’ve seen before, which covers about half of the ingestion failures that used to land on-call. That turns roughly 20 engineering hours a week of human labor into tokens.The other half still needs a human decision or outreach to get missing data.

My second project: automated code review

We put up a lot of PRs, and a lot of PRs means a lot of waiting. Sometimes I’d get deep into a problem, put up the fix, and then wait while someone loaded my context into their head, only to stamp it anyway. After a few weeks, I was frustrated enough to bring it up with my teammates.

It turned out to be a shared problem. They pointed me to an internal doc showing that about 80% of our PRs merge without substantial human comments. That means most of our PRs could be stamped automatically. It also means the PRs that need careful review get lost in a sea of stampable ones. To my surprise, the day after I complained, we started building a code review agent.

Most code review tools go after the PRs that warrant the closest scrutiny. We started from the other end and taught our reviewer to recognize what’s easy to review: documentation updates, feature flag flips, and the PR shapes we see over and over, like the charge-code fix above. Clearing those off human reviewers’ plates frees their attention for domain-heavy decisions.

Before we let it approve anything, we pulled our full PR history, flagged every PR that was later reverted or broke something, and ran the agent over the whole history. That showed us which categories of change it could judge reliably and which it couldn't, and limited it to the categories where we were confident it wouldn't miss a breaking change. Because we kept that scope narrow, the reviewer hasn’t wrongly approved a PR in the month it’s been running.

We built the reviewer for our codebase and taught it how we review, which keeps it cost-effective: the average review costs about a dollar and takes about five minutes. It also closes the loop with Mack, which opens and maintains PRs based on reviewers’ comments. 

Of the PRs it runs on, the reviewer now approves about 40%. That cuts our wait time and leaves more engineering time for the PRs that need it.It’s also a step toward our vision for the Loop software factory: code and configs that heal themselves, with Mack writing the fixes, the reviewer clearing the routine ones, and engineers stepping in only where judgment is needed.

Why these worked

Neither project is astonishingly clever, but they have three things in common.

First, the problems are tightly scoped. Ingestion failures differ in their content, but the shape of the problem repeats, and stampable PRs work the same way. You can trust an agent that only handles a shape it has seen a thousand times far sooner than one that has to handle a universe of potential tasks.

Second, we built on reliable signals. For ingestion, we can check for things like charges summing to the invoice total, dates being within a valid range, or the invoice number not being one we’ve already paid. For review, a PR either broke something later or it didn’t, and we have our full PR history to backtest on. Those signals mean we don’t have to get either system right on the first try. We can keep measuring and hill-climbing.

Finally, Loop’s existing primitives let me move fast. The persistent agent infrastructure gave me the orchestrator, so I didn't have to solve the hard, generic problem of keeping an agent alive for days. Mack gave me a PR author that could take feedback. When I had a question about why something worked the way it did, our internal agent could find the answer even if it was buried in a Slack channel, a wiki page, or an old PR. I spent very little of my first quarter on plumbing and almost all of it on our actual problems.

I also want to acknowledge all the amazing people that have made me look forward to coming back to the office each week. I’m glad I didn’t automate my job away, just some of the annoying parts, and I’ll see everyone again next Monday!

Get Started

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.