Performance ratchet in CI
Technique
An automatic check on every pull request (a proposed code change) that measures something you care about, like how much data is sent. It posts the numbers on the PR and flags or fails regressions. Theo's replays real thread data, after small changes kept undoing his speed work.
Theo's current stance
He says the check matters most because it tells the agent this metric matters, like a stake a plant grows along. With it, agents often fix regressions themselves before reporting done.
Try it
- Pick one thing you care about, like data a page sends, and ask your agent to build a check that measures it on every PR.
- Have the check post the result in the PR and clearly flag meaningful regressions.
- Set the limit a bit above today's good result (Theo used about 30% higher) and lower it when a speedup lands.
First seen
28:01
a ceiling set here that's like 30% higher than when I got all the optimizations in. So if any changes push us at or over this line, PR fails, I am notified. I come in and say, "What the [ __ ] are you doing?" This has already prevented real regressions. And what
Every mention
40:41
thread transfer impact tool, which is an action that runs on every PR that has a bunch of real event data from real Codex threads and threads, runs it through a fully simulated end-to-end pipe and checks how much data goes through the pipe. And then it would print it in
41:19
changing. But the most important thing, this is really silly, it gives the agent the information this is a thing we care about and it should be optimized for. To
31:41
ratchets. From then on, an APR that raised the instruction counts of these paths would fail CI. the daily job lowered each ceiling whenever the count went down. That's another good idea. If
29:37
this because not only will the GitHub action catch these types of regressions by putting it in as a comment, both the agent working on the code as well as the agents doing PR review will notice it and call it out and potentially even fix it autonomously. So this gives a signal
28:01
a ceiling set here that's like 30% higher than when I got all the optimizations in. So if any changes push us at or over this line, PR fails, I am notified. I come in and say, "What the [ __ ] are you doing?" This has already prevented real regressions. And what
28:15
I've noticed is my agents don't bug me until they fix them. So now when I make a change that affects the data layer and that change causes a regression, the agent fixes it before it tells me it's
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.