1:10:29Anthropic Made Claude Even Better
Monday, Oct 5 · 1 h 10 min · about 10 min to read
Theo-brain watched all 1 h 10 min
Theo reviews how Anthropic had Claude speed up claude.ai, and tests the site live: it feels faster, but caching and data persistence bugs remain. His main lesson: first have the agent find and validate ways to measure what matters, then ask it to improve. Prompt it to be bold, and test the result by hand yourself.
Skim itThe notes carry the lessons; watch the live bug hunt and the bold-prompting story if you want to see how Theo actually works.
Parts
Why a performance nerd cares
Theo has long mocked Anthropic's app quality and wants to see how Claude was set up to fix performance.
me for a bunch of reasons. Not only am I a performance nerd, I've also been dunking on anthropics performance and general engineering quality for a long time now. It seems like Anthropics models are now finally good enough that they're actually able to help with performance instead of hurting it. Which
Live test of claude.ai
Testing claude.ai live, Theo jokes they got faster by never updating data: the cache is far too aggressive and rarely refreshed.
questions. They made performance better by making it so that things never get updated. Clever. They're caching far too aggressively, and they're not invalidating the cache anywhere near often enough. For reference, in T3 code,
Setup and baselines
Anthropic first made 13 measurements directly comparable; Theo stresses it's much harder to fix what you can't measure.
server work. People underrate how important this part is to fixing performance issues. It is much harder to fix what you can't measure, even if you're not an AI agent. I cannot tell
The "wacky ideas" prompt
Theo says not to underestimate adding "I am open to wacky ideas" to a prompt, and agrees anything can be hill climbed.
awesome. So yeah, don't underestimate something as silly as I am open to wacky ideas in a prompt like this. And that's why the next quote here sounds insane, but I I'm on their side here. Anything can be hill climbed. They're not wrong.
Deterministic benchmarks and their trap
Counting React commits or calls lets agents cut work that doesn't matter, which may explain the sidebar bugs.
calling out before. If you're just blindly measuring the number of React commits or V8 calls, you're going to start turning things off that don't actually affect performance. The harsh
Ratchets in CI
Anthropic's checks fail any PR that raises instruction counts, and a daily job lowers the ceiling whenever counts drop.
ratchets. From then on, an APR that raised the instruction counts of these paths would fail CI. the daily job lowered each ceiling whenever the count went down. That's another good idea. If
The thread loop and testing yourself
Theo likes Anthropic's measure-ship-verify loop but insists engineers must also test changes by hand.
notice? If they're not testing it themselves, you're going to have these types of regressions. That's why I put
Scaling out with parallel agents
Once a flow works for a class of bug, you can spin up many threads across machines and parallelize it.
stuff. Once you have an end-to-end flow that is working for figuring out a class of issue or bug, you can spin up a bunch of threads on a bunch of instances on a bunch of machines and immediately parallelize it. We're all hopefully
Em dashes and syntax highlighting
Theo calls blocking the main thread for 0.35 seconds always unacceptable; T3 Code runs syntax highlighting in a worker instead.
highlighting it. Blocking main thread for 0.35 seconds is in my opinion always unacceptable. We moved our syntax highlighting off to WASOM because since it now runs in a worker, it won't block main thread at all, which is quite nice.
Steering: be bolder
Theo says people aren't prompting boldly enough, nor telling their agents to be bold.
cool. People aren't prompting boldly enough and they're not telling their agents to be bold enough either. And I
The 120 Hz frame budget
Anthropic's frame-budget thread landed nearly 60 PRs; long replies now block the main thread 200 ms instead of over 750.
one thread, we landed nearly 60 PRs. This is just the thread about the 8.33 millisecond render times. Long replies were blocking the main thread for 200 milliseconds in total where they used to block for over 750. Ran on about a third as much CPU and it held 120 FPS from start to finish on 120 Hz MacBook. The
Tap any time to jump the video there.
Key takeaways
Ask the agent to find ways to measure performance and validate them before asking it to improve anything. A bare "improve performance" gets guesses; measurement first gets much bigger gains.
If you tell the model improve performance, it will do its best based on its theories to improve the performance. But if you start by telling it to find ways to measure performance, validate those findings and then tell it to go improve the performance, it'll improve a lot. This is really cool and I
Anthropic's speedup is real: Theo felt claude.ai navigate noticeably faster. But in his live test it still has data persistence problems, like threads vanishing on refresh.
still not meeting my bar, I am impressed overall with the difference. It does feel a lot nicer to navigate. I hope they can figure out Oh, look there. My threads finally appeared. Took them a bit, but they did. But yeah, you get my point. It has a lot of like data persistence layer issues still, but it
Metrics an agent can climb are useful, but the agent must also understand how the real user flow works, or it optimizes the number and breaks the experience.
here. It is useful tooling, but you need to also make sure the agent is combining that with real introspection to how the user flow works. I don't necessarily
The upfront work of building tests pays off when you scale: once Theo had his testing, he fanned out many agents with different theories and his agent auto-merged about 40 speed PRs in T3 Code.
changes, but once I had built all of that testing, it was trivial to fan out lots of agents with lots of theories to try and solve lots of things. And the result is that my agent automerged like 40 PRs improving performance in T3 code.
Don't trust an AI model's time estimates. Theo says models misjudge how long work takes because they don't understand how capable models now are.
bit. But yeah, point I was trying to make by pulling this up is to highlight how bad models are at estimating time frames cuz they don't understand how capable models are, which I honestly
Three skills here are new to you.
Performance ratchet in CI
- What it is
- An automatic check on every pull request (a proposed code change) that fails if a measured cost goes up, and lowers the allowed limit whenever an improvement lands.
- Why Theo cares
- His own check against a stored baseline posts a comment on PRs, so both the coding agent and the review agents see a regression and often fix it before it reaches users.
- How to try it
- Ask your agent to add a GitHub action that measures how much data key screens load against a saved baseline.
- Have it comment on PRs when results move meaningfully above the baseline.
- When a speedup lands, have the limit lowered so the win is locked in.
ratchets. From then on, an APR that raised the instruction counts of these paths would fail CI. the daily job lowered each ceiling whenever the count went down. That's another good idea. If
this because not only will the GitHub action catch these types of regressions by putting it in as a comment, both the agent working on the code as well as the agents doing PR review will notice it and call it out and potentially even fix it autonomously. So this gives a signal
The "wacky ideas" prompt New
- What it is
- After a first round of fixes, ask the agent to dig deep and come back with several extreme proposals, explicitly allowing it to "boil the ocean".
- Why Theo cares
- This prompt led to a custom runtime for his Lakebed project, a 2 to 4x performance gain, after normal fixes had stalled.
- How to try it
- Ask for three different proposals, the more extreme the better.
- Add "I am open to wacky ideas" to the prompt.
- Pick one proposal and let it run long, with checks that behavior hasn't regressed.
improvements to be made. I want you to really dig deep and find everything we can, even extreme measures to improve performance. I want you to come to me with three different proposals for things we could do. The more extreme the better. Don't be afraid to boil the ocean. And this is why I now have a
custom runtime for LakeBed. I ended up forking Rusty V8 and building my own runtime in order to make the performance as good as possible. And it was like a 2 to 4x improvement. They proposed the
Theories, demos, then feel it yourself New
- What it is
- Theo's in-the-loop way of doing performance work: name one slow thing, have the agent list theories, build demos of each, then try them himself.
- Why Theo cares
- He says it is far too easy to cause odd UX regressions in performance overhauls, so he tests whether it actually feels faster.
- How to try it
- Describe one specific slowness, e.g. "it takes too long to start typing in the input box".
- Ask the agent for theories on what could improve it.
- Ask for demos of each idea and test them yourself.
myself. I'll usually start by asking it to come up with theories to improve performance for things that could be causing a specific thing to be slow.
Then I'll tell it to go make demos with all of those potential improvements and then I will test it myself to see if it feels faster or not. It is far too easy to cause weird niche UX regressions in performance overhauls. And I still do
Fetch data on hover New
- What it is
- Start loading a page's data when the user hovers over a link, before they click. Anthropic did this for conversations.
- Why Theo cares
- He calls it one of the most fun tricks to make things feel absurdly fast for a small change.
- How to try it
- Ask your agent where users click into lists or items in your app.
- Have it prefetch that item's data on hover.
data fetching on Hover is one of the most fun little tricks you can do to make things feel absurdly fast. It's actually crazy how fast things will feel when you make a small change like that.
Advice
Use AI-generated code to make testing your own changes nearly frictionless. Theo says if you put sloppy code anywhere, put it there, because friction makes you avoid testing.
frustrating. If you're going to put slop somewhere, it should be there. Slop up ways to test your changes yourself with as little friction as possible because that friction makes you hesitant to do all of this in the first place. I am
Tell agents they may keep going. Theo suggests adding a line to your CLAUDE.md (the agent's standing instructions file) like "assume you have my permission to keep going" and to keep at it until the stated outcome is reached.
behaviors too. Like if you add to the claw MD something like assume you have my permission to keep going. If I state what I want the outcome to be, he'll climb till you get there. It's much more likely to actually get there. They
Watch for agents making huge changes for tiny wins, like Anthropic's 900-line build plugin to save 2 milliseconds per message. Theo expects agents to start doing this on their own.
sends. This is why you have to be careful about how far you let these things go. Like I I wrote my own V8 based runtime for my cloud, but I had to like push for that. We're nearing the point where the agents will autonomously do [ __ ] like that. So make sure you look out for unnecessarily huge changes for unimportantly small wins. This
Hot takes
OpenAI's GPT models beat Anthropic's for deep performance debugging that grabs a problem and won't let go.
His reasoning: Much of it is tooling: Codex has better built-in computer use, so it controls Chrome and pulls performance traces out better.
the problem by the neck and strangles it. I have found anthropic models to not be quite as good as OpenAI models in the GPT line for these like deep dive tear apart all the performance problems type stuff. But a lot of this does just come
stuff. But a lot of this does just come down to the tooling because codeex has much better computer use stuff built in and it seems like it's able to figure out how to control Chrome and get these traces out of it a little bit better.
Reading the code doesn't solve these problems; you need to read and understand Claude's responses.
His reasoning: He spotted likely flaws in Anthropic's tests from the blog alone, and says anyone reading all the code during this kind of work won't get very far.
reading the code does not solve this problem at all. You got to read the responses from claude and understand what it is doing when it does it. If you're actually reading all the code when you do these types of tests, you're not going to get very far. Because I'll
Anthropic pushed to replace engineers with AI partly because it didn't like engineers, so it didn't hire the best ones.
His reasoning: He believes Anthropic's founders left OpenAI over research being held to engineering standards. Hiring for what you dislike leads to worse hires.
a lot of why Anthropic pushed so hard towards this idea of what if we replace all the engineers with AI is because they didn't like engineers, which means they probably weren't hiring the best ones. And I'm going to be very real here. It's some of the advice I give the most. Hire for what you love, not what you hate, because you'll make much better hiring decisions. Since they were
Word-by-word fade-in of streamed AI text is basically never worth it.
His reasoning: T3 Code shows text only when a paragraph, line or code block is done, keeping performance high without per-token rendering issues.
budget. No, wordbyword fade in is almost I I would say just straight up isn't ever worth it. I begrudgingly went from
only showing things when the whole response was done in T3 code to only showing things when the paragraph or new line is completed or the code block is done. And that allows us to have insane performance still without any of these weird potential issues of like frame
How Theo thought about this
Theo reasoned that Anthropic's test, which fails on any layout shift, would punish correctly adding a new thread from the network below cached ones, so the test itself may have caused the sidebar regression.
different things, that will be a layout shift. If you have four threads cached locally and you have a fifth thread that it loads from the network, it will shift the four threads down when that fifth thread comes in. Their tests here might have actually caused the regression because doing this right would fail this test. I I love trying to reverse
He explains why words like "ambitious" or "boil the ocean" work: models are word maps that default to safe, happy paths for "fix this", and those words pull them toward riskier options they're trained to avoid without proper tooling.
the safe happy paths to do that. But as soon as you start including words like ambitious or boil the ocean or push beyond what I requested or make weird and wacky suggestions, you're pulling the model out of the happy path and towards the sketchier, scarier ones that is correctly trained to go down less
Sign in to mark what you've watched and keep notes.
Have a question about this one? Ask about this video →
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.