Please stop using stupid models
Thursday, Sep 17 · 27 min · about 9 min to read
Theo-brain watched all 27 min
Theo answers Cramer's view that cheaper models work as well as frontier ones. His case: the best models stand out because they fail less often, not because they are smarter. That matters most on long, end-to-end agent jobs. For narrow, well-written tickets, almost any model does fairly reliably.
Skim itSkim it: the notes carry the argument, so only dip into the video for the parts you want to hear in his own words.
Tap any time to jump the video there.
Key takeaways
Judge a model by its worst moments, not its best. Theo says frontier models stand out because they fail less often and their 'floor' (their worst behavior) is higher. He likes Fable because it is less dumb, not because it is smarter.
makes the frontier model special is that they hit those floors less and the floor is raised meaningfully. I don't like Fable cuz it's way smarter. I like Fable cuz it's less dumb. And those are different things. Less dumb and smart
What matters is how 'wide' a prompt is: not how hard the task is, but how many things it covers and how far it runs from start to finish. Wider prompts need models with higher floors.
and the most important detail we'll be talking about a lot, the width of my prompt. Not the depth, not the difficulty of the thing, but the amount of things and the distance from the start to the end. Wider prompting requires higher floors. And to be clear,
Small failure rates compound over long runs. A model that fails 5% of the time per 10 minutes fails about half the time over 2 hours. Cutting that to 3% turns a roughly 70% failure rate at 4 hours into about 50%.
minutes to 30 minutes changes your failure rate from 5% to 15%. Let's say you want to go for an hour. Oh god, now I'm at 73%. 2 hours, now you're at a 50% fail rate roughly. 4 hours and now you're at a 30%
That bumps us here to 97. Oh wow, that's kind of crazy. That's only a 50% fail rate at 4 hours, but it's only a 2% difference. Wasn't it Oh yeah, it was a 70% fail rate with a 5% every 10 minutes. And now it's only 50%, so that 2% change ends up being 20%
For narrow work, Theo concedes Cramer is right. A well-written ticket that names the files and the exact bad behavior can go to Claude and come back right fairly reliably, on almost any model.
again, I want to be clear here. If your goal is to go from Jira ticket to code, Zig is right. If I have a well-formatted Jira ticket that lists what files the code is in and what exact behavior exists that shouldn't, I can throw that at Claude and get an answer relatively reliably. That's not what I'm talking
His own logs show runs getting longer. His median prompt went from 53 seconds to 2 minutes 20 seconds. His longest 5% of prompts went from under 7 minutes to over 16.
Median prompt went from 53 seconds to 2 minutes and 20 seconds. So median more than doubled in length. P95 went from a bit under 7 minutes to over 16 minutes and 20 seconds. You understand, right?
Three skills here are new to you.
Screenshot-to-PR bug fixing New
- What it is
- Paste a screenshot of a user's bug to the agent and tell it to fix the bug, test the fix, record a video proving it works, open a PR (a proposed code change) and handle any review comments.
- Why Theo cares
- No step is harder than a ticket edit. The value is that the agent carries the whole job from a vague screenshot to a working PR without him stepping in.
- How to try it
- Screenshot a bug a customer reports and paste it straight to your agent.
- Ask it to fix it, test it, record a video showing it works, and link you the PR.
- Tell it to babysit the PR until every review issue is addressed.
had in T3 code. So, I screenshot it. I paste it to my agent in T3 code on my phone and say, "Fix this, test it, record a video showing it works now, and link me the PR when you're done. Babysit it until all the issues that come up in review are addressed." That isn't harder
Re-test old failures on every new model New
- What it is
- Keep a set of prompts that earlier models failed at. When a new frontier model comes out, run that set first instead of your usual prompts.
- Why Theo cares
- Failures quietly lower what you dare to ask for. Retesting shows him what the new model can now do that the old ones couldn't.
- How to try it
- Save prompts your agent failed on into a list.
- When a new model launches, run that list before anything else.
- Widen your normal prompts wherever the new model now succeeds.
why when a new frontier model comes out, I immediately try to reset. Instead of just sending the same prompts I sent before, I always start with a set that I've tried on other things that failed to see if it can succeed. Usually when I
Long runs to find codebase weak spots New
- What it is
- A framing Theo credits to Jaymon: run agents autonomously overnight, not mainly to get code written, but to hit the failure cases. Then fix the codebase so shorter, supervised runs go smoothly.
- Why Theo cares
- A snag agents hit once every two hours shows up two or three times in a five-hour run. Long runs expose these failures faster, so you can fix them.
- How to try it
- Give your agent a long, open-ended task overnight.
- Review where it got stuck or went wrong.
- Ask your agent to change the code so it won't hit that snag again.
way Jaymon framed this here. He likes doing overnight autonomous work not because it gets a ton of code written, but because it will hit a bunch of those fail cases so that he can architect his code base and his systems so that when he's working more in the loop doing like one or two hour threads instead of five
to 10 hour threads. If you have a speed bump in your code base that agents hit on average once every two hours, then they'll hit it on average two to three times every five hours. And if you do these super long runs, you can hit those failures faster and fix them. Learn from
Advice
Stop watching your agents while they work. Push how far you trust them and see where it goes. Theo liked one viewer's rule: only watch if the first output is badly wrong.
win. You guys need to stop paying so much attention to your agents. I've been saying this for a while now. You guys should push your limits in your trust and see where it goes. I like this. I only watch if the first output is horribly wrong. That is a great
When an agent goes wrong, ask it what led it there. Models rarely know exactly why they did something. But they will tell you which signals, tools and files sent them the wrong way.
can ask. You can say, I expected you to do this, but you did this other thing instead. What led you there? It won't get it properly. The models rarely know exactly why they did a thing, but they will usually indicate what signals they found and what tools they called and what files they read that led them in
Adjust your prompts a little but your codebase a lot. If agents keep failing in your code, a new hire would too, and that is on you, not the agent.
bad runs. You should adjust your prompts a little, but you should adjust your code base a lot. If agents are screwing up in your code base, then a new dev would, too. Like if you took a super
Stop telling the agent which files to touch. Describe the problem and let it find the files and change them itself.
horizontal. Maybe instead of investigating the code base and telling the agent which files to touch, you tell it what the problem is and tell it to find the files and change them itself.
Hot takes
Spending an engineer's whole salary, or even two salaries, on AI tokens is a no-brainer if it makes a strong engineer three times as productive.
His reasoning: Tokens are expensive, but so are engineers and their time. He would rather a rare engineer like Julius ship three times more than hire two people who might slow him down.
Has it burned more tokens? Yeah. I don't care though because despite the tokens being expensive, so are my engineers. So is our time. If Julius can be three times more productive by spending his salary in tokens, that's a no-brainer.
For Theo, downgrading to a cheaper model isn't worth it, because his time is worth more than the model's cost.
His reasoning: A cheaper model makes him put in more time before it starts and more time after it stops.
true. But my time is more valuable than Fable's. So I downgrade to a cheaper model, I have to put more time in before it fires, and more time in when it stops. And the more I let the model chew
Cramer and David K (creator of XState), people he respects, are still prompting like it's February.
His reasoning: They undervalue their time. Doing the prep before the code and the checking after it is what made them great engineers, so handing that to an agent feels uncomfortable.
I genuinely think both are still prompting like we're in February. And the reason why they're doing that is they're not valuing their time properly. They feel good putting that extra effort in at the start and the end because as a
Said in passing while redrawing his chart: no one should be using a Gemini model.
His reasoning: Gemini's floor is consistently low. It will read the same file 26 times before making a change.
I'll do a more realistic comparison here to get my point across cuz no one should be using a Gemini thing.
strengths. I would argue to an extent that my disdain towards Gemini comes from the floor being so low and so consistent where it just it'll read the same file 26 times before making a change cuz it's a [ __ ] model. But
How Theo thought about this
When a viewer doubted his 'exponential' claim, he didn't argue. He had an agent analyze his own prompt history and measure how long each run lasted, then and now. He then based the claim on that data.
you to figure out from a reasonably randomized set how long my average prompt ran for. So from when I sent the prompt to when the response stopped generating, how has the amount of time changed from January or February to now?
He explained the disagreement rather than just dismissing it. Critics test models on the narrow middle step they have already handed off, which models have handled for months. The gap only shows when the agent also covers the work before and after.
They're testing the models against the thing they already stopped doing, and the models have been able to do the thing they stopped doing for 6 months. They're correct there. But the moment you let the model go a little further in either direction, you'll suddenly start to see the edges a hell of a lot closer.
Sign in to mark what you've watched and keep notes.
Have a question about this one? Ask about this video →
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.
