Stop Searching For the Perfect Model
Sunday, Nov 15 · 28 min · about 10 min to read
Theo-brain watched all 28 min
Theo says to stop hunting for the one right model per task. Pick a default model, then bring in models from different families to review hard changes until they all score them highly. His Council skill does this. It raised review scores a lot and caught real bugs, but it burns huge amounts of tokens and only makes sense if you stack flat-rate subscriptions.
Skim itThe notes cover the main ideas, so skim the video only if you want to hear Theo explain the Council setup and its costs in his own words.
Tap any time to jump the video there.
Key takeaways
Don't route models by task. Instead, bring every model's different training and way of exploring into review on risky changes. The variety is the value.
very different way. Rather than making sure you use the right model for the right task, I want to make sure all of the models and their unique weights, their unique trainings, their unique ways of exploring things are brought in on hairy changes. I built this skill
His real Council runs: Opus, the model that wrote the code, first rated it 7.4/10, and it averaged 8.9 after two or more rounds. Of 144 reviewer scores, 128 ended at 9 or higher.
was started at 7.4 out of 10 and after two or more scored rounds, it would end average 8.9 out of 10. That's a pretty sizable bump just from having the agent
as well. Across all of the 144 reviewer scores, 21 times it started at below a seven and only twice was it still below a seven at the end. But of all of the work I have done, 128 of the times reviewers came in like this. They ended at a nine or higher. So
Unlike code review bots that read a pull request (a proposed code change) after it is filed, Council reviewers work on the same computer. They can run and test the code and ask the main agent questions.
stuff. You know how powerful it is to give the agent the computer you wrote the code on to test it there? All three of these agents are able to run the code, to test the code, to ask questions about the code and get an answer from the parent agent. They can do things.
The real goal is that the agent only comes back when the code is ready to merge. Theo says about 80% of the PRs his threads open can now be merged right away.
roughly now at the point where when I open a PR that one of my threads made, there's an 80-ish percent chance I can merge it right then. And whether you
It is expensive. Theo hits his weekly limit on Muse's $200 plan, often within two to three days, just from Council feedback loops.
this. I have not been using Muse for my day-to-day work a whole lot, but I've been using it inside of the Council a lot. I am hitting my weekly limit on Muse every single week, often in like two to three days on the $200 plan just from doing these feedback loops. These
Skills in this one.
Council skill
- What it is
- A skill where the coding model calls reviewers from other model families to review a change. The work is not done until every reviewer scores it 9.5 out of 10 or higher.
- Why Theo cares
- He says he has been blown away by how useful it is to have different model families give feedback on changes in a scoped, reasonable, measurable way.
- How to try it
- Ask your agent to set up a review step that calls models from at least two other model families on each risky change.
- Have each reviewer score from 1 to 10, and require a minimum (Theo uses 9.5) before the agent comes back to you.
- Have the final scores listed at the bottom of the pull request so you can see who reviewed it.
other thinks it's a five. I have been blown away at how useful it is to have different model families giving feedback on changes in a scoped, reasonable,
then merge. Council 9.5 is a verification clause. This is you should not bug me because your work is not done until you run through council and all of the models reviewing give you a 9.5 out of 10 or higher. I do specify here that
Sol loop: one model writes, another reviews
- What it is
- Theo's earlier, simpler version of Council. Opus makes the changes, then GPT-6.1 Sol reviews them. They go back and forth until both agree the change is good.
- Why Theo cares
- He found Sol incredibly thorough at reviews: it would tear every little thing to pieces. While he was building T3 Code's guest-share feature, the Sol loop caught three real security holes.
- How to try it
- Have your main agent write the change, then hand it to a different, more nitpicky model to review.
- Tell them to keep going back and forth until both agree the change is good.
very minimal version of this where I found that GPT-61 Sol was incredibly thorough at reviews. It would tear every little thing to pieces. So I'd have Opis make changes and call a delegate with Soul to review the changes and give feedback and then they go back and forth until both agree the changes are good.
other agents reviewing the implementation. Soul loop was super helpful when I was building this and it caught three real security holes. One was that a guest could still load HTML
Custom scoring bar for reviewers
- What it is
- Instead of asking reviewers how good a change is, ask them to score something you care about. One example: how likely Theo is to be okay with it. Another: score it several ways at once.
- Why Theo cares
- It is a lossy but powerful way to steer lots of agent work toward your taste. For a T3 Code user, Corey Quinn, he told Council its job was to make sure the PR was in a state Corey would be happy with.
- How to try it
- Write down how you like things built, so reviewers can learn your taste.
- Ask reviewers for separate scores, e.g. how severe the problem is, the risk of future problems, and how likely it is to annoy you.
it to give two scores or three scores. I had a round where I had to review every PR in three ways. How severe is the problem it fixes? How likely is this to cause problems in the future? And how likely is this to piss Theo off? And the
what he likes. And third and most importantly, I want you to bring Council back in, but this time Council's job isn't to make sure the changes are good. Council's job is to make sure the PR is in a state where Corey would be happy.
Advice
Stick with one default model; Theo uses Opus 5.5. He switches mainly to use up tokens he has left on another subscription, not to match models to tasks.
guy, and I still am that guy. It is very rare that I swap off of my default model once I've picked what I'm using currently. The main reason I switch between models is because I have tokens to burn from a different subscription. Other than that, I'm usually just sticking with at this point as of filming Opus 5.5. So if the lesson you
Don't run Council if you pay per token through the API (pay-as-you-go access). It only makes sense if you stack flat-rate subscriptions.
you should not use this in its current state if you're paying API prices for your tokens. If $10 of Opus costs you 10 bucks, don't do this. If you are stacking subscriptions, however, that's when this gets powerful. As an
Cap the number of review rounds and let the agent give up if it is going in circles. Tell it plainly that its job is to keep the scope small.
need to put a cap at the end here of only go a certain number of runs and or a tap out if you feel like we're not going anywhere and we're running in circles. Do not let repeated reviews result in scope creep. It's your job to fight to keep the scope limited. I added
Explicitly give agents permission to delete code or scrap the PR, because they are oddly scared to delete things.
don't care. This is important because agents need the permission to do things that they're scared of and deleting code for whatever reason agents are very scared to do. Second, I want you to
Hot takes
Getting several models from the same company to review is like asking the same person at different times of day: the mood changes, the ability doesn't.
His reasoning: He believes Anthropic's current models, Opus, Sonnet and Haiku, were all trained on data from one larger in-house model, so they are very similar.
the 55 models from Anthropic are distilled from the same base model that is likely Fable 55 or whatever equivalent they have internally. They used that model to generate data that they trained Opus, Sonnet, and Haiku with, which means all of these models
are very similar. What I have noticed over time is that getting different models from the same family to review things kind of just feels like asking the same person at different times of the day to review something. Like if you
Opus's reasoning traces (the visible 'thinking' text) are garbage, so don't watch them.
His reasoning: He watched it reason through five wrong things, then do exactly the right thing, after he had already sent a correction. He guesses the traces are hidden on purpose or summarized by a dumber model.
noticed this. The reasoning traces for Opus are garbage. I don't know if they're intentionally obfuscating them to make the model harder to distill against or if they've a dumber model summarizing or what. The reasoning traces for Opus are [ __ ] garbage. And
OpenAI's models, especially Sol, won't stop on their own when a review loop stalls. They keep fixing tiny nits until the PR is too big to merge.
His reasoning: He found Opus honors a 'stop if we're not making progress' instruction and OpenAI models don't. So he lowered Sol's reasoning effort from extra-high to high, because at full effort it finds too much.
noticed that OpenAI models do not do this. And Soul in particular will just keep [ __ ] going trying to squash every single tiny little nit until it's made a god PR that's just way too big to even be considered to merge. There is
now. I'm going to knock down from X high to high because if it goes too hard, it will find too much [ __ ] and it won't go very far. I am going to actually make
How Theo thought about this
He built Council to test one thing: was Muse's feedback as good as Sol's and Opus's? Instead of a single winner, he found all three regularly made valuable new catches the others might have missed.
on hairy changes. I built this skill originally because I wanted to validate how good Muse's feedback was compared to Soul and Opus, and I wanted a single skill to have all three come in and review. And what I found was that all three regularly brought up valuable novel catches and found things others might have missed. And then when I
He weighs cost against the time saved. A runaway thread spun up 600+ sub-agents and grew a 1,000-line PR to over 20,000 lines. Still, he says even a 5% better chance a PR is mergeable is worth the tokens to him.
had one thread that had spun up like 600 plus sub agents in order to go through at least 20 rounds of review like this, and it just wasn't making progress. It took what was supposed to be like a 1,000 line of code PR and it bumped it up to like 20,000 plus trying to address every single edge case one of the agents
that is why I like Council. Is even if this is a 5% increase in the likelihood that the PRI open is ready to merge, that's worth the token burn for me. I'm
Sign in to mark what you've watched and keep notes.
Have a question about this one? Ask about this video →
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.
