Model bake-off on your own codebase
Workflow
Have several models each review the same codebase and suggest areas to improve, then use strong models as judges to score the suggestions. In Theo's test, Grok 4.7 came just behind Astra on viability, and Fable 5.1 made fewer suggestions that still scored well.
Theo's current stance
Theo calls it a Hail Mary test, but says the results were useful.
Try it
- Ask each coding agent you use to go through your codebase and identify areas to focus on for improvement.
- Have Fable 5.1 and Astra act as judges and score each model's suggestions.
- Compare the scores against what each run cost before choosing a default model.
First seen
22:14
that said, I decided to have a bunch of different models go through T3 Code's codebase and identify areas that we should be focusing on for improvement.
Every mention
22:14
that said, I decided to have a bunch of different models go through T3 Code's codebase and identify areas that we should be focusing on for improvement.
23:22
was scored, I set up a set of judges with Fable 5.1 and Astra. So, even though Fable was half the judging work, it actually scored itself below Grock.
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.