Eval first, then shrink
Technique
First build an eval (an automated test set) from real cases where agents misused your product. Get to a perfect score, then shrink prompts or tool descriptions while keeping that score. Theo cut T3 Code's tool descriptions to a third of their size this way.
Try it
- Gather past cases where agents did the wrong thing in your product and have an agent turn them into a test set.
- Ask the agent to reach a perfect score and commit that.
- Then ask it to shrink prompts or descriptions to a third of the size while keeping the perfect score.
First seen
27:24
took all the times in my own history that models did the wrong thing, I categorized them, and then I turned it into an eval. And then I ran it against Sonnet 5.5, and of all of the things that I had collected, it got 60 or so out of the 88 correct, which means the other 28 were wrong. So, I now have a
Every mention
27:24
took all the times in my own history that models did the wrong thing, I categorized them, and then I turned it into an eval. And then I ran it against Sonnet 5.5, and of all of the things that I had collected, it got 60 or so out of the 88 correct, which means the other 28 were wrong. So, I now have a
29:19
there, make sure all the models we care about get a perfect score there. Now your job is the opposite. You need to delete as much as you possibly can from the descriptions because those are bloating the context. I want it to be a third the size it was before and still get a perfect score. And this thread ran
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.