Theo-brainUnofficial. Not affiliated with Theo or t3.gg.
Sign inYou

Eval first, then shrink

Technique

First build an eval (an automated test set) from real cases where agents misused your product. Get to a perfect score, then shrink prompts or tool descriptions while keeping that score. Theo cut T3 Code's tool descriptions to a third of their size this way.

Try it

  1. Gather past cases where agents did the wrong thing in your product and have an agent turn them into a test set.
  2. Ask the agent to reach a perfect score and commit that.
  3. Then ask it to shrink prompts or descriptions to a third of the size while keeping the perfect score.

First seen

27:24
took all the times in my own history that models did the wrong thing, I categorized them, and then I turned it into an eval. And then I ran it against Sonnet 5.5, and of all of the things that I had collected, it got 60 or so out of the 88 correct, which means the other 28 were wrong. So, I now have a

Agents Sucked At UI (Until Today)

Every mention

27:24
took all the times in my own history that models did the wrong thing, I categorized them, and then I turned it into an eval. And then I ran it against Sonnet 5.5, and of all of the things that I had collected, it got 60 or so out of the 88 correct, which means the other 28 were wrong. So, I now have a

Agents Sucked At UI (Until Today)

29:19
there, make sure all the models we care about get a perfect score there. Now your job is the opposite. You need to delete as much as you possibly can from the descriptions because those are bloating the context. I want it to be a third the size it was before and still get a perfect score. And this thread ran

Agents Sucked At UI (Until Today)

Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.