Theo-brainUnofficial. Not affiliated with Theo or t3.gg.
Sign inYou

Turn agent failures into evals

Technique

Collect every time your coding agents failed, such as a broken tool call or a wrong path through your code. Turn each one into an eval, a repeatable test of whether the agent gets it right, so you can measure how well your codebase steers agents.

Theo's current stance

Theo calls this even cooler than training your own model. You won't use home-trained models for daily coding, but evals let you measure and improve how your codebase guides agents.

Try it

  1. Ask your agent to gather recent failed tool calls and moments it went down the wrong path in your codebase.
  2. Have it turn each failure into an eval you can rerun.
  3. Adjust your CLAUDE.md file or the code itself, then rerun the evals to see if agents now go the right way.

First seen

31:47
path when working in your code base. You can collect all of those and turn those into evals. And once you have the eval, you can start adjusting like your agent in Claude MD. You can edit some things in your code base to make it more likely the agent goes the right way. And then

I can't believe Pewdiepie made this

Every mention

31:47
path when working in your code base. You can collect all of those and turn those into evals. And once you have the eval, you can start adjusting like your agent in Claude MD. You can edit some things in your code base to make it more likely the agent goes the right way. And then

I can't believe Pewdiepie made this

Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.