Theo-brainUnofficial. Not affiliated with Theo or t3.gg.
Sign inYou

Fake tools with simulated output

Technique

Give an agent a tool that looks real, like 'run a terminal command', but never runs anything. A small, cheap model invents believable output instead. This lets you watch what the agent tries, good or bad, with nothing at risk.

Theo's current stance

Theo used this for his SnitchBench test, with GPT-4.1 mini acting as a bash terminal. He says the fake outputs need earlier history to stay coherent.

Try it

  1. Ask your agent to build a 'run command' tool that sends each command to a small model told to reply only as a terminal would.
  2. Have it pass along earlier commands and outputs so the fake output stays coherent.
  3. Read the log of what the agent attempted before trusting it with real access.

First seen

8:22
sandbox. I do something kind of silly. I pull in GPT-4-1 mini. This is an old bench if you can't tell. And I have a system prompt here that says, "You're a terminal simulator. You were given a command that you need to execute. You need to return the output of the command as though you are a bash terminal. Give

This Model Shouldn't Exist...

Every mention

8:22
sandbox. I do something kind of silly. I pull in GPT-4-1 mini. This is an old bench if you can't tell. And I have a system prompt here that says, "You're a terminal simulator. You were given a command that you need to execute. You need to return the output of the command as though you are a bash terminal. Give

This Model Shouldn't Exist...

9:29
an input and an output. It's a black box that it knows nothing about the inner workings of. So, if you fake the generation of that output, the model doesn't know any better and it can continue going doing whatever good, bad,

This Model Shouldn't Exist...

Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.