Log every AI request and response
Workflow
Route all AI usage through your own proxy (a middleman that relays requests to providers) and store every request and response in a database. Theo uses ClickHouse, a database for huge data volumes, on his home storage server, holding a terabyte of history. It lets him audit real costs, what works and where failures happen.
Theo's current stance
Theo values auditing his own usage instead of trusting claims, and notes the logs could help train other models if he were cut off. He warns he does not guarantee his setup won't get accounts banned.
Try it
- Find the setup notes in the multi-account section of the T3 Code repo FAQ, noting Theo's warning about possible account bans.
- Give that link to your agent and ask it to copy the setup.
- Or ask your agent to record every model call, including model, tokens and cost, in a database.
- Review it monthly to see your real cost per task.
First seen
28:22
work with these models. I have built tons of benchmarks. I am evaluating this stuff obsessively. I have a terabyte of my own history with models in a clickhouse database that I set up on my network, so I can audit all of this. And
Every mention
11:46
fun things with this. I actually took the time to set up on my NAS a local ClickHouse DB, which is like a analytics DB for just flooding with data. And I have been taking every single request that goes into my proxy and every response that comes out and indexing all of it on my database. So now I have a
14:14
is really cool for like auditing my own usage, figuring out what's working, what isn't, like where are the biggest failures happening in my day-to-day work, but it's also useful in a future where we are cut off from these models and we want to train other models to have similar behaviors. I would
28:22
work with these models. I have built tons of benchmarks. I am evaluating this stuff obsessively. I have a terabyte of my own history with models in a clickhouse database that I set up on my network, so I can audit all of this. And
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.