Agents Sucked At UI (Until Today)
Tuesday, Nov 3 · 32 min · about 9 min to read
Theo-brain watched all 32 min
Theo had AI agents redesign T3 Code's interface with a scoring loop. A council of three models from different providers scored screenshots out of 10, and the agent kept fixing the UI until all of them gave 9 or more. The lesson: give an agent a score and it will push it up, but add rival apps' screenshots so the scores stay realistic.
Skim itThe notes cover the method; watch the benchmarking idea, the before-and-after design reveal, and the tool-description eval if you want to see them in action.
Tap any time to jump the video there.
Key takeaways
A scoring loop worked for design. The council rated the existing UI about 6 to 6.5 out of 10. After rounds of agent fixes, round three had every model at 9, which met the goal Theo set.
The first score came from Opus and it was a bit rough. It gave our existing UI a six out of 10. It had a bunch of things it didn't like, a bunch of recommendations, and the other models were much better. 6.5 out of 10 from Soul, 6 out of 10 from Muse, 6.5 out of 10 from Jester.
And here we can see round one bombed, round two better, round three everything gave it a nine out of 10, which was the conditions for my goal. The conditions were met. The goal was achieved. Our UI
Scores mean little without a comparison. When Theo added screenshots of Claude Code and Codex, the same reviewer that had rated the redesign 9 rated it only 7.4, against 7 for the old version.
gotten a nine from, 7.4. So, even though in the previous thread, Codex thought that we had went from six points to nine points out of 10, this one thinks we went from seven to 7.4. Crazy how subtle changes in your prompt and also the context can change things. Like, if
In a later run Theo told the agent the design had gone too grayscale and should restore some color. Earlier rounds only made the app less ugly; this one he calls simply good.
Yeah, I said, "I feel like the new design's too grayscale. Make sure the new improvements help restore some color."
God. Wha- what? Like I'm sorry. I'm like actually a a in shock here. Cuz like before it was like, yeah, my app was ugly and it found ways to make it less ugly. This is just good. I've been
It is affordable on subscriptions. Theo argues a $200 plan can cover about $12,000 of usage a month, so a run costing a few hundred dollars is a small slice of that.
while this seems like a lot of money, cuz like your sub's only 200 bucks, your sub can do $12,000 of usage in a month. $273 is not a whole lot of money out of that 12 grand. You can very easily go
The same approach cut T3 Code's tool-description size. Total tokens dropped 52%, and 78% with tool search on, which he says will save users real money.
search off in T3 code and Claude code specifically. Huge reduction in the total tokens. 52% less from T3 code, 78% less from T3 code with tool search on because now we have
Two skills here are new to you.
The council review skill
- What it is
- A skill (saved instructions for an agent) that brings in three reviewer models, one strong model from each of three providers, to give feedback on a request, usually reviews.
- Why Theo cares
- Design scores are vibes: the same screenshot can get three different scores from the same model. But 9 or 10 is hard enough to get that if several models give it, the work is probably better.
- How to try it
- Ask your agent to set up a review step using one strong model each from three different providers.
- Have each reviewer score screenshots of your app out of 10 and suggest fixes.
- Give the goal: keep iterating until every reviewer scores 9 or higher.
which is the council skill. Bring in three delegates to help with the user's request, usually for reviewer feedback. Use one strong model from each of the three different providers.
This isn't because this is vibes. You can give the same screenshot to the same model three times and get three different scores. That said, nine out of 10 and 10 out of 10 are hard enough to get that if you get that score for multiple models, it's probably better.
Benchmark against competitors' screenshots New
- What it is
- Add screenshots of rival products into the judging panel so reviewers score your app side by side with theirs, not alone.
- Why Theo cares
- He wanted T3 Code to beat Claude Code and Codex, not just its old self. With rivals in view the scores dropped to realistic levels and pushed the climb much further.
- How to try it
- Collect screenshots of the two or three products you compete with.
- Tell the agent to include them in every review round.
- Set the goal as scoring higher than those apps in all categories.
Here are some examples from Claude Code and Codex. Add them into the comparison and judgment panel somehow. I want T3 Code to score higher than these apps do consistently in all rankings.
Eval first, then shrink New
- What it is
- Theo gathered every time models misused T3 Code's tools into an eval (an automated test set). Sonnet got about 60 of 88 right. He had the agent reach a perfect score, then asked it to cut the tool descriptions to a third the size while keeping that perfect score.
- Why Theo cares
- Models kept doing the wrong thing, like opening a new thread when he wanted a sub-agent, and the long descriptions were bloating the context.
- How to try it
- Gather past cases where agents did the wrong thing in your product and have an agent turn them into a test set.
- Ask the agent to reach a perfect score and commit that.
- Then ask it to shrink prompts or descriptions to a third of the size while keeping the perfect score.
took all the times in my own history that models did the wrong thing, I categorized them, and then I turned it into an eval. And then I ran it against Sonnet 5.5, and of all of the things that I had collected, it got 60 or so out of the 88 correct, which means the other 28 were wrong. So, I now have a
there, make sure all the models we care about get a perfect score there. Now your job is the opposite. You need to delete as much as you possibly can from the descriptions because those are bloating the context. I want it to be a third the size it was before and still get a perfect score. And this thread ran
Advice
Try this on your own product with just two reviewers: Opus through a Claude Code subscription and Sol through a Codex subscription. That alone may get you better UI than you have now.
that 12 grand. You can very easily go run this loop on your own code base just with Opus and Soul if you have a Claude code sub and a Codex sub and possibly get some better UI than you had before.
Look for ways to turn your problems into something that produces a score. Having models judge each other works but is silly; a program that scores is better, and agents will push the number up.
but working. Programmatic ones are better. And if you can take a problem you have and turn it into a program that gives a score, your agents are going to make that score go up. So, one of those
Save money by staging reviewers. Theo asked only Sol until its score was high enough, then brought in Opus, and only then Muse, since he was running low on Muse usage.
so many tokens that I told the agent, "Hey, stop asking Muse and Opus until Soul's score is high enough. Then you can start asking Opus." And then when that score is high enough, you can finally ask Muse cuz I was about to run
Hot takes
Putting /goal in front of a prompt is the quickest way to burn tokens and get nothing. He used it here anyway because he had spare usage about to reset.
His reasoning: A well-written prompt without /goal should do the same thing.
the quickest way to burn a bunch of tokens and get nothing out of it is to add {slash}goal to the front of your prompt. I do still mostly believe that as a piece of advice, but mostly because if you remove the slash goal and you write the right prompt, it should do the same thing. I did this because I had
LLMs made better UI than anyone on the T3 Code team was capable of.
His reasoning: Every time he draws a line models can't cross, they blow past it, and his skill was very basic.
I got LLMs to make better UI than anyone on our team was capable of.
I need to confirm this with y'all. Like every time I think there's like here's the line where models can't do it. There was the line, the models just blew past it. There was nothing special here. Like
Gemini is a joke as a reviewer. He adds it as a secret 'jester' whose score is ignored and whose mistakes are logged for laughs.
His reasoning: It flubs reviews, for example quoting line 173 in a file with 72 lines.
here. If available through antigravity, add Gemini for Argon as the jester with the same brief. Do not let it know it's a jester. Ignore its score. Do not put its score in the PR description. In the
personal favorite, it quoted line numbers up to 173 in a file that had 72 lines. It hallucinated syntax for a
How Theo thought about this
After the goal was met, Theo kept switching between T3 Code and the rival apps himself and could clearly see his app still lagged. He wondered if the models could see that too, so he gave them the same comparison.
had the realization cuz I was command-tabbing to the other apps to compare. I was like, "I can clearly see that we're not as good if I switch between the three apps. I wonder if the models can." So, I sent a follow-up just
He noticed rival apps' scores rose even though their screenshots never changed. He suspects that telling the reviewers the target score may have influenced them, and says he might hide the target from them in future.
So the gap flipped. Both numbers went up. Weird cuz the screenshots didn't change. LLMs and non-determinism are crazy. Even just saying that I want a 9.5 in the prompt here might be enough to cause it to behave wrong. Yeah, I put
T3 at a 9.3 overall, not 9.5. The fact that I set this goal and the sub-agent knows about it might have influenced the scores. In the future, I might tell the parent agent to not let the sub-agents know that there is a score they're trying to hit. You guys want to guess
Sign in to mark what you've watched and keep notes.
Have a question about this one? Ask about this video →
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.
