Theo-brainUnofficial. Not affiliated with Theo or t3.gg.
Sign inYou
38:30
0:00

So I've been testing Gemini 4...

Wednesday, Oct 14 · 39 min · about 9 min to read

Theo-brain watched all 39 min

Theo tested Google's Gemini 4 Argon on real code work. It aced his skateboard-trick benchmark, but in a codebase audit 24 of its 36 claims were wrong, and its bug fix was a bloated 1,200-line change that failed the automated checks. Its real edge is YouTube access, which his businesses will keep using; if you use it for code, have a stronger model check its work.

Skim itThe notes cover the conclusions; watch the two stretches where he walks through his audit scores and the failed pull request if you want to see the evidence.

Tap any time to jump the video there.

Key takeaways

In Theo's test where models audit a real codebase, Gemini 4 Argon got 24 of its 36 claims wrong. The run would have cost only $3, but the majority of what it said about the code was false.

would have only cost $3, but on the less bright side, of its 36 claims, 24 were false. The majority of the things it claimed about the code base were wrong. Do you understand how egregious that is?

The $2/$10 per million tokens price looks cheap but isn't. The model uses lots of tokens, and it rarely gets cache hits (reusing input it was already sent, at a 95% discount): 70–80% for Gemini versus 99%+ for other models.

problem is that it sucks at hitting cash. I have had the worst rate of cash reads from any model I've used recently with this release. I'm talking like 70 to 80% cash hit versus other models where I see 99% plus. It means that that 95% discount on cash reads is not going

Asked to fix a database connection bug in his app T3 Code, Gemini took 30 minutes and wrote a 1,200-line change that should have been about 30 lines. The automated checks (CI) failed and it never fixed them.

Because this change should not have been 1,200 lines of code. This change should have been 30. At this point, Ithing. CI failed. It just didn't write the code right. The check and the lint both failed and it never bothered to fix it or take care of any of the high or

Gemini's real edge is Google's own data. Google gives it generous YouTube access, so it is one of the best ways to pull a video transcript, and Theo's businesses will keep using it for that.

Gemini models. If you want to get a transcript from a YouTube video, the Gemini API is one of the best ways to do it right now. And we've been using the

There is a hidden risk. Your Gemini account is your Google account. A ban could cost Theo his email, saved passwords and YouTube channel, whereas a banned Anthropic or OpenAI account is easy to replace.

banned. If my Gemini account gets banned, I might lose my YouTube channel cuz my Gemini account is my Google account. So, I could lose my email, I could lose my billing system for my credit cards, I could lose my saved passwords in Chrome, I could lose my

Two skills here are new to you.

Anthropic's official design skill

What it is
A skill (a packaged set of instructions an agent loads) from Anthropic that steers models toward better-looking web pages.
Why Theo cares
Without it, he got cluttered text, ugly fonts and the same awful top navigation bar on every design. With it, designs looked decent and unique. He says the separate 'taste' skill made things worse.
How to try it
  1. Ask your coding agent to turn on Anthropic's official design skill before generating any page design.
  2. Don't use the 'taste' skill alongside it.
  3. Compare how different models design the same page at whichai.dev.
I'm showing now are using the official Anthropic design skill. So, if you aren't getting good designs out of the model, definitely try turning that on and see if you have better results. If I

GPT-6.1 Soul as a code review model New

What it is
OpenAI's GPT-6.1 Soul, used to review code and find bugs rather than to write code.
Why Theo cares
On a real bug audit it missed some serious issues that Opus 5.5 caught. But it said nothing false, took 10 minutes and cost $1, against Opus's $26. That accuracy at that price is why he likes it for reviews.
How to try it
  1. Have your agent run code reviews with GPT-6.1 Soul.
  2. For high-stakes reviews, also run Opus 5.5, which found more of the severe issues.
Opus 5.5 found, but it didn't say anything incorrect. It took 10 minutes, and it cost a dollar. Insane. This is why I like 6.1 Soul so much as a review model. Muse Wonder scored quite a bit

Iterate on UX with a very fast model

What it is
For interactive parts of a product, such as buttons and flows, use a fast model and go back and forth many times instead of hoping for a perfect first try.
Why Theo cares
He built the Slop Politics site entirely with Astra on ultrafast. OpenAI models aren't great designers, but the speed let him ask for versions and refine without waiting.
How to try it
  1. Pick a fast model setting for UI work.
  2. Ask for several versions, give feedback and repeat until it feels right.
design, but since I was using ultrafast, I could tell it what I wanted and ask for different versions and go back and forth until I got it just how I wanted without having to wait constantly throughout. So, I think that speed is

Have a stronger model vet another model's work New

What it is
Hand a weaker model's pull request (a proposed code change) and its conversation to a stronger model, and ask it for an honest critique.
Why Theo cares
He pasted the Gemini thread into Opus 5.5. It explained where Gemini went wrong, including fixing the wrong half of its own diagnosis and ignoring existing fixes. Opus then wrote the real fix in 21 lines.
How to try it
  1. Give Opus 5.5 the pull request and the original agent thread.
  2. Ask it where the work went wrong and whether the PR should be closed.
  3. Ask it to make the fix itself if the PR is bad.
the PR and said, "This is opened by Gemini for Argon. I think the PR is garbage and doesn't really fix the problems that it barely stated. Go through the T3 code thread, figure out where it went wrong, give me your honest thoughts on the work that Gemini did. You agree it's slop, close the PR." I

Advice

Avoid the max reasoning levels. In his audit, Astra on Max scored worse than Astra on X High, and he says max settings do more damage than good.

improved. Also, funny, uh 6 Astra on Max scored worse than 6 Astra on X High. Once again, as I've been saying for a while, you should probably avoid the max reasoning levels. They do more damage than good. You might notice something's

If you use Gemini 4 Argon for real work anyway, run everything it produces through GPT-6.1 Soul or Opus. In one audit, 56 of its 78 findings were wrong.

Astra gave its findings a two. Of its 78 findings, 56 were wrong. That is a 70% failure [snorts] rate. This model is more likely to gaslight your agents than it is to give useful feedback in reviews. If you do decide to use this model for real work for some reason, for the love of God, please put the things it does through Soul or Opus or some model that doesn't have this egregious

If you have complex vision tasks, like labeling parts in a photo, test them on Gemini 4. Google models have historically led there, though Theo didn't benchmark it.

in this space. I did not benchmark any of that because I was just using it for code. But if you have complex vision tasks, it is definitely worth running them against Gemini 4 just to see if you have a real benefit. You might be

Hot takes

Anyone who finds Gemini 4 Argon comparable to Opus 5.5, Fable 5.1, Astra or GPT-6.1 Soul isn't working on real projects.

His reasoning: His audit numbers. He invites people to show him demos if they disagree, and says he'd like to be wrong.

I don't think anybody who finds this model on the same frontier level capability as things like Opus 5.5, Fable 5.1, 6 Astra, and 6.1 Soul, or even Muse Wonder. I don't think anyone who finds these models comparable is actually working on that real of stuff. Show me demos if you disagree. I would

Google's habit of launching at an introductory price and then doubling it is wrong. Argon goes to $4/$20 per million tokens on December 1st.

His reasoning: It makes pricing hard to plan around and discourages building on the model. Labs should launch at a price they can actually serve and lower it later, not raise it.

charge a reasonable price up until a certain date, and then they double the price after. So this model's actually going to be $4 per mil in and $20 per mil out starting December 1st. It'sbut I still hate this. Just give us a price that you can actually serve it at. Lower the price in the future. Don't raise it. The only lab that regularly

He wants to auto-close any pull request to T3 Code written by Gemini 4 Argon or the standard Muse Wonder, and says he isn't joking.

His reasoning: He has wasted more time and tokens vetting Gemini 4's code than he would have spent writing it with a smarter model in the first place.

would like to find some way to auto close all pull requests that are filed using models like Gemini 4 Argon or the normal Muse Wonder when they are filed to T3 code. I am not joking.this week. I have wasted more time in tokens trying to vet code that was written by Gemini 4 than I would have spent writing the code with a smarter model in the first place. Gemini is not

How Theo thought about this

He discounts headline evals by asking what was actually measured. Gemini scored a perfect 100% on his skateboard-trick benchmark, which he enjoys but says says little about real work. He likewise distrusts its low hallucination score because he has watched it be wrong a lot.

doing the thing. But again, when you say a model has 100% success in our evals, the eval and its relation to the work you're doing matters and I don't think a lot of people are making a living naming skateboard tricks. I wish they were. Iparticularly good score. Sadly, from my experience, this benchmark is not that realistic because I have seen this model just be wrong a lot. Here is where we

He tested Gemini on a problem whose fix he already knew, so he could grade it. It found the right diagnosis but built the wrong fix, and it missed existing pull requests that already solved the problem.

having some connectivity issues. I already knew a lot of the fixes. I was just curious if Gemini could find them.doesn't do that. In fact, this model will hallucinate its own incorrect solution instead. The correct answer was in the PR tab on our GitHub and not only did it not find that, it invented its own wrong answer. It also solved it with

Sign in to mark what you've watched and keep notes.

Have a question about this one? Ask about this video →

Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.