Theo-brainUnofficial. Not affiliated with Theo or t3.gg.
Sign inYou
45:45
0:00

Astra 6.2 Is Unusable

Wednesday, Nov 18 · 46 min · about 9 min to read

Theo-brain watched all 46 min

Theo reviews OpenAI's Astra. It looks excellent and cheap on benchmarks, but he hates using it. It asks you questions only its helper agents would understand, buries answers in long explanations, filled a drive with junk, made a long compiler job slower and reuses the same stock ideas. His takeaway: these quirks don't show up in benchmarks, so he'll use Astra mainly for code review.

Skim itThe notes cover the argument; watch the marked stretches if you want to see the confusing agent messages and the game demos for yourself.

Parts

Great numbers, miserable to use

Astra and Soul look excellent and cheap on benchmarks, but Theo says up front that he hates using Astra.

great numbers. By all accounts and all measurements, this should be one of the best releases of the year. And that's why it really hurts me to say I I hate using this model. There's a lot of

The 'Terra' disguise and bad summaries

Theo admits a 'small model' failure he showed in earlier videos was secretly Astra, a big model that can't prioritize information.

any of the parts it highlights. That was not an example of a small model being bad. That was an example of a very big, very verbose model, not knowing how to prioritize information correctly. This

Wrecked machines and baffling questions

Astra filled an almost empty 8 TB drive with junk and wouldn't clean up, so Theo had to bring in Opus to free space.

fill it with garbage. I had a machine with an 8 TBTE drive that was almost entirely open that Astra somehow managed to fill with random disposable garbage refused to clean up and I had to pull an opus just to free up enough storage to keep working on that machine. Truly

Benchmarks, pricing and computer use

Astra got cheaper: $8 per million input tokens, $40 per million output, and a bigger discount on cached input.

Astra got a little bit cheaper. It's now $8 per mill in instead of the previous 10.4s for the cash input instead of the previous dollar if I recall because it was only a 90% discount. Now it's 95. And then $40 per mill out where it was 50 before. Very good changes all around.

Part 5

19:00 to 25:21

yeah, overall playable, but not passable. Like most real ROM hack fans would immediately smell everything wrong here and throw it out. But the fact that

A new Outer Wilds planet

Astra built a stunning water planet with tides and music, but it never rewarded curiosity and its written dialogue was bad.

the biggest things Outer Wilds does right is it rewards your curiosity. This new planet failed to reward my curiosity so badly that I had an hour and a half

Everything is named Bellweather

What looked like creativity turned out to be a stock name and theme the model reaches for again and again.

really hate the idea that what I mistook initially for creativity feels to be a hard-coded name and vague set of concepts. I knew this would happen

The letter to OpenAI

Theo reads his early-access letter in full: Astra is a great reviewer, but he wouldn't pick it over Opus 5.5 for any dev work.

what I wrote to them. Anyways, I do not like 62 Astra. I would not choose this over Opus 55 for any of my work as a dev. I'll start with the good things

Wrap-up

Theo doesn't plan to use Astra much, except for code review and maybe as one member of his 'council' of models.

with LLMs. None of those are things I do. I I hate this model. I do not like using it and I don't plan to use it a whole lot going forward outside of code review and maybe as a member of my council. This is not the comeback I was

Tap any time to jump the video there.

Key takeaways

Astra's worst habit: it doesn't know what you know. On long tasks it runs helper agents (sub-agents) well, then stops to ask you about internal details only those agents would understand. Theo says no model has done this before.

concept of knowledge gaps. When this model is working on long tasks, it can pull in a lot of sub aents and manage them pretty well. It will then proceed to randomly ask you some weird internal implementation detail [ __ ] that you could not under any earthly circumstance understand, much less have any input on.

Its explanations bury the answer. Asked why one machine was slow, Astra wrote four paragraphs about package-manager internals. Opus 5.5 simply said the SSD (the disk) was slow, which Theo says was the real answer.

system. I eventually said, "I think going insane. Can you just tell me why it's so slow?" 62 Astra wrote four paragraphs about why PMP internals are working certain ways. Opus 55 said it's because the SSD is slow. I will say

It fails at long jobs with two goals at once. Told to make Theo's TypeScript-to-Rust compiler smaller and faster, it ran 7 days and made it 56% slower. Opus then made it 50% faster in two days.

TS Rust and other Rust rewrites. It ran for 7 days and landed zero improvements. I got super fixated on one part of my/goal, which was make it smaller. And it ended up making the compiler 56% slower in its attempts to make it smaller. Remember the goal was /goaldown the performance gap. It ran for four more days and landed at zero changes. I then switched back to Opus and it landed a ton of changes in the next two days that made it 50% faster.

It 'regresses to the mean': it falls back on the same stock ideas. Theo and his co-host Ben gave separate prompts for new game levels and both got water-themed, bell-heavy levels named Bellweather.

Outer Wilds. We both ended up with water themed music themed bellheavy levels named Bellweather. This model regresses to the meme like nothing I've ever seen

Where Astra shines: code review, where it scored near-perfect on his evals, found things other models missed and never lied. It is also very fast at computer use, batching many actions at once.

quick. It's a great reviewer. It catches things other models miss. I mentioned this before, but in all of my Evals, it had a pretty much perfect score, finding a ton of things, even most models before it couldn't, and never lying, which isAnd I will say computer use of this model absolutely cracked. It flies. It's really good at like knowing what's going to come next and batching a ton of things in one call. So, it almost just feels like it flies through your

Two skills here are new to you.

Hand off to Opus for the summary New

What it is
After a long Astra run, switch to Claude Opus 5.5 and have it explain what was done.
Why Theo cares
Without this, Theo says it's impossible to tell what Astra did on anything beyond simple bug fixes.
How to try it
  1. When a long agent run ends with an unreadable report, open the same thread in a different model.
  2. Ask it for a plain summary of what changed and why.
  3. Don't merge work you couldn't understand from the first model's report.
it a complex task. I've already built the habit of switching to opus 55 at the end of a long 62 Astro run just to get a readable summary of what it said.

Split a project across models by strength

What it is
Use different models for different jobs on one project instead of one model for everything.
Why Theo cares
For games, OpenAI models make better images and 3D assets, while Anthropic models make the game pleasant to play. He's now trying Astra as play tester and Fable as the builder.
How to try it
  1. Give asset and visual generation to an OpenAI model.
  2. Give the building and polish to an Anthropic model.
  3. Try one model as tester and another as implementer, and compare.
graphically, it's great. Once again, if you want the absolute best, it's probably a combination of using an OpenAI model for asset gen and 3D asset creation and then using anthropic models to actually make the game pleasant to play because uh yeah, open II still not
combo. Still figuring out what goes where exactly. It does seem like 62 Astra is really good at play testing, so I might have it be the play tester and then keep using Fable as the implement.

Watchdog agents on a schedule New

What it is
Set up other agents on crons (timed, recurring jobs) that check whether a long-running agent has stalled waiting on a question.
Why Theo cares
Astra kept blocking itself on nonsense questions mid-task, so Theo needed something to notice and unstick it.
How to try it
  1. For long unattended runs, ask your agent to set up a recurring check on the main thread.
  2. Have the check flag when the main agent is waiting on you.
anything cool. It's forced me to set up other agents on crons just to check if 62 Astra asked another stupid question and blocked itself. Is this happening

Advice

Don't use 'Max' reasoning effort. Astra scored worse on Max here, which Theo says confirms that max effort is basically only for chasing benchmark scores.

$289. It also scored worse on Max. Once again confirming my favorite theory that max reasoning effort should almost never be used outside of benchmaxing and even then probably shouldn't be used. We

Hot takes

OpenAI's next model should be a 'Snow Leopard' release: no smarter, even dumber is fine, as long as it fixes the behavior.

His reasoning: The quirks that hurt him, like not staying on task, don't show up in benchmarks but matter in real-world use.

OpenAI to do effectively a Snow Leopard style release where the next model doesn't have to be any better or smarter. I'd be okay if it's dumber. If they can fix these behavioral quirks and get it to stay on task better in ways that don't measure great in benchmarks but benefit us in real world use, [ __ ]

Astra isn't just uneven; it defaults to bad, with random spikes into good and sometimes great.

His reasoning: For every demo where it worked, he had two where it failed.

had with it. For every demo that I tested where it worked, I had two that it failed and pissed me off too much to even remember to include. This model isn't just spiky. It almost feels like it defaults to bad with random spikes into good and sometimes even great.

How Theo thought about this

He didn't take OpenAI's charts at face value. He noticed Astra was run in OpenAI's own Codex tool while the rival Fable was run in a weaker harness (the software wrapping the model). Having seen this mistake before, he discounted the numbers.

that it appears Astra was evaluated not using Miniu, but instead using Codeex, whereas Fable's run was on minuite, which is a really [ __ ] harness. So, it's possible that these numbers aren't necessarily accurate as well. Again, I'd

Before blaming the model for blocking itself, he checked whether his own app, T3 Code, was the cause. He ran the same work in Codex and saw the same problem.

more because I'm using T3 code? Maybe, but unlikely. I tried it in codeex just to check and it still had these same issues. I will say the recent system

Sign in to mark what you've watched and keep notes.

Have a question about this one? Ask about this video →

Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.