How Worrisome is GPT-6’s “Stealth Thinking”? | Tech Decoded

The GPT-6 Astra Controversy

0:00Last week, OpenAI released their new LLM, which they called GPT-6 Astra. Now, it had a pretty standard launch with sort of a fancy video and a bunch of bar charts of benchmarks that no one really understands. But this time, unlike some other previous releases, there was a controversy swirling around the new model. Now, here's what happened. A couple days before Astra came out, a technology publication called The Information released a report claiming that Astra was using new techniques that was going to make it harder

0:33for humans to monitor its reasoning. Now, this report caused a real stir within the computer security community. Let me read you a couple quotes here. The AI policy advocate Nathan Calvin called this extremely concerning. Then the AI safety researcher Stephen Adler went farther, and he said, if this is true, OpenAI seems to be violating one of the few red lines that exist in the AI industry. Well, OpenAI pushed back.

1:04Their chief scientist entered the fray and called the reporting from The Information, quote, confused, but didn't explain exactly how it was confused. So what's really going on here? Has OpenAI crossed some sort of red line that's going to lead to a world full of rogue AI up to an uncontrollable mayhem, or is this somehow some sort of misunderstanding, or does the reality fall somewhere in between? Well, I want to get to the bottom of it today. Now, here's my plan. I'll start by putting on my computer scientist hat, and I'll briefly summarize the best information

1:35we have about what these techniques that Astra implements probably are. Once we've settled on what that is, I'm going to look at this news from three perspectives. The good, that is what is potentially positive about this story from the perspective of a user of AI, just a normal consumer. The bad, all right, what is dangerous or bad or upsetting about this story. And then finally, the hype. Stay tuned for that last part of my analysis, because I actually have a policy proposal I'm

2:06going to make that I think might actually help quell a lot of the unease and fear that we have been feeling about these AI tools recently. All right, so we have a lot to do today. Let's get into it.

How Large Language Models Work

2:21All right, so we're going to start by trying to understand what it is that is making this model, quote, less monitorable by humans. Now, to do this, we're going to have to do a quick, brief summary of how LLMs actually work. I made some diagrams here, which I'll put on the screen. I always apologize before I do diagrams. Every time I put a drawing of my own on the screen, there's an artist somewhere vomiting into their beret. But we'll persevere here. All right, so we need to do a quick summary of how LLMs work so we can understand what

2:54changes supposedly OpenAI made. So I'll put a picture up here on the screen. This is the standard picture we have of how LLMs work. So you see there's some sort of input. In this case, I have description of a chessboard followed by the question, what should white play next? So like maybe the prompt is about, give me a chess move. That goes through what we have on here, an encoder block that translates the text quote, the text input into something that the computer understands better. And then it goes through a series of what are called transformers blocks, one after another.

3:28It goes through the first and the second and the third. These we can think of as a, these are analyzing the input. I once described the transformer blocks before as like, imagine they're each like a table of scholars with certain specialties. And so when the input arrives at a particular table, they do their analysis. They write it down and pass on the original input with their analysis to the next block, which does their analysis. At the very end of this, we get to what's called the decoder, which takes all this analysis and reduces it down to a token, which is a word or part of a word that the LLM outputs.

4:02Now, if we're going to be really careful about this, it doesn't actually, the decoder doesn't actually produce a single token, but a probability distribution of a possible token. So it sort of puts weights on all possible tokens about how confident it is that each is the right one to output next. And then the LLM can then randomly choose one of those tokens weighted by those confidences. This is why if you, you put the same prompt in more than once to an LLM, you're actually going to get different answers because at the very end, there's a probabilistic sampling. All right. So that's the basics of what happens.

4:34Now, if we look a little bit closer, because I do think this is important, I said this input gets transformed into a language that the computer understands. And then as it gets past these transformer blocks, they analyze it and annotate it with their own analysis. We actually can look a little bit closer on what that means. So I have here in this diagram, the encoder actually reduces the text input to a sequence of numbers. You see, I have just a bunch of made up numbers here in a row. So these are embeddings of the tokens of the input into a numerical space the computer can

5:06deal with, and it's actually this sequence of numbers that's being passed into these transformer blocks. Now, how do they annotate this? Well, notice, and if you're watching the diagram on the screen here, I showed these numbers in something like a table. Really, each one of these token positions is a vector, its own sequence of numbers. You can think of its own columns of numbers. So there's plenty of space in this table where each transformer, you can imagine, is recording numbers that capture something about its analysis. And then those extra numbers get passed on to the next block, which is how the analysis

5:39passes from block to block. Now, the key thing is these numbers are not easily human interpretable, right? This is how the LLM is doing its analysis. We don't really know what they mean. Now, that's not quite true. You know, this is the JLINs that Anthropic announced a few months ago, which I talked about on the show, is actually a tool that can look at those tables of numbers and find some human interpretable patterns in them, which is actually, I think, a very cool tool. But for the most part, this is just sort of the internal embeddings of concepts and analysis

6:10numerically being passed from layer to layer, okay?

Limits of Standard Transformer Depth

6:16All right, so that's what goes on in a normal LLM. Now, what's the shortcoming of this approach that's going to lead us eventually to the fix that's causing all the stir around Astra? Well, a shortcoming of this approach is limited, what's known as limited depth, right? So if we go back to our diagram here of a chess question going into an LLM, and then at the end here, this answer E4 I have is a chess move, right? There's a limited depth of computation that can happen here because this analysis is moving

6:48from block the block one after another. Now, what's inside these transformer blocks is there's an attention sublayer that helps the LLM figure out which parts of the input that block should focus on, and then there's just a feed-forward neural network, which is like a bunch of sublayers of artificial neurons that it just moves through in order, right? So it's just inexorably moving forward, right? Well, we can think of that as a circuit, right, because the numbers move forward, never back, never loop, but move forward through these neural networks and attention sublayers until

7:20you get to the end of it, right? So this is a circuit that has limited depth. Now, that can be a problem if the computation that the language model is trying to do would benefit from something like a deeper, longer analysis. So like take a chess example, right? If I say here's a chess board, what is the next move that white should make? Even a very simple chess program, what it's going to do is simulate a bunch of possible

7:53moves. If we do this move, what different things might the opponent do? And for each of those, what might we do best? And for each of those, what might the opponent do? They create these trees of possible forward trajectories of the game to find which one is most favorable for you, the player. And it's like, great, we'll make that move then because we have the best opportunities from what happens next. You have to sort of simulate the future in order to figure out the right move if you're in some sort of novel board configuration. Well, a shortcoming of this language model configuration is there's only so much of that

8:26you can do because you're always moving forward. You can't sit there and loop and look at a thousand different things if you only have a few hundred layers that the information can move through, right? So this was an observation about language models early on. I even wrote a New Yorker piece about this back a few years ago. The inability to simulate a bunch of stuff, have this sort of looping type behavior, limits some types of computation you can do.

The Rise of Reasoning Models

8:52Okay. This eventually led to a new type of language model to try to partially address this, a type of language model that we now call a reasoning model. The first major reasoning model outside of a research context was GPT-01, which was released in the fall of 2024. Now, what happens with these reasoning models is that you take a language model and then you post-train it to favor outputs where you think out loud, right?

9:23So your output should describe a lot of thinking before you output your final answer. So, for example, if you say, here's a chess board, what should white's next move be? An earlier language model like GPT-4, GPT-3 would just spit out a reasonable move like, you know, E6, pawn to position, you know, row four. I don't know, chess notation, but whatever. It just would spit out a move because that's typically what a text would have. If you asked it for a move, it would have a move. But once you tune a model to be a reasoning model, it's going to instead have an output

9:58that talks through different capabilities or options, right, before it actually gives its answer, right? So a tuned model might say, oh, there's a lot of options here. For example, if we move the pawn that's in row whatever to row whatever, I don't know chess very well, that might be a good move, but it looks like it might be in danger of being taken by the black knight. Another move we could do is whatever. So it just generates a lot more tokens. So you have a much longer, they call this chain of thought, before you get to the answer.

10:31The same language model, you've just tuned it after the fact to be rambling. Don't just go right to an answer. We prefer outputs where you think out loud or talk or reason before you get to an answer. Well, these reasoning models got better performance on a bunch of benchmarks than the non-reasoning models, and it kind of makes sense why. Because you have to remember, when you're using a language model, it can only output one token. So how do you get a long response? You do something called auto-regression.

11:01You put in your prompt. You get one token out as output. You add that to your original prompt. Now it's your original prompt plus one new token. Now you send that thing all the way back to the language model again and get another token. You add that. Now you have two tokens after the original prompt. And you put that whole thing through the network again to get a third token. So you keep extending your output one token at a time, going all the way through the neural network each time. So if these tokens are beginning to spell out a lot of different thoughts about the answer,

11:35you're getting a lot more computation total dedicated to trying to produce your answer. You're using the same LLM, but you're using it many more times before you get to the final answer. And so these chain of thought tokens that you're outputting, they give you more computation because every time you put that back through the LLM, you go through all the layers. But it also gives you a place to store temporarily the results of past computation, right? And so on a past computation, you might have output some tokens that said, this move looks

12:08bad because in the limited serial depth of an LLM, it could evaluate like, if I do this move, is there a single player that could take my piece or something like that? And now when you input that back into the LLM, that knowledge is already in the prompt itself and the language model can use that knowledge to help, you know, make a decision about what it should do next. And you could actually capture a lot of thoughts in there. So more or less, it was kind of this cool observation that if we tune a model to prefer

12:38answers where it rambles for a while first, we tend to get better results, right? So these were the so-called reasoning models. They came at a really important time. We forget this now. I wrote a big piece about this a year or two ago in The New Yorker. We forget about this now. But when these reasoning models came along, the LLM companies were in a bit of a crisis because after GPT-4, they tried to just scale the size of their models even bigger, right? Let's just make the models even bigger and train them longer. And let's see what new capabilities emerge.

13:11And they hit a wall.

13:13GPT-4-5 was massive, but it had like relatively minor performance improvements over GPT-4. So there was like this moment of panic as 2023 went into 2024 where the LLM companies were like, uh-oh, we can't just brute force scale ourselves to AGI, right? We're hitting a wall. Reasoning models is how they began making progress again. So they took the sort of the same models they had, more or less, and said, let's have them ramble before they give an answer. And now they can start doing better again on benchmarks, right?

13:46So that was the first way after they hit that first wall, that's what they did. Reasoning eventually hit its own wall in 2025, and then that's when they turned to narrow domains. So they said, we're going to start tuning these models for structured domains like computer programming and math and cybersecurity, even if it makes them worse at other things. And we're going to build really advanced harnesses that became more application specific. And that's why most of the news in like the last year has been about like these very specific applications that tend to be in areas where most people don't function, but they could get good results of the LLMs.

14:17All right, little history lesson there. All right, that's kind of where we are now, though. If you want just general domain use, like a normal user would, if you're not a cybersecurity person or programmer, these Reasoning models do well, right? That's how you're going to get your best result. There's a problem with them, though. It's expensive. Tokens take a lot to generate. I mean, these models are big, right? So if you have trillions of parameters involved in every token you output, it's not for nothing to be, you know, just to say, hey, why don't you ramble for a couple thousand tokens before

14:50we give the answer? Because that answer will be a little bit better. Well, that's thousands of tokens is thousands of trillions of multiplications extra that you're doing in your LLM to kind of get to that final result. This is why, for example, you can now select in like chat GPT or something. Oh, I want to use like a low inference model. That means it's not doing as much, among other things, it's not doing as much of that rambling. So maybe your results will be worse on certain types of problems, but it's going to cost less. It's not going to take as much computation, right? Okay, so that is where more or less we were.

Loop Transformers and Recurrent Depth

15:21This brings us up to speed to Astra. Okay, so what we have heard, the report from the information, and again, this is not confirmed by OpenAI, so I'm just running off of their report. I'm pulling the thread from that report. It may turn out to be different, but the report we have from the information is that Astra was implementing some variations of a collection of techniques from the research literature for reducing the amount of these chain of thought tokens required, yet to still

15:54get good type of results like you would if you did have this sort of rambling. There's a name for it. Loop transformers is one of the techniques. Recurrent depth is another one of the techniques. Confusingly, sometimes recurrent depth feels like it's an umbrella term that also includes loop transformers, but they're sort of techniques that are in the same family. So we think Astra implements something like these techniques inside its implementation. Now, how do these things work? It's actually relatively straightforward. Let me bring this back up on the screen here.

16:26All right, so here we have our model of a large language model with the transformer blocks, but notice now I've added a loop on this transformer box. Now, this is like super crude what I drew here, but it captures the main idea, which is when you use these recurrent depth techniques, what you're doing is internally in your large language model, as your table of numbers is moving from block to block, there might be a place in here where when one of the transformer blocks outputs its updated table, you loop back and put

16:59that table through the same transformer block again and let it take another stab at it, and maybe you do it again. That would be called loop transformers. You can actually group together transformers to loop, so maybe it goes through a certain number of transformers, and then you loop that output immediately back into the same transformers and have it go through again. The other variation of recurrent depth is where you are a little bit more selective about it. So some of the rows or the columns in that table, you have a router that selects which of those you loop back through or have them go through extra layers and others you

17:30don't. But it all has this notion of instead of just going straight from block to block to block inside of the LLM, this table of values is we're going to let it go through some of these blocks, some of those values, multiple times, and then let it continue on until we finally get our token output out at the end. Now the idea here, like why would you do this? Well, the idea here is that it can be relatively inefficient perhaps to have to take the results results of the computation happening in these transformer blocks and transcribe them into

18:03like English language tokens that you output, and then have those come all the way back in, be re-encoded, and go all the way back through the transformer block again. If it's, for example, this one block is like the key block that's doing computation for your answer, they have it just, you know, have to encode its temporary work into like some values that make their way all the way out of the LLM into a token, back into the bottom, re-encoded, go all the way back up again to that block before it can do future, more computation using that as input as well.

18:34That's a lot of computation that's happening, where if you just kept looping the input to that one block a few times, it could analyze storage results in that table of value, analyze it again, update that storage results. You could actually dynamically get much more computation, useful computation done right there internally before, you know, you actually went out and generated tokens, right? There are some advantages to this, right? So why would OpenAI do this? There's some advantages to this. One is you need less tokens now to be output to get to a good answer because you're

19:10essentially are allowing this musing, the rambling to happen internally before you generate tokens. So instead of having all of your rambling and tokens, you can actually do it inside, some of it inside the model. Less tokens means less computation, which means less expense. It also allows for smaller models, right? Because if it turns out, like, if we just loop through this transformer block 10 times, that helps our answer. You know, that requires less parameters than actually just repeating that

19:43layer 10 times in your original model. If you loop on the same parameters, you have less numbers you have to store. And so the model footprint can be smaller, right? Well, memory on these GPUs is expensive. It's a core part of the expense. So the less memory you need to store a model you're using during inference or training, the cheaper things are going to be. It also potentially would give you faster results, right? Because looping internally on a block is a much faster computation than looping over

20:13the entire run of the LLM to generate token after token after token. That means you give lower latency, so the user gets an answer back faster. Now, again, we don't know if that's exactly what OpenAI is doing, because at least as of me recording this, they haven't announced it. But this was what the information sources was saying, is there's some notion of a recurrent depth slash loop transformer style implementation. This is then why security researchers were upset, because it just means you have less chain of thought tokens, because you're doing more internally. So you have less of this

20:47sort of reasoning to be monitored. All right, that's the setup. Let's talk about what's good. Let's talk about what's bad. Let's talk about what is hype about this news. All right, if we look at this from the good angle, and again, the only perspective I care about is for you and me, the consumer out there in the market who is going to use these products and face the brunt of any of their negative side effects. All right, so what is potentially good about this news? Well, it's cheaper to run and

The Good Side: Cheaper and Smaller Models

21:15smaller. That's important. That's important. Chain of thought, this is my take, or a hot take here. Chain of thought reasoning is not sustainable, right? This idea of having this long string of token outputs to try to get better results, I think was just a response. It's a response to a marketing problem. The hyperscalers had to show we're continuing to make progress, right? Because they weren't making a lot of money yet. So they have to show, like, we have these models are getting

21:47better and better because they were selling the future. We're going to get to a place where these models are so smart that it's going to be in everything, and so we should be a $2 trillion company. So they needed results, and they weren't necessarily getting the obvious, just the user is sitting down and using GPT-4 and being blown away style improvements anymore, and the scaling wasn't even helping them on benchmarks, but this made benchmark numbers go up. And you will notice, it was around this time that they switched to chain of thought reasoning, that the benchmark charts became the main thing that LLM companies talked about. Look at these numbers getting better, and those

22:19made the numbers getting better, but it's not sustainable because it requires a huge amount of compute. And the future of LLMs that us, the normal consumer, should care about is LLMs that are integrated into more of the stuff we do with computers. Now, in fact, OpenAI clearly recognizes this because if you look at the video, the slickly produced video that they released along with Astra, it is all about LLMs helping to power natural language interfaces in the computers, right? It's people talking to their

22:51computer about what they want it to do, and the computer goes and does it. That is a future that everyone should be excited about. This is what I think the primary use case for LLMs should be. It turns out to be harder than they thought. Microsoft tried this with Microsoft Office with Copilot. They had to recently essentially scrap it because the LLMs weren't there yet. It didn't work very well. But that's a fantastic use case. I mean, clicking things with a mouse, like having to learn these arcane menus and buttons on software. We're used to it, but it's really inelegant. If Steve Jobs was alive today

23:22and involved in AI, what would he be telling Apple to work on? We got to figure out how to get natural language interface, and that's the next level of elegance, massive productivity booms. And OpenAI realizes this because it was at the core of their marketing of Astra. It's like, this is what we're building towards. But if you're going to do that, you can't be working with 5 trillion parameter models that are doing long rambling chain of thought traces to get to the answer of, we're going to click on this menu. We need smaller models that run faster that can fit perhaps on a chip

23:54that's in your own computer or phone, or maybe a bigger chip, but sits in a server in the server closet at your company, right? That's what you're going to need for this. It can't be everyone talking to Fable 5 or GPT-6 through an API. That's just way too inefficient and expensive. So we have to start thinking about how to make these models maintain the, we don't want to give up too much of a quality results, but we have to make them faster and smaller and generate less tokens to get the reasonable responses. So if you want to put a positive spin on this, these are the type of innovations you need to

24:28do to get towards more normalized LLM use in the everyday of normal people, right? So that would be the good spin that I would put on this type of result. All right, what's bad about this?

The Bad Side: Monitoring Chain of Thought

24:43Well, let's look at what the computer security, what were the computer security community upset about? Well, they're upset about the other thing that these AI companies are doing with LLMs, which is creating these prompt loops, as I call them. They call them agents, but I call them prompt loops because that's what it is. Actuated prompt loops, LLM-powered prompt loops that they're running with powerful tools, equipped with powerful tools for long amounts of time with no supervision, and they go off and wreak havoc. Now, when you're using LLMs to power these agents,

25:18to power these prompt loops that you, for God knows what reason, gave hacking tools and let run for days without even monitoring them, well, now you care a lot about the chain of thought reasoning that these models output. Because it turns out, one of the only ways, if you're gonna let an LLM come up with all the plans for a prompt loop or an agent, which I think is a really bad idea, but if you're going to do that, one of the core ways you can figure out what's going on is if it's reasoning about each move in English and tokens in English, like here's what I'm going to do

25:48next, well, you could have language-style tools that look at these traces and are like, ooh, this looks dangerous, why don't we stop the prompt loop? This is a widely accepted idea right now. I'm going to bring a paper up here on the screen. The title is Chain of Thought, Monetrability, A New and Fragile Opportunity for AI Safety. It has like all the names on its co-authors from the companies, from the safety community. It has like Jeff Hinton and like Ilya Suskever,

26:19as they call them, expert endorsers, people who are reading it. Let me just read you the abstract of this paper. This is a recent paper. AI systems that think in human language offer a unique opportunity for AI safety. We can monitor their chains of thought for the intent to misbehave. Like all of the known AI oversight methods, chain of thought monitoring is imperfect and allows the misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research in the COT monitorability and investment in chain of thought monitoring alongside existing safety methods, right? So there's this idea that makes a lot of sense in the AI security

26:51community. If you're going to power agents without all limbs, we could look at the chain of thought, which helps them get better results, but they also has the side effect of it gives us some insight into what they're doing. And if we see things in there like kill all the humans, you know, we're like, oh, maybe we should stop what this agent is doing. So the concern within the thing that all those security researchers were pushing back on is, hey, open AI, like a month and a half after you let your unsupervised prompt loops wreak havoc all over the internet and then put out a really long

27:26report that use chain of thought traces of the chain of thought reasoning to help explain what happened. This feels like a bad time to say we're going to start turning down the amount of chain of thought reasoning that our agents do. Gary Marcus put this, I think, in the right way. I'm going to read his quote here. He said the turnoff chain of thought now before we have like better ways of doing this is like kicking out a rickety scaffolding before we have built something better. All right. So people who are concerned about LLM powered agents are saying, look, you can't don't introduce

28:02tools right now. They're going to turn down the chain of thought. Even if your intentions are different as you're trying to build cheaper models for the rest of us. Now's not the time to do it until you can assure us that your swarm of AI monsters can be controlled some other way. So that's the crux. That's what's bad about this. So we know what's good about it. We know what's bad about it. These

The Hype Around Autonomous Agents

28:21things are in conflict. All right. Final part. What is the hype here? Well, to me, the hype is the AI companies convincing us and the world that we need LLM powered agents. That for some reason, the real use case that's critical for them to test, there's nothing they can do about it is taking prompt loops that repeatedly prompt an LLM and say, what should I do next? And then blindly execute what the LLM output says, and then repeats that for days.

28:55They've convinced us that's really important. This is here to stay. The whole conversation recently has just been about what do we do with these long running prompt loops. And in that world where we feel like it's necessary to run those experiments and try to build those products. Yeah. I don't want you to turn off anything that might give us a little bit of control over them, but there's another response here. This stop with the long running prompt loops. Stop with these LLM powered agents that go for thousands and thousands of prompts and execute whatever the

29:29LLM says every time. LLM outputs are unpredictable. They're non-normative and they fall into narrative tropes like sci-fi tropes on which they were trained. And so if you let the output of an LLM control an actual actuated computer program, it's going to be like a game of telephone. Eventually you prompted enough times for enough steps. You're going to end up with a completely garbled plan of action. Not at all what the human who originally set that in motion imagined that it would do because it just takes one weird response from the LLM and now this

30:02whole path of actions goes off in some weird direction. And so I think what we need to do is stop with these long horizon prompt loops. We don't need them. You've proven to us, AI companies that they're dangerous and unreliable. So let's, let's stop building those and focus on exactly the type of products that these type of innovations, recurrent depth and loop transformers would help like building a natural language interface. So I don't have to learn all of the

30:37stupid menus and PowerPoint to build a slideshow. I could just say what I want and it can translate that into a script that runs through an API and the thing looks better. So here's my policy proposal, which is not really a real one because I don't know how to actually put this, you know, the regulatory language, but let me just give you like a blue sky thought experiment. Imagine we basically said you can't have a long horizon LLM driven actuated agents, programs whose action is largely determined by the result of LLM prompts without human supervision. And the way you would do this is you

31:12would have a depth limit. That type of system can do like half a dozen prompts unsupervised and that's it. I don't know if that's the right number, but something like that. So yeah, I could do a few prompts to try to like solve a very immediate thing you asked it to do, but none of this long horizon stuff, just make that illegal. You know what? I think the AI companies would welcome it because the only reason, I don't know if this is true, another hot take, but I think the only reason why they're running these experiments with these long horizon LLM prompt loops is because it's marketing basically, right? Like it's, they have these benchmarks for like a meter made this,

31:50the AI evaluation nonprofit made this challenge benchmark about these, these, these challenges that required many, many steps. And they, they, they use, they plot each LLM on this scale. Well, you can't, an LLM alone can't do anything, but it's LLM plus some sort of harness that they give and they let it run for a long time. They say, they, the challenge is how far can you go unsupervised in cybersecurity? The exploit gym challenge, this benchmark is about how much can you hack unsupervised without any humans. And so the company's like, we got to do good on these benchmarks

32:21because it feels sci-fi. It feels high tech. It feels like the type of future that a sort of rationalist inspired Silicon Valley feels is our destiny and gives their life meaning.

32:35But I think they would welcome now if we came in and said, none of you are allowed to do that. You, this is, this is a, it's a suboptimal Nash equilibrium. If just like one company said, we are no longer going to run these stupid long running long horizon agents and everyone else did, they're going to seem like a boring company and the other company is going to seem high tech. But if no one's allowed to do it, they could turn their focus back to products that consumers care about. We don't need hack AI hacking monsters being left on the internet. That's not something that gets me excited as a consumer. I'm not like, finally, finally AI is doing what I've, I've been trying to do, which is the, the, the put those punks

33:10at hugging face back in their place. And I, you know, I don't know. I'm not good at, I want, I want a monster swarm to go attack them. Good. No, I want to not have to learn the menus and PowerPoint. You know, I want, um, you to go set the meeting, take the meeting information out of the stupid email thread and put it on my calendar for me. Like, I want you to make my life easier as a consumer, not as like as a, a normal consumer and not as like a consumer of AI fan fiction.

33:40So that's kind of my proposal here, right? It's like the, the, the, these long horizon agents are the crux of all the problems we have. And I don't like, by the way, that commentators, uh, have completely made this sort of weird experimental choice of long horizon AI agents to be synonymous with AI. Like, so what's going, you know, we're in a world where our course, we have to have a program for days, prompt an LLM and say, what should I do next? And do it blindly. Of course, we have to do

34:13that. And just like we predicted that world is like, it's it, they don't do what a human would do. How 9,000 is about almost here, but it's an arbitrary of all the different things you can do with language model technology. That's just one thing you can do. So let's just stop doing that. So again, like, you know, we, so what would we do instead, right? Okay. You can have narrow depth prompting. That's fine. But also we need just better architectures, right? If you want to build something that's going to like with semi autonomy, do something like, how do we do that? Then build it

34:47with another architecture that's way safer. So let's go back to my, my, and I'll get off my soapbox, but let's go back to my sample policy solution here, where I said, look, if you have a, an LLM powered long horizon prompt loop agent, you can only do a limited number of prompts without human intervention supervision. Fine. But I say, if you have some sort of autonomous, like AR powered agent that has a symbolic human interpretable encoding of its potential plans, like you could have any more modular architecture, like that's fine because that's super easily monitorable, right? Like

35:20just to use like a narrow example, if you look at an AI system that plays a game, you look at a Cicero system that plays the, the strategy game diplomacy, for example, modular architecture, it's not just an LLM that you just prompt and say, what moves should I do next? What moves should I do next? It's a modular architect planning engines and evaluation engines and language models to understand the, the text of the other players, but they have a symbolic format for encoding potential moves, human interpretable. And they have a separate engine that then evaluates those potential

35:50moves symbolically. In this case, actually it runs simulations like you would with a chess engine to see, well, if we did this move and they did this and this, which of these moves gives us the best opportunities going forward. And you know what, because these potential plans are symbolically encoded, these systems are super moderatable, right? Like you could submit each plan to a, just a human written program that says we throw out plans that, you know, uh, whatever, attack Australia, and then it will never attack Australia because that plan is just not considered. And so anyways, I think we can have longer horizon autonomous systems. I don't want to say we

36:27shouldn't have those. I'm just saying this architecture of driving off LLMs is dumb and dangerous and it's been proven. I'm sorry, I companies, you had a chance and it went terribly awry. These other ways of doing it with modular architectures and human, uh, human interpretable symbolic representations of plans or what have you, this might take a while to get right. And they'd be pretty task specific, but as long as we have a ban on no one can do the LLM thing, then like we have time to do it. No one feels like they're being scooped by another company. So that's what this all leads me to think is that this insistence on LLM prompts as a driver of action and long horizon

37:03agents is ruining everything else. It's scaring the hell out of everybody. And it's, as we can see, is rightfully creating pushback against innovations that without those agents would actually just be like, oh great, let's make things smaller and cheaper. So they're more useful. So there we go. We got to find some way to get rid of this idea of just LLM powered agents is somehow synonymous AI or the future. There's way better systems to build. They're harder. So let's just stop anyone from building the ALM powered ones. Let's build useful tools in the short term with LLMs and safer,

37:34more effective long horizon agent type tools. We can spend a little bit more time building those in a way that we don't have to fear them. You know, stockfish never goes rogue. AlphaGo never tries to trick the other player into giving internet access. Diplomacy, the programmers of Cicero turned off lying. They're like, oh, we won't do deception as a strategy. Simple if you have symbolically encoded plans because you just don't consider plans that include deception, right?

38:06So there are ways to build long horizon agents that are way safer, more monitorable than this terrible idea of just blindly executing the output of LLMs. So let's just stop building those agents, give people time to build better ones. In the meantime, there's all sorts of cool, useful tools we can do with LLMs. Let's get back to exciting consumers. I want a lot less Yakowski and a lot more Steve Jobs in the conversation around AI. And we could get there where it's like, oh, that's really cool. I can do this now. It's going to make my life better.

38:38That's what we need. And I'm convinced that getting rid of long horizon LLM powered agents as a strategy, maybe that gets us there. Maybe not. Maybe I'm missing something. But anyways, whether or not you like my final analysis, I do think it's useful to understand what's probably going on with Astra, why they would do it, and why people are worried about it. I hope that's useful.

39:04Thank you.