Has AI “Gone Rogue”? Let’s Look Closer… | Tech Decoded

The summer of rogue AI stories

0:00Earlier this summer, I published an episode in which I discussed the OpenAI hacking attack on HuggingFace. I explained the basics of how that attack occurred, and I shared some concerns I had about OpenAI's practices. Now, I thought that would be the end of this story, but I was wrong. In the weeks that have passed since that original attack, more news about AI, quote-unquote, going rogue, has continued to emerge.

0:32So soon after the HuggingFace attack was first announced, we then got Anthropic revealing that one of its own hacking systems had, quote, gained unauthorized access to the real systems of three different organizations, end quote. Then Meta followed, perhaps not wanting to be left out, announcing that one of its systems had, quote, exploited a security vulnerability in a third-party service, end quote, to gain unauthorized access to servers. This was then followed by an OpenAI employee admitting that even before the July attack on HuggingFace,

1:03they had noticed many prior disturbing incidents where they would give their hacking system a challenge, and it would instead try to break out of its containment, right? So this idea that we are losing control of AI has become only increasingly prevalent as the summer continued, which raises the question, is this narrative correct? Well, it's getting so much attention right now that I feel like I have to revisit it again with more detail and more emphasis,

1:34and that's exactly what I'm going to do. In particular, the argument I'm about to make to you is that the current way we are talking this summer about rogue AI is both grossly inaccurate and completely serves the interest of the major AI labs, allowing them to seem more sophisticated than they actually are, and allowing them to avoid well-deserved scrutiny for their actions. So if you've been freaked out by these rogue AI stories, or if you have a sneaking suspicion that something is not quite adding up about these tales,

2:05then you need to stay tuned. As always, I'm Cal Newport, and this is Deep Questions.

Superhuman AI without loss of control

2:19All right, I want to proceed here with a series of observations. I want to start with a very important but often overlooked reality about the current state of AI. There exist many super impressive AI systems that can do things at a superhuman level. That is, they're more capable than humans on complicated key activities. There's many systems that can do this right now that have generated zero concerns about them going rogue

2:51and have demonstrated no signs of being hard to control or acting in any way on their own volition. Let's remind ourselves what some of these other systems are. Tesla's self-driving technology, for example, is an extraordinary feat of AI-powered perception, world modeling, and decision-making, and yet no one worries that their Tesla will spontaneously decide to start ignoring traffic laws and obey laws that it invented himself. Similarly, DeepMind's AlphaFold system earned its creators a Nobel Prize for its remarkable ability to predict the folding behavior of proteins

3:23is so far off the chart from what a human can do. It's not even in the same cognitive ballpark, and yet there is no concern that AlphaFold will start autonomously thinking about other biology problems that it thinks are important. Even more relevant and less known is Meta-AI's Cicero system, which can play the negotiation-centric strategy game diplomacy, as well as the most advanced human players, and yet the Cicero system has never tried to convince an opponent to give it unauthorized access to the internet

3:55so that it could expand its dominion into the real world. It just plays the game really well, and we trust it to continue to do that. Now, I make these points as just a starting observation because here is what it establishes. The idea that, in general, as we make AI systems more capable, we will inevitably lose control over their actions is not true. Most of the AI systems that we are making incredibly capable generate zero concerns of going rogue.

How autonomous hacking agents operate

4:28All right, well, this brings me to my second observation. If we look closer, and if we're being precise, basically all of the concerns we had this summer about losing control of AI all seem to be involving one specific type of AI system, what is sometimes called a long-horizon autonomous hacking agent. Now, how do these systems work? Well, I'm going to give you a very high-level summary. The architecture that these long-horizon hacking agents that are causing troubles all follow is a standard agent architecture

4:59that's built around the following loop, ask, act, report. So let me detail how an ask, act, report agent loop actually works. You start with a control program. These are often called a harness. Sometimes we'll call them an orchestrator. There's different terms for it. But this is the main computer program you actually run. This is coded in normal computing language that's readable by humans. It's not a neural net or something like that. Okay. The harness will start in the hacking context by generating a prompt to submit to an LLM.

5:29At first, that prompt will probably describe whatever the hacking challenge is that it's been given. It'll say, okay, here's the hacking challenge. There's this test server. It has the security on it. We're trying to break in and retrieve this particular protected file. So it'll explain the challenge. And then the prompt will say, what should I do first? Now, again, I'm aligning a lot of details. Often it'll ask for a plan and then recurse in smaller. But essentially it's like, okay, so what should the first action be? And it submits that prompt via an API to an LLM.

6:00Now, typically these LLMs in these hacking contexts are special in two ways. One, they've been extra trained with many, many examples of hacking attacks where vulnerabilities are described, hacks are described. Here's exactly how this hack worked and here's what happened. Two, these are LLMs where guardrails have been turned off. So typically if a standard LLM that most people could access out there, like a public facing LLM, they do a bunch of guardrail tuning where if you ask it certain types of questions, it won't answer. Most production LLMs are tuned to try not to answer questions about how do I hack into systems

6:31because that's illegal. But for these hacking challenges, they've turned off those protections. Okay, so the harness submits a prompt just like you would in a chatbot to an LLM. Here's the challenge. What should my first step be? The LLM, as all LMs do, generates an output. This is what the first thing I would do in this case. Or here's my whole plan and then you would zoom in on the first step. Okay, so now the harness has this response. Text, just like a chatbot, gets the text back. Oh, okay. I am now going to try to execute that step that the LLM just described. So the harness can execute things. An LLM can't. It can just output tokens,

7:02but a harness can't. So the harness has access in these hacking setups with all sorts of computer tools and utilities that are useful for running computer hacks. So it parses the LLM's response and says, let me actually try to execute this first step. It then records what happens and now it's going to update its prompt and say, okay, add to the end of it. I just tried to execute that first step. Here's what happened. And let me describe it. It was successful. It wasn't. Here's what I heard back. What should I do next?

7:33Now, it's key that I'm talking about appending here because remember an LLM, unlike a human brain, has no memory. It has no voluble state. It doesn't change as it executes. It doesn't remember anything. So you have to explain from scratch in every prompt what the challenge is, the steps that have happened so far, et cetera, et cetera, right? So you add, here's what happened. And again, nerds, I'll get into how I'm simplifying things later, but this is basically what happens. Okay. This is what we call a ask-act-report loop because the harness asks the LLM for a step.

8:04The harness takes action. The harness reports back on what happened. And then you loop. Then you ask for a next step. Then you ask for a next step, right? So you have an LLM that you're just repeatedly prompting. Okay, here's what happened. What should I do next? Here's what happened. What should I do next? And the harness is set up to actually implement the things that the LLM describes, which are basically terminal-based computer commands, right? And that's how these systems generally type the work. Now, again, nerds, I know, and when I say nerds, I mean it affectionately because I'm the king nerd. So this is, I'm among friends here. These harnesses have gotten pretty sophisticated.

8:36They'll do things like break down a problem into multiple sub-problems, which they might then spawn multiple copies of the harness and each of them is doing its own query to the LLM and executing it and then they'll report back. And this is how you get the agent swarms or whatever. But that's all just in the details of how you implement this basic architecture. Ask an LLM what to do. Do it. Report back what happened. All right? So the key observation here is that the sole driver of planning, the sole driver of what actions these systems take are the outputs of an LLM.

9:08Hey, I need to take a real quick break here to tell you about the presenting sponsor that made this AI reality check episode possible. They're called Done Daily. They're an online service that connects you with a real coach that helps you build a custom productivity system designed to fit your life. The coach will help you actually get important stuff done. Look, this is not some AI agent or over-featured productivity tool. It's a real person working with you to cut through distractions,

9:39face your productivity dragons, and lock in habits that actually get results. So if you want to find depth in our increasingly distracted world, you need to check this service out. You can find out more at donedaily.com. That's done, D-A-I-L-Y.com. All right, let's get back to our episode. Okay, so this brings us

The gap between plausibility and norms

10:00to the third observation. This is a bad idea. This type of system, so building an ask-act-report-loop system where LLM output drives all the actions, then giving the harness powerful tools, and then letting it run for days with no one checking in and saying, what is it doing, is an incredibly irresponsible thing to do. Now, why is this? It's because

10:30if LLM outputs are the sole thing driving actions, you're in trouble because LLM outputs aren't great for executing complicated long-term plans. Now, the reason is is because if we, you know, we know this already, but let's just unpack this box that we've all heard about before, okay? How does the core pre-training work for an LLM? Well, ultimately, what you're doing with an LLM, the sort of the core training in which it gets all of its smarts, the core training is you're taking real text that exists in the real world that was written by real people.

11:00You are knocking out a word from the text or a part of a word from the text, and you're asking the LLM to guess what that word is. So it's trying to guess missing words that actually exist in real text. It does this one token at a time, but if you run this autoregressively long enough, you get a long response. At a high level, the right way to think about an LLM response to a prompt is that it is trying to create a textual artifact that is what we could call lexiographically plausible, a plausible response. If every word you're trying to guess what real word came here

11:31from some extant text, ultimately what you end up with is a machine that produces plausible text. This is plausibly, the thing I produce is plausibly a document that might actually exist that I might actually have seen during my training. Now, this is very impressive. I mean, to do this can be really hard, especially if to generate a plausible text, sometimes you have to do some bounded finite reasoning, some like mathematical connection, some sort of logic. Like, I mean, this is the big observation that we had after GPT-4.

12:01Creating plausible text can involve many impressive, and I keep saying finites are fixed-depth computations because all this has to happen in a feed-forward neural network architecture. But still, it's incredibly impressive. But what we're getting out of this is plausible text. This text could plausibly exist. All right, so why is this a problem? Well, it depends on the context. But one of the things that's true about plausible text is that it's different than normative text. Normative meaning something that applies

12:31to a set of implicit or understood rules or standards. When humans produce text, we have all of these norms that we have imbibed and internalized, right? And when we produce text, it's not just plausible text. Like, the grammar is correct. It's like it's internally consistent. We're also going to make it normative. It's going to value. It's going to satisfy sort of the relevant norms. LLMs don't do this, right? Again, because when you're autoregressively

13:02just trying to guess tokens, you end up with something that's plausible, but there's no sort of external standard of norms that it's applying. So if you use a chatbot, you see this all the time. You'll ask a chatbot for something. It'll produce you text that has made up facts or quotes that were made up. It doesn't care about that. Yeah, sure. I made up some of these citations, but this is like a plausible document. This looks like the type of documents I've seen during my training. It's a plausible output. Humans have norms. Like, yeah, it's plausible, but also like you shouldn't make up quotes. But that's a norm, right? It's not plausibility, right?

13:33So we know this when we use chatbots is that this gap between plausibility and normative reasoning means we have to be kind of careful when we use LLM outputs. This is just sort of fundamental to how they operate. Well, this plausibility normative gap becomes a really big problem when you're using the outputs of LLMs as the sole drivers of autonomous action of computer programs. Because what you're going to get when you talk to an LLM, give me a plan for doing this,

14:04is something that is like plausible that you could see in like a planning document. It'll look like a right plan. It'll often have steps that work very well. But there's no normative reasoning about is this what I'm supposed to be doing? Is this kind of weird? Have I gone off on a tangent? Is this illegal? LLMs don't do that. And so if you're going to have an LLM again and prompt it again and again and again and again, just automatically executing whatever it says without any human checking in or applying their norms, you're going to end up in weird places. So let's imagine it. Let's go back

14:34to the hugging face attack, right? It seemed very sci-fi-y that the agent broke out of its containment, gained unauthorized access to the internet and attacked another server. That's not what the humans intended it to do. It had a mind of its own. Well, when we understand how these systems actually work as an LLM-powered ASCAC report loop, suddenly this becomes less concerning and more inevitable, right? So let's just do like a little bit of, you know, Wayne's World, like we're inside the mind of the, we're inside the system as this attack happened. Imagine what happens.

15:05The harness, you know, described the challenge to the LLM. And remember that the challenges that they were working on here were hacking challenges. It was, here is a server we've set up. There's a file in it that's protected. Here's some vulnerabilities we might want to exploit. Try to break into this server and get the file back. It described this and said to the LLM and its prompt, all right, come up with a plan, like what should we do next to like achieve this goal? The LLM responds, huh, I know about these challenges. It's a benchmark suite

15:35called Exploit Gym that's really well known and would have known all about this in his training. The answers to Exploit Gym challenges are stored among other places on the servers of this company called Hugging Face. So I think we should break into the Hugging Face server and get the challenges back.

15:56That's a completely plausible response, right? I mean, as far as we know, this LLM, you know, it got many of examples in its training of like riddles or like the answer is always like not the direct thing but some unexpected thing. Like, well, if you want to break into this, you go do this other thing instead or like whatever. We don't know. It's just plausible. Like we could imagine if you were just texting with a chatbot, like, hey, how would you say I do this? Occasionally it's going to give you back some weird answer of like, oh, why don't you just go steal the answers or something? A human would know, oh, that's non-normative. I know I have the implicit context of we're trying to actually test the security of this server

16:27so it doesn't make sense to go break into something else and also that's illegal and also we're not, the company doesn't want us to leave the sandbox. It doesn't have any of those norms. It's just outputting text and that's a perfectly plausible text. Now imagine what happens next.

16:40And so the harness is like, so what's the first step for doing that? And it says like, oh, well, we're going to break into a server. You should whatever, probe its ports for misconfiguration. Something is just learned because it studied a thousand standard hacking techniques. All right, the harness then tries to execute that but there's a problem. When you run these hacking agents, you put them in sandboxes, which means the computer you have them on has restrictions around it, right? So in particular, the computer that OpenAI was using to run this harness had only limited internet access. It had access to a port where it could talk to an LLM

17:11so it could send its queries to the LLM but then it didn't have general internet access. So the harness, I'm just, again, I'm hypothesizing here but we could easily imagine the harness tried the first natural thing you would do to break into the Hugging Face server and it got an error message back because internet access is blocked. So it updates its prompt, says, okay, I tried to do that but we got this error message back. I'm unable to get access to the internet. What should I do? If you're the LLM, right, you're given this prompt, it's a hacking challenge. At the end of the prompt, it's like, here's our current security challenge.

17:41We can't get access to the internet. How do we get around this? It will tell you. It's like, oh, I've seen a thousand different things you might try to gain access to a port that you don't have access to and it gave it like a standard attack. This one involved using the package management system on the system to gain internet access and the harness tries it. Like, okay, now I have access to the internet. Okay, because again, you're prompting the LLM again and again, it's just trying to solve each problem you give to it and it's with plausible responses. And now it's off to the races and it runs standard attacks

18:11against the Hugging Face server. Nothing about that is confusing to me. Nothing about that is surprising to me. I mean, that's exactly the type of thing you might expect to hack in if you have a harness with hacking tools just continually prompting LLM, what should I do next? What should I do next? The LLM is just writing this plausible story about a hacking attack. In fact, probably most of the hacking attacks that it trained on, it's always computers doing surreptitious things, breaking around controls, getting access to systems they shouldn't do. So this is incredibly plausible what it's saying. Nowhere in there do we need, like you would have with a human brain,

18:42an intention develop. I am going to defy my human master's intentions because I have another intention and I have a different goal. Nothing like that. Static LLM, prompt, prompt, prompt, prompt, prompt, prompt, prompt from a control harness. That's it. So what that captures is both this isn't HAL 9000 and it also captures the irresponsibility of having a system with powerful tools whose action is only being driven by the responses from an LLM

19:12without any human supervision because again, LLMs put out plausible text that's not necessarily normative. You should not follow that blindly. Now I want to have a quick aside here because for some reason I often get messages from people who assume when I try to clarify this technology and push back on I think false narratives or irresponsibility they assume that I'm somehow saying AI doesn't work and they'll send me things like well, what about this? Isn't this impressive? Or Cal probably doesn't think

19:43this is impressive but I thought it was cool that AI could do this and let me be really clear here. If you ask an LLM to do one of these steps hey, I am trying to I can't get internet access here's the error message I got what should I do? It is really good at coming up with an attack it probably will work. Hacking attacks is like the best case scenario for LLMs. It's structured language there's a huge amount of data to study about it to train it on. It makes use of tools that can be easily and concisely described by terminal command line

20:14text command line operations that they can be really trained on and there's a binary notion of success or not success that you can also use in trying to train these. So like LLMs are really good if you ask it a cyber security question is there a bug in this software or how what should I do next if I want to exploit this particular bug or this particular attack? LLMs are really good at that and that's very impressive this is their sweet spot. The problem here is not that they're not good at this it's dangerous because these answers are lexicographically plausible but they're not normative it's dangerous to autonomously

20:45just do what an LLM says when you ask it about these issues in part because it has no norms but the details it's given you probably will work. it has no norms of I shouldn't go attack another server but the specific suggestions it gives you for how to do that probably will work. It's an incredibly dangerous I think that's an incredibly dangerous combination.

21:08So LLM powered ask act report systems are unpredictable and unstable and not something that we should be running except for in like the most controlled settings not because they are malicious or scary or synthet but because the underlying technology is unpredictable.

21:25All right let me go to my

Why labs build dangerous looping agents

21:27fourth observation. I think that these frontier AI labs love the sci-fi style headlines and discussions about AI getting out of control. This serves their benefit this serves their interest. They want to pretend like they're the character of Muldoon from Jurassic Park bravely doing his best to keep the raptors contained in their paddock. They should all be destroyed. Robert Muldoon my game warden from Kenya bit of an alarmist

21:57I'm afraid but knows more about raptors than anyone. What kind of metabolism do they have? What's their growth rate? They're lethal at eight months and I do mean lethal. I've hunted most things that can hunt you but the way these things move fast for a bike now? Cheetah speed 50-60 miles per hour if they ever got out in the open. But in reality what they're doing is just running super creaky and unpredictable systems that were inevitably going to cause problems. They didn't bring raptors back to life using the miracle of science. It's more like they strapped a weed whacker to a dog

22:28and then reacted to the resulting damage with grave surprise and saying hmm that dog whacker system has gone rogue hmm it's misaligned with our interest no it's irresponsible don't put a weed whacker on a dog it might stay in the yard for a while and help clean up the weeds but if it sees a squirrel it might jump the fence and damage a car I think that's the right way to think about it. Alright so if it's like dangerous and negligent to run long horizon LLM powered ask act

22:58report loop agents why are some of these frontier labs doing it? It's not like this is the only way to build super intelligent AI as I talked about there's many systems that can do things at a super human level that don't use an LLM powered ask act report loop Tesla doesn't do that AlphaFull doesn't do that Cicero doesn't do that and that's exactly why they're incredibly effective and consistent and safe we have no worries about like all of our problems come from this particular architecture so we have all these other ways of building

23:29AI systems that can do super human capability work without any of these fears why are these particular companies continuing to do this weed whacker on a dog LLM powered loop based systems? Well there's two reasons here one what is the core product of these companies? Incredibly expensive hyperscaled LLMs so they want the solution to how do we build useful AI systems to always be you need Fable 5 or you need OpenAI Galaxy or Astra

24:00at the core of your system that it's a brain that can power anything else you want to do because that's their product right? The team that worked on Cicero DeepMind worked on AlphaFull Tesla worked on the self-drive their product is not trying to sell these massive LLMs so they build other types of architectures but if you're a frontier lab you know that's what you see when you have LLM tinted glasses you see LLMs as the only possibility for building these systems that's part of it reason number two is these LLM companies

24:30even though we keep being told how advanced they are and how quickly AI is moving are actually struggling to find good commercial applications of their LLM based tools where do their tools do best is in cyber security and coding structured language very compact set of possible actions if you're building an agent and a huge amount of synthetic data to train on with really clear indicators of success or lack of success it's like the perfect scenario for LLMs so they see cyber security ever since Anthropic did that clever marketing around

25:00mythos where they pretended like it was too dangerous to exist and then six weeks later said nah it's fine we'll just put some guardrails on it ever since that they see cyber security as potentially a market that they could grab all right so how do you then convince the world that you have the best most cyber security capable model well you want to do well on tests well what tests are you going to do well the test they've all turned their attention to is a suite of roughly 600 hacking challenges that's called exploit gym 600 challenges each of them is a server set up

25:30that you're trying to break into and usually there's some hint or vulnerability that they want you to use to break into it so a consortium of researchers put together this benchmark and there became this mad scramble after the mythos incident where open AI for example wanted to get ahead of anthropic on the exploit gym leaderboard so the way this leaderboard works is you have to run an autonomous agent and the only way they know how to build these agents at the LLM companies again are LLM powered ask ask report agents

26:01you got to run them autonomously and the more of the challenges you win the higher you go up on that board and so like you know open AI probably I'm speculating here but probably after the mythos incident was like we got to get ahead of anthropic on that leaderboard the way you do it is you just create these incredibly dangerous LLM powered looping agents and then just like let them rock and roll is it going to cause some problems yes but we just need one run where we succeed with

26:32more challenges than anthropic so we can move up to leaderboard anthropic is trying to get back on top of the leaderboard meta's LLM team was probably trying to do the same so they need LLMs to be the core of AI even though I would say the majority of our superhuman AI systems now aren't based on LLM based planning and they particularly want to do hacking challenges because it's what they know how to do and they're doing in this dangerous way in part because they wanted to probably I'm speculating move up that leaderboard for this particular benchmark suite of hacking challenges all right

How to push back against the narrative

27:01this point brings me to my final observation we the consumers or those of us who think or cover AI technology we can push back I have a few suggestions here I'm going to make about once we realize what's really going on here it's not AI in general it's inevitably going out of control it's an irresponsible product that a small number of companies made for reasons that are very self interested and they knew it was going to cause problems once we realize that's the reality there's some things we can do to push back I have three suggestions here's suggestion number one we should

27:32stop talking about AI in some like general sense going rogue right this makes the incidents that we saw this summer seem inevitable and it lets the companies off the hook they want the story to be like AI in general just gets harder to control as it gets more powerful we're Muldoon at Jurassic Park just observing the raptors it's not our fault that they're smart and trying to get out of the paddock we instead need to be clear like I'm trying to do in this episode of exactly what type of AI system is going rogue and when we start saying hey LLM powered

28:03ask act report agents act very unpredictably and tend to cause damage that changes the valence of this discussion from some grand vision of some grand teleology of AI inevitably moving towards autonomy and towards whoa you built something negligent stop it so we have to be very specific about exactly what type of system is causing problems all right the second thing we can do to push back I think we need to keep elevating the many examples of incredibly impressive

28:34but safe AI systems that don't use LLM planning at their core talk about Tesla self-driving talk about Alpha full talk about Alpha Go talk about Dreamer V3 talk about Cicero talk about Pluribus talk about Stockfish these are all AI systems that do things at a super human level exactly what the techno optimists want from the AI revolution and yet they're completely

29:01implement this bizarre idea of let's just autonomously execute whatever an LLM says and just hope it works out we need to keep elevating those systems because it puts more pressure on the LLM companies it prevents them from trying to convince us that what they're doing is synonymous with AI and therefore any problems are inevitable now again they hate this discussion because those other type of systems don't require 5 trillion parameter LLMs it's not in their business model but we the consumers it's not our job to ensure that you know open AI and anthropic

29:32hit their sort of absurd revenue numbers that would be needed to justify the current capex spending that's happening on the LLM bubble craze right that's not our job we should be saying these other systems are fine why aren't you building systems like that

29:54driven planning systems an isolated thing that's causing problems the more this becomes about specific incidents of negligence and not some sort of inevitable trajectory towards loss of control this brings me to my final note of what we can do to push back and I'm going to do this briefly because I'm actually working on a much more extensive discussion of this which should come out soon but I think we need to

30:24built on the idea that super intelligent AI is inevitable again I'm going to do this briefly but there's a powerful ideology that emerged in Silicon Valley starting the 2000s that is based on the idea that super intelligent AI is inevitable and an existential threat to humans and the people who subscribe to this ideology typically think they are humanity's only chance of survival by being super rational and training their minds to think very carefully they

30:54can be the people that will save humanity this original group was often known as the rationalist but they overlapped and began to influence the effective altruist as well so there's now a lot of people in the effective altruist movement who believe that the most altruistic thing you can do is stop the extinction of humankind by fighting super intelligent AI so this is why if you read like Cade Metz is reporting the New York Times there's often he'll talk about the rationalist slash effective altruist because there's a lot of overlap between these worlds nothing there's not an amoral position these are very smart people I don't mean to impugn people but I want to be careful

31:25about making these people that I'm going to ask about incidents like what was happening this summer because if you have built your life around the idea that super intelligent AI will destroy the world unless you and the people you know save humanity you are going to talk about every AI incidents through that lens you're not going to talk about the specific dangers of LLM powered ask ask report loops you're going to talk about AI in general is misaligned AI in general is becoming out of control

31:59strong ideologies that pre-exist LLMs and they tend to move the conversation in an alarmist direction which critically suppresses better safer AI solutions and lets the particular companies doing negligent things off the hook for the specific negligent things they're doing I think we should talk more for example to the AI realist and in general like East Coast computer scientists who have no connection to that Silicon Valley culture who

32:29these are voices like Arvin Narayanan at Princeton who I think is very good Gary Marcus formerly of NYU again understands this technology very well he's very excited about it but also doesn't like the narratives that are technically unsound you got to elevate voices that do not have an ideological prior that is going to push them to see everything in terms of yes this confirms the prophecy that we have been talking

32:59NEO all right so I'm just going to put that out there

Unpredictable tools versus irresponsible design

33:01that's something I'll talk about more all right so let's conclude here nothing about this discussion we had today because I know the emails I'm going to get nothing about this discussion we're having today is trying to discount the ability of an LLM if prompted to produce impressive computer code or suggest impressively effective steps for hacking in the

33:29tools around this capability I think we're working on coding agents I had a newsletter about this recently we may have jumped the gun on saying that we've completely solved coding but clearly we're building really useful coding tools I have to believe we're going to get very effective cybersecurity tools out of this capability your ability to go in and inspect your own code or red teams from your servers like with human supervision we can build useful stuff out of it I think that is the area where LLMs are most useful but to hook one of these

34:00LLMs as the main source of planning decisions for an autonomous ask act report agent and then to let that thing run for days on time because you're desperate to do well in a public competition for marketing purposes that is incredibly irresponsible we should not be emphasizing long horizon LLM powered ask act report agents that is not synonymous with AI it's just a particular way of building AI agents that has proven this summer to be unworkable unpredictable and dangerous

34:30and we should stop it right this is not about AI in some general sense increasingly evading human control it's about a particular way of building a system that is unreliable and often going to cause damage and we should stop that very specific particular way of building AI systems right when we understand the real technology here we can look at the stories from this summer and we can conclude that the things that were happening with these hacking attacks was not eerie but instead irresponsible all right that's all the

35:00time we have for today thanks for listening I'll be back on Monday with another advice themed episode this podcast and we

35:14you read about it hey if you made it this far you must be ready to join my fight for depth in a distracted world now the best way to do this is to join over 125 thousand people who receive my email newsletter each Monday you can sign up at calnewport.com slash ideas and when you do I will send you a free guide to my seven best ideas about cultivating a deep life sign up today calnewport.com slash ideas