“Rogue” AI Not Inevitable. It’s a Choice.

The Fear of Rogue AI

0:00The big fear in AI at the moment is the idea that we're losing control of these systems. I think these concerns really got started last summer when OpenAI revealed that some of their AI agents had hacked into Hugging Face, a third-party company, even though they had not been asked to do that. As the summer went into the fall, OpenAI revealed that that was actually just one of many unauthorized attacks that their agents had executed, perhaps not wanting

0:30to be left out. Anthropic then came out last week to say one of their AI agents had submitted a false report to the Philadelphia Police Department. The narrative here is clear that as these systems get more powerful, they become harder to control. This is scary, but is it right? As a computer scientist, I actually have some real issues with this idea that we cannot control AI and that these incidents of rogue systems that we're seeing are somehow unavoidable. The truth

1:03here is actually both more reassuring and more infuriating once you know it. So let's get into it.

Zooming Into the AI Landscape

1:13All right, so I want to start this discussion by making probably the most important point that I will make in this episode, which is AI in general is not going rogue. When you hear the heads of Frontier Labs saying AI is getting out of control, we have to slow down progress in AI, this is an inaccurate way of speaking about our current moment. Artificial intelligence covers a huge variety of different technologies, and it's actually a vanishingly small subset of all

1:48the different AI systems that exist and all the AI research that's going on. There is only a very small subset where we begin to see the rogue AI incidents that have gotten everyone worried in recent months. So I want to try to make this clear with some graphics here. I'm going to pull up on the screen a sort of diagram of what we're going to call the AI landscape, where I'm listing a lot of the different types of AI that we might see today that's impressive. This is not a complete list, but on here, if you're listening instead of watching, we have self-driving cars, we have game engines, we have protein folding, like alpha fold coming out of DeepMind. We have medical image

2:22processing, you know, studying medical films to look for signs of diseases or cancers or pathologies. We have image and video generation. And among this big list of various things in the AI landscape, there's one in particular that's labeled LLM-based systems. So if we want to get to the systems causing problems, we have to zoom in on LLM-based systems. So we're zooming in on this landscape just to LLM systems. So we see we have the full AI landscape out here. Now we're zooming in just to LLM-based systems. So is

2:57this where the problem is? Not exactly, because there's lots of different LLM-based systems. We have chatbots in here. We have text generation and editing. We have natural language interfaces to software. We have language translation. We have text generation. So we have a variety of different things you can do with LLM-based systems. But there's one particular thing you can do with LLM-based system that causes, it gets us closer to our problem area. And that's over here where we have LLM-based agents. So let's zoom in on LLM-based agents. Is this, so now we've zoomed in twice, is this where all

3:30the problem is? Yes, but not all LLM-based agents. So now we've zoomed in twice. What do we see here? We have coding agents. We have math agents like the type that worked on the problems that OpenAI recently announced. We have computer use agents such as Cloud Cowork. And in this universe of LLM-powered agents, there's one particular type that I've labeled on here, open-ended LLM-powered agents. That is where all of the problems are happening when it comes to these rogue AI incidents. All right?

4:01So if we imagine we had to zoom in there multiple times, that's three big zooms to smaller and smaller regions of the AI landscape before we get to the very specific type of system where we get things like the hugging face attack or the anthropic agent filling out illegal police forms. All right. So we aren't dealing with AI in some general sense going rogue. We're dealing with a very specific type of LLM-powered agent that I'm going to call an open-ended LLM-powered agent. To go forward here,

How LLM Powered Agents Work

4:36I want to give you a brief primer, which I've talked about before, but I want to return to here about LLM-powered agents before I talk about what specifically makes an LLM-powered agent open-ended. All right. So for those who don't remember, when you hear about an LLM-powered agent, it's easy to conjure some sort of notion of a complicated program that is sort of approximating the human mind that's constantly learning and making decisions and morphing maybe and evolving and swarming and doing all sorts of scary things. But actually at the core of LLM-powered agents is

5:10a computer program. So I'm going to draw this on the screen here for those who are watching. So we have a computer program. I'll put it over here. I'll put P in the middle of it to represent program. This is a normal sequential computer program written by a human in a normal programming language. It's completely understandable by humans. In fact, a lot of agents, the program itself is really not that interesting. All right. So what does this program do? Well, it has access over a network connection to a large LLM. So I'll label this big green blob over here an LLM. Okay. So

5:44there's an LLM over in some data center. Even that's not even that accurate. There's not like a machine that hosts an LLM. You have many racks of GPUs where each rack of GPU is perhaps virtually simulating some subset of the weights in the LLM. And different queries will go to different GPUs to have the same weights. There's no actual system where the LLM lives. There's just a bunch of copies of the weights spread out over a bunch of GPUs. Okay. So what the agent program does is the following. It will send a prompt to the LLM. I'll draw an arrow going from the program to the LLM.

6:19That prompt, this is just in text, will in essence say, I'm an agent. Here's my goal. Here's the tools, like the software tools I have available, the tools I can call. And here's my current status. What should I do next? So it submits that to the LLM just like a chatbot user would type in a prompt to send to a chatbot. The LLM, as LLMs do, it generates a response, sends it back. Like this is what I think you should do next, given your goal status and current tools. The program then executes whatever it is that the LLM suggested. Because remember,

6:55it gave it constraints. Here's the tools I have that I'm able to call. And so the LLM response might say like, great, you should move this file from this directory to that directory using the MV command that you told me you have access to, right? So then the program actually executes whatever step the LLM told it. Then we loop. And we do that again and again and again, right? So this is why I sometimes call LLM-powered agents prompt loops, because I think that gets to the core what's really going on here. It's a program that repeatedly prompts an LLM, says, what should I do

7:29next? Does that step? Repeats. All right. Here's my new status. I'm a, you know, I'm a program. Here's my tools. Here's my goal. Here's my status. What should I do next? Why does it have to repeat itself so much? Well, remember, LLMs have no memory. Nothing adjusts. They're static. So every prompt that goes to that LLM has to explain everything the LLM needs to know to suggest the next action to the program. That's what an LLM-powered agent actually does. Now, there is an issue with these that people have known since the beginning of messing around with them, which is using an LLM as the sole driver

8:03of your actions as a computer program. So I will do whatever an LLM tells me to do. This is risky business. And anyone who's used a chatbot knows why. It's because LLM outputs are engineered to be plausible, right? So they're always a plausible response to whatever input you give it. They understand well. They're tuned to try to produce a plausible output. I've explained these mechanics before. It goes all the way down to their pre-training, which is about guessing tokens from real text. And if you do that auto-aggressively for many tokens in a row, what you get is a plausible

8:35extension of the original input prompt. So they give you plausible answers. Whatever it spits out is a reasonable extension of whatever original prompt you gave it. But these answers are not normative. Normative meaning subcoming or obeying the sort of like natural norms that a human would think makes sense when you're trying to give an answer, right? So they don't necessarily have human morality. They don't necessarily have notions of truth or falseness. They don't have a notion of maybe it's bad to make up a citation in a legal case like that super illegal. There's no notion of

9:08that. There's no world model or moral core or policy actor that's trying to check its answers to see if it passes muster. It's just giving you a plausible answer. That could be tricky when you are changing these responses into actions because what's going to happen is they might start suggesting actions which are completely reasonable actions that will work and will make things happen but are non-normative. They might be unexpected. They might be unpredictable. They might be amoral. They might be illegal. Anyone who's used a chatbot enough knows this, right? You chat

9:41enough with a chatbot. They occasionally lock in on weird dark directions or unexpected directions or you have to go back and say, no, no, let's try this again. When you're an agent, the problem is there's no human. You're not a human. The agent's just a program that's blindly executing what the LLM says. You can get into weird places if you run this prompt loop experiment of let me blindly execute whatever output an LLM gives. And yet when I was showing you the landscape of LLM agents, I said there's multiple types of agents that exist that don't cause much trouble. So we had,

10:16for example, OpenAI's Astra-based math agents that were involved in the recent math slop drop of those 322 somewhat inscrutable lean proofs. Clearly the LLM involved with these agents is very large and very scaled. They trained it on math extensively and it was able to do very advanced mathematics. So by you know, some notion of smart as a very smart agent, they also ran these things for a very long amount of time, you know, hours at a time. So thousands of prompts trying to explore these problems. They never went rogue. OpenAI is not saying good news, bad news, good news. We got a,

10:52you know, a finite time blow up of the Navier-Stokes equations. Bad news is on the way, the agent also, you know, ordered a bunch of hookers to the OpenAI headquarters, right? That didn't, it just worked fine. Same thing with coding agents. LLM-powered coding agents are widely used. Millions of software developers use these every day. Many of them are connected to some of the most advanced LLMs we have and they run many autonomous steps. And we also don't hear a lot about coding agents being used by professional software developers going rogue and trying to hack into other people's servers or doing

11:26things that are well outside of what the programmers thought that they might do. We even have data about this. I'll bring it up on the screen. OpenAI in March actually released a study called How We Monitor Internal Coding Agents for Misalignment. So remember, agents prompt an LLM for all of their actions. So if you want to see what an agent is doing, you can just look at all the LLM prompts. In this study, OpenAI looked at tens of millions of prompts to LLMs coming from agent programs. And what they found is the types of rogue behavior they most fear, they call high severity incidents like

12:02sandbagging, sabotage, scheming, the stuff that made us nervous about hugging face didn't occur at all. So coding agents are not having this problem either. They want to take a quick break from this conversation to cool down and to talk about some of the sponsors that makes this show possible. You all know that I love working with Cozy Earth because I'm a giant fan of their products. Now recently, I have been using, and when I say I, I mean my family because they keep stealing it from me,

12:35the Cozy Earth Bubble Cuddle Blanket, which is a perfect combination with their famous bamboo sheet sets, which we're obsessed about and have multiple pairs of and travel with because we like them so much. The Bubble Cuddle Blanket, it's soft, it's heavy, you really can cuddle in it, and it just gets passed around my family, especially now as it's getting colder in the mornings as fall advances. There's always a competition for who gets the Bubble Cuddle Blanket. It's the perfect addition

13:08to their already fantastic bamboo sheet sets. So consider picking up a Cozy Earth Bubble Cuddle Blanket, and for sure you need some of their bamboo sheet sets. Get them today. Look, there's no risk in giving this a go because they have a 100-night trial and a 10-year warranty. If you don't love them, you can send them back, but you won't because they're great. So go to CozyEarth.com and use code DEEP for up to 20% off home and sleepwear. That's CozyEarth.com, code DEEP, and when they ask you

13:39where you heard about them at checkout, let them know you heard about Cozy Earth right here in the Deep Questions podcast. I also want to talk about our friends at Gusto. Look, no one starts a small business dreaming of all the payroll tax forms they're going to get the file, right? That's the type of thing that you procrastinate on because it's boring and it gets in the way of actually growing. This is where you need Gusto to handle those type of administrative tasks in the background so you can focus on what really matters. Gusto is an online payroll and benefits software tool

14:14built for small businesses. It's all-in-one, remote-friendly, and incredibly easy to use so you can pay, hire, onboard, and support your team from anywhere. You get unlimited payroll runs for one monthly price, no hidden fees, no surprises. You can also save time with built-in automated tools like offer letters, onboarding docs, direct deposit, and more. It's ranked number one on G2's highest satisfaction products list for 2026 and trusted by over 500,000 small businesses, including the ad

14:44agency we work with here at the podcast. So what are you waiting for? Get started and try Gusto today at Gusto.com slash deep, and you'll get three months free when you run your first payroll. That's three months of free payroll at Gusto.com slash deep. One more time, that's Gusto.com slash deep. All right, let's get back to the show. So how do those type of LLM agents avoid trouble where other

Why Targeted Agents Avoid Trouble

15:08types of LLM-powered agents, like the ones we heard about this summer, don't, that they do cause trouble? Well, there's three things that keep the, I call these targeted LLM-powered agents, like coding and math agents. There's three things that keeps them on the reservation. One is narrow goals. They're given a pretty specific goal. Like if you're a math agent, it is, I want you to, you know, write this part of a proof in LaTeX, which is like a markup formatting mathematicians use to write up math papers. You're writing up a proof in this format into this file. It's a

15:41narrow goal. Turns out when you give LLMs, say, give me a step to make progress towards this narrow goal. They're trying to give you steps that help you make progress towards that narrow goal. They're not going to go off and do something rogue necessarily. The second check that helps keep these targeted LLMs from going rogue is they have a restricted tool set. So the tools the agent program has available to actually take actions in the real world is appropriately restricted. I mean, if you're a math, if you're the math agent, maybe you have the ability to do searching on archives and math

16:14papers. I'm not quite sure if that's all trained in or if they could also do real-time searching contextually on relevant papers. And you have the ability to read and write files. You can actually write proof text into a file within a limited directory structure. So you have limited tool sets. You can't do much damage. They also have oversight. So this is particularly true of the coding agents. If you did write a very simple coding agent, gave it tools for the terminal and just said, go on and write this code, they will get in trouble. Not in the rogue sense, but just

16:45they'll do unexpected stuff. They'll overwrite files. They'll delete files they shouldn't have. And so a lot of effort has gone into the agent programs for coding agents. Think of it as hard-coded oversight to automatically notice and avoid the most common types of issues that a coding agent might accidentally do. We know this in part because the code, the source code for Anthropics agent, Claude Code, accidentally got released. It's 500,000 lines of code in the program

17:19that prompts the LLM. And a lot of it is just hard-coded special cases and checks. They have pattern matches in there. This is just through experience. The programmers who worked on it are programmers. So they kind of know the types of things to look out for. So there's all sorts of hard-coded special cases in that program. Hey, if it says, you know, RM-RF, which, you know, means delete a bunch of files without asking, they've hard-coded something to say, eh, maybe don't execute that. We'll put that one aside. They actually have pattern match. It's

17:50interesting. They have pattern matching in the Claude Code harness to actually look for common terms from whatever the user is typing in that indicate frustration, like cursing. And if it sees there's frustration words in the user's prompt, the program will assume, I don't know what the LLM has been telling me to do, but it's probably pissing off the user. So why don't we stop now and sort of check in with the user before we take any more steps? There's also a lot of just hard-coded checkpoints in the coding agents. You know,

18:21for simple programs like amateur people making interfaces for their clubs that they're a part of, that they vibe code, that's fine. But for professional programmers, they're checking in way more often. They have, you know, recursive, they have testing going on. They want to check in. All right, tell me what you did here. I need to check this. I need to run test on it. So it's a lot more interactive than you would think. So narrow goals, restricted tool set, and hard-coded oversight can make for the right task. LLM-powered agents never go rogue. So why are we having rogue agents

The Danger of Open Ended Agents

18:53still? They all come from what I've been calling open-ended LLM-powered agents. And why did they have trouble where these targeted LLM agents do not? First of all, they replaced narrow goals with broad goals. Look at the hugging face attack, right? Like they're given a lot of latitude. Like here's your goal, but you have sort of complete latitude in how you make this happen. Because that was the nature of many of the agents involved in the hugging face attack were being used, not all of

19:23them, but a lot of them were being used as part of a cybersecurity benchmark suite known as exploit gym, where the whole idea is to be really creative and find whatever way you can to try to accomplish the goal. Typically, it's like breaking into a test system, right? So they have broad goals. They have a broader range of possible actions. They're not being constrained. They're given a powerful tool set, right? They're given tools, these powerful tools you need to win at hacking challenges. So now they have very powerful software. In particular, they have access to these package services where you can

19:58download special packages to help you do particular types of attacks. So they can, it's like, I'm trying to think of a good analogy here. The good analogy would be if you have in targeted agents, you're giving two or three tools to the agent from your toolbox. These open-ended agents, you give them a credit card and access to Ace Hardware. They can just go get whatever they need to do. You can get into more trouble that way. And finally, they have very limited oversight. First of all, because these are tests being done rapidly, unlike a coding agent where you have, you know, a year or two of people

20:30painstakingly hard coding in special cases, these, they just rock and roll, right? They're just like, let's just see what happens. And on purpose, if you look closely, like at the Redwood or meter report on the hugging face attacks, the engineers just let these things run. They weren't really trying to check in. They had very little monitoring going on. You very easily, there's all sorts of checks you could have easily put into the agent program itself. Like, every time I'm trying to, you know,

21:00access a new domain, let's just stop and have a human look at it. Let's have a human check every hour and look through the log of all the prompts to the LLMs. Let's even just have another LLM summarize that. And like, let's just check every hour to make sure that we don't have something unusual going on. They almost like purposefully didn't want to have oversight. Not only did they not want to have oversight, but as the reports make it clear, they would see that something really troubling was happening and they would just restart and let it run again and come back a few more days. They had reports from previous experiments because OpenAI, their new misalignment reporting

21:35framework has them say when things were detected and when they actually occurred. They had detected and have reports of, hey, these agents are doing unexpected, unpredictable activity outside of the sandbox. They had those reports already before they started new experiments with similar agents that then went on to do the same sort of thing. So it was almost like purposeful lack of oversight. So if you have really broad goals, you say you do what you need to do to solve this problem. We have basically no constraints on it. You give access to a wide, diverse, and powerful set of tools and you have very little oversight. Those type of LLM-powered agents are going to cause problems.

22:09Why are they going to cause problems? Is it because somehow this static LLM distributed and copied over thousands of GPUs as actually some sort of coherent, sentient entity? No, it's because LLMs are plausible but not normative. And if you're just going to keep asking an LLM again and again, what should I do next? What should I do next? They're going to give you weird responses eventually. They're going to lock in the paths you wouldn't expect. They're going to be non-normative. They might do things that are harmful or illegal. And if you give them broad goals, powerful tools, and no oversight, there's nothing stopping that from unfolding in ways that's going to be

22:42unpredictable or harmful. The key point I want to make about this is there's nothing inevitable about the AI industry needing to design and deploy open-ended LLM-powered agents. It would be completely reasonable, for example, in this current moment for the industry to say, that's a bad idea, but you know what? Targeted LLM-powered agents, we're finding use cases for it. We're getting better at these coding agents. We're figuring out math agents. They're a little

23:13inscrutable right now, but they're powerful. So clearly, we're going to be giving a year or two, we can get a great tool for mathematicians. There's got to be a hundred other sectors where we can build targeted LLM-powered agents that are, you know, they might sometimes do stuff you don't expect, but aren't rogue and are trustworthy and generally do good stuff for you. And you could build out a really wide industry, just like the original software industry of lots of different tools, each of which tuned to a different task or industry, and each of them bringing in revenue. And you could build a really solid, very lucrative technology industry on top of that, right?

23:45There's nothing wrong with that. You could also then say, hey, you know, these targeted LLM-powered agents, this is maybe not the most effective way to do it. Actually, what we really want to do for a lot of these tasks is neurosymbolic systems. To borrow a term from Gary Marcus, a lot of the original AI landscape I showed you, where LLMs were just one subset of it, a lot of those systems, like the systems that play games very well, the systems that drive cars automatically, AlphaFold, these are neurosymbolic systems. It's not a program just prompting an inscrutable LLM and

24:17just doing what it says. They have some neural networks in there that are trained with machine learning. They have some symbolic components in there. These are programs written by humans. They're understandable by humans, and they all work together. Turns out if you do neurosymbolic systems, you don't need a 5 trillion parameter LLM that has to use up all the water of Idaho just to run. You can actually fit them all on a single chip I can run in my basement. And so there's this whole exciting trajectory we could have of targeted use cases where we could use targeted LLM-based agents and neurosymbolic agents to bring the price down and the effectiveness up even more, and boom,

24:51it's a massive industry and makes all of our lives better. There's no reason, right? We don't need open-ended LLM-powered agents for AI to be useful to people. And yet, we see an insistence on building and testing them. So why is this? Well, first of all, I think we have to clarify, it's not everybody building open-ended LLM-powered agents. The two companies in particular that are really pushing this research hard is open AI and anthropic. Now, to be clear, this doesn't mean that no other

25:26hyperscaler-style AI lab is working on these. We know other companies have had some issues with these. Meta and Google in particular has had a little bit of issues, but they're kind of, I don't know, I feel like the cases we've had of rogue AI from like Google, for example, I'm not super worried about. They were doing some open-ended agent testing, but they were more constrained for cybersecurity. They sent it off to this company called Irregular to run the test, and Irregular set up the sandbox

25:56wrong and gave it full access to the internet. And it did, it tried to guess some passwords or something, right? And I think they're like, okay, whoops, you guys said wrong, let's not do this. Meta had a similar issue, and then they kind of backed off it a little bit. But open AI and anthropic are pedal to the metal. This is where like the big, especially open AI, this is where like the big case after case of months of actual ongoing hacking. Like they are doing this at a level that is in my, I don't know how the numbers, but seems like 10x what anyone else is doing. So those

26:28two companies in particular are super insistent on open-ended L-empowered agents. Come hell or high water, we got to keep running those things. Why is that? Well, if we look closer, open AI and anthropic do not like that alternative future that I just gave to you. They do not like the idea of a more normal AI technology landscape where you have a huge number of AI, task-specific AI systems like coding agents, like the math agents, just multiplying that across a hundred other different

27:01types of task areas. And you have all these systems, a lot of them LL-empowered and targeted, some of them neurosymbolic powered, and there's just a lot of really useful AI tools in the world. They do not like that vision. And there's three reasons why. Vision number one, why those two labs and particularly don't like those is competition, right? That vision, when you get task-specific to targeted LL-ems, does not require that you have the biggest, baddest LL-em on the block. We see this with coding agents. I mean, this has already become a problem for open AI and in

27:34particular anthropic that makes a lot of money from coding agents is that you can use smaller LL-ems. You can use cheaper LL-ems. You can route to different LL-ems based on the complexity of the coding tasks that you're doing. My prediction about the coding, the targeted LL-based agent coding agent space is that we're going to see a new generation of very specialized, pretty cheap LL-ems. And I think they're going to be LL-ems that are going to be able to run on a rack. You can put in your own server room at your own office that have been super tuned for particular languages. I'm a big believer,

28:09talking to a lot of programmers, that the next evolution of coding agents is actually going to be the emergence of a sort of intermediate pseudocode language that is forgiving. It's not syntactically demanding, but it's pretty accurate. And a programmer can write really quickly in the pseudocode and get much more exact translation from that in the code, thereby significantly reducing the amount of checking and verification that's required right now where the code production is a little bit more scattershot and inscrutable. This is going to be able to run on much smaller LL-ems. All of that's bad if you're in the business of building giant LL-ems, right? So you've got to

28:44understand Anthropic and OpenAI only make money when you need their newest, latest, greatest LL-em to make progress. If a cheaper LL-em is enough, you're not going to them. They're Ferrari, right? And these cheaper LL-ems are Honda. And they want to keep people wanting to buy Ferraris, even though they're sort of ludicrously expensive. So they don't like that vision. They hate the idea of neurosymbolic systems because, yeah, those might have neural networks in them, but they're small. Man, you look at these superhuman game-playing engines. They can play games better than humans. These things use small

29:21neural networks compared to what's going on over at OpenAI at Anthropic. I mean, you can run, you know, Dreamer V3 fits in the memory of a single GPU. You can run Stockfish, the chess engine that can play chess better than a human. You can run that on a laptop. That's scary to OpenAI at Anthropic. So that's one reason why they don't like the vision I gave where you don't need open-ended LL-em agents. Reason number two is they've been convincing the military in particular that open-ended LL-em-powered hacking agents. Sure, they might be unpredictable. Sure, they might break into like places you don't

29:55want, but weapons are dangerous and this is a weapon we need. So they have this sort of military industrial complex type argument that open-ended LL-em-powered hacking agents in particular, so this particular type of open-ended agent has to be in our nation's arsenal. And so that's another reason why they think we have to have these. We need things that can run for days at a time and run these sort of complex hacks. I think that's highly suspect. I think Syntaur hacking and defense where you have highly trained cybersecurity experts working alongside models and much shorter horizon

30:31agents that have been trained on cybersecurity is much not only safer, but much more creative and powerful and effective than just letting an agent run by itself for days at a time. I don't buy this idea that if you don't build this highly unpredictable, combustible, long-term agent yourself, that if someone else does, you're somehow less prepared to defend against it, which I don't think is true. But that's one of the reasons why another big reason they give is we have this particular cybersecurity market. The government spends a lot of money. The military has the most secure budget of anything. We want in on that. All right. Reason number three why they don't like this vision, and this is

Why Labs Push Open Ended Systems

31:04probably the most foundational, is that open AI and Anthropic, and particularly Anthropic, but open AI Anthropic, believe that you need open-ended LL-empowered agents to realize their vision of superintelligence. Remember, those two labs, this is different than almost any other worldwide or in America companies working on hyperscaled LLMs. Those two labs in particular were explicitly founded with the mission of trying to usher in superintelligence safely. Their vision is superintelligence will either be a vengeful God

31:39that kills us all, or it will be a benevolent God which will give us a transhumanist utopia, and they feel like their job is to make sure that we get the good option. And remember, we've talked about this in recent weeks. The people that are yelling, the Jacob Croxtons of the world, all these people funded by effective altruist money who are yelling and influencing congresspeople about we need to regulate to prevent, you know, superintelligence. They're all from the same family. They only differ in the fact that Dario Amadei sort of thinks we're ready to do it, and other people in his family are thinking he's not ready to do it yet. And actually, even, it's more

32:13darker than that, because actually, the more you push at this, the more you see a lot of these EA-funded voices that are pushing for this type of regulation. They're actually on the same team as Amadei, because what is this regulation going to deliver, if you really look at it? A regulatory capture landscape in which, essentially, only Papa Amadei gets to build the big LLMs and the open-ended LLM-powered agents. Him working with third-party auditors, which are other firms like Meter that are founded by the same people, funded by the same people that funded

32:47Anthropic. They all know each other. They'll come in and help monitor them. And so, basically, it's all sort of an effort, in some sense, to be like, yeah, we're trying to bring in the benevolent AI God to get our transhumanist future, so we need to stop other people from messing around with this. Let Papa Amadei do his thing. Billions of these sort of biological bootloaders might die, but we need to get, you know, we need to get our consciousness uploaded in the digital beams that are going throughout the light cone. So, they really feel like you can't get recursive self-improvement,

33:19which is going to deliver superintelligence, unless you keep pushing aggressively forward with open-ended LLM agents, because long-horizon autonomous activity is what you're going to need to have their LLM-powered scientists that can do their own automated research and push AI forward first, so they get to the superintelligence first. They get to it first. They control the God. We have a chance at transhumanism and utopia. If they don't, then maybe someone else will do it, or we'll lose the chance for the utopia altogether. So, there's three reasons why they don't like a future where you say,

33:50open-ended LLM-powered agents don't work. They're dangerous. It's a small part of the whole AI landscape. Stop doing that. We can have a huge, thriving, highly lucrative, useful AI industry, but they don't care about that. All right. So, what should we do, the rest of us, the normal people? Because remember, what is this argument? Right now, all of this debate is between billionaires, right? We have the billionaires funding the effective altruists who are also funding the lab. So, you have billionaires like Dario Amadei fighting with

34:21other billionaires that are from these effective altruist organizations, trying to figure out together how they're going to usher in this utopia. And then you have the accelerationists, which are billionaire venture capitalists that are arguing with them. And so, it's billionaires fighting with billionaires fighting with billionaires. It's like 100 people. They're all really rich. What about all the rest of us? We are very happy with a future like I just described. Most people would say, yeah, do that first thing where you don't do this one type of experiment that's unpredictable and harmful, but you do all these other things where you're building really useful

34:52AI tools that industry after industry, task after task makes our life easier and creates a lot of jobs and it explodes the sort of tech industry to be even bigger than it is now. That sounds great. That's what the regular people probably want. It's the billionaires who are like, forget that. We're trying to, for the accelerationists, either get another billion or for the EA Dario Amade types trying to get to the utopia. So, we're just letting billionaires argue with billionaires about who gets to control the world while everyone else just has to sit here, be stressed out,

35:25and say, excuse me, sir, but none of this helps out. What are you doing? You're not even trying to make a product. You're just trying to, it's just like Dungeons & Dragons. Like, what are you guys doing? You guys, it's just like Twilight Imperium made real. Are you LARPing? Like, what's going on here? So, I think the regular people need to actually push back and here's how you do it. There's two things I want to suggest. Number one, vocabulary. Refuse to play the game of using the word AI when what we're really talking about is open-ended LM-powered agents. Be specific. It puts the pressure back on the companies. When the companies say AI is getting

36:00too powerful, AI is getting out of control. You imagine it's like, as one of my producers, Nate, said, it's like a Pokemon that's evolving into bigger and more dangerous configurations as the game unfolds. And all you can do is try to slow down the rate of this one beast from growing. That's not what's happening. There's thousands of things we do with AI that are really useful. It's this one thing that's a problem right now. So, use the right vocabulary. Don't say,

36:30I'm worried about AI going rogue. I see that in the opening paragraph. And if you look in major newspapers, every article at any of these attacks, always first paragraph, AI going rogue. No, no, no, no, no, no. Open-ended LM-powered agents once again have shown that they are unpredictable and can cause harm. The more specific we are with that vocabulary, the more pressure it puts on those two companies to say, why are you doing that experiment? So, I think vocabulary really matters. Number two, when we're advocating for regulation or control, the goal for regulation, the average person

37:05has to speak up and say, the goal here is to maximize useful products from AI while minimizing harm. That is what you typically want at a market regulation. Maximizing utility to the public while minimizing harm. That's what you want from market regulations. The regulation debate that's happening on the Hill now, which is powered by the effective altruist billionaires and their sort of silent partners in the billionaire founders of the AI companies, the type of regulation they're

37:35talking about is not about maximizing utility for the everyday person. And it's not necessarily about minimizing all the harms you would want. It's about consolidating control of LLM progress so that Papa Amade can be the successful shepherd to bring us to a transhumanist utopia. So, if all of these EA-sponsored bills get past, these super intelligent bills, like you would see Senator Sanders standing next to Nate Soares proposing, if these things get past, it's just going to lead to a world in which

38:08we consolidate most of the LLM-powered agent work to basically anthropic. Open AI is kind of a mess, but maybe also kind of open AI, but basically anthropic. And them, with the help of their third-party monitors, which are friends of theirs funded by the same people, will then be able to do whatever they want to do with these open-ended LLM-powered agents. And when they go off and do even worse things as they give them more powerful tools and more runway and more long horizons, they all just keep saying, AI is unpredictable. We're doing our best to keep it under control. So, the regulation

38:38the billionaires are talking about does not maximize any utility. The utility they're maximizing is increasing the probability of getting to a transhumanist utopia. That's not what the public wants. We didn't ask for that. And they're not directly minimizing the harms you get by running open and LLM-powered agents. So, we need the demand as the people, not the billionaires, but the people. We need to demand actual market regulation. Create a market that creates useful tools. These hacking monsters aren't useful to us. It's a fever dream of Dario Amadei envisioning his transhumanist utopia.

39:13Running these type of open and LLM-powered agents is harmful to us. So, why don't we stop that until you can show us you have a more controlled way of doing that? And let's focus on what's going to get us a lot of competition with targeted LLM-powered agents, neurosymbolic agents, task-specific agents that are completely controllable, completely understandable, and very useful. Okay? So, this has become my new call for people. I really think this is, what this is really about is billionaires versus everyone else. We got to put it that way. The

39:44billionaires want to be like, oh, we're on your side. We're trying to prevent, you know,

39:49remember Terminator? You don't want that. The robot stepped on the skulls. Remember that? The robot in Terminator stepped on the skulls. We don't want that. We got your back. And we need to be like, yeah, we saw that movie. Stan Winston made that. That was an animatronic. We want useful products. We want you to stop building harmful things. So, there's an analogy I want to give to kind

An Analogy of Rocket Roller Skates

40:11of end with, right? Because I want to summarize this all together. When I've summarized all this together, what I'm really saying is AI is a very large topic, and there's lots of different types of systems, many of them very useful. And you got to zoom in three times in that landscape to get to the one type of system that's really causing a lot of these AI goes rogue concerns. And there's no real major reason to be running those type of systems except for these really unusual, eccentric, and self-interested concerns of the billionaire utopians. It's not inevitable that

40:47AI, as it gets more powerful, goes rogue. As AI got better at chess, it didn't go more rogue. As it learned how to predict folding protein patterns did not lead to, you know, the design of rogue. It didn't go rogue and tried to build its own designer organisms. AI getting more powerful does not mean it has to go rogue. It's this specific type of system. So, here's the analogy I want to leave you with. Maybe we'll make this kind of clear. Imagine that Elon Musk released rocket-powered roller skates. I'm using Musk because, like, this feels like something he would release. I don't know.

41:19Rocket-powered roller skates. And a lot of people get hurt because you're putting rockets on your feet, and they're crashing in the walls and breaking their bones or whatever. And it's kind of like an uproar about this. Like, a lot of people are getting hurt using rocket-powered roller skates. Now, imagine then Elon Musk came out and said, I have grave concerns. I believe we have to pace the frontier of wheel transportation because look at all this. People are breaking their bones. It's only going to get worse. We would say that's ridiculous. The car companies would say, what are

41:52you talking about? The bicycle companies would say, what are you talking about? The scooter people, the skateboard people, they're like, our wheel transportations aren't breaking any one bones. They work just fine. We have safety regulations. They're very useful. People really love them. Maybe you should stop building rocket-powered roller skates. That's really what's happening here. And if Elon Musk came back and said, yeah, yeah, yeah, but you don't understand. We've spent the last 20 years on blogs talking about how rocket-powered roller skates are part of this prophecy that leads to, it's going to lead to a utopia. We would say, okay, Mr. Musk, we have this nice jacket for you

42:27to put on. And there's some straps in the back. Don't worry about that. This room has pads. It's very nice. You're very cool. Grok, grok, grok, right? That's what we would do. But that's essentially what's happening right now. These small number of companies are producing the AI equivalent of rocket-powered roller skates. A lot of people are breaking their legs, and they're saying we have to slow down transportation. In fact, not only that, but let's let Dario Amadei be in charge of all of transportation because we got to be careful about it. The person who built the rocket-powered roller skates, let's put them in charge of all of it because we can't trust anyone

42:58else. I'm kind of straining this analogy a little bit, but hopefully that makes a little bit of sense. So anyways, I don't mean to get fired up, but I do think this has become, it's a populist argument. I mean, it's people versus the billionaires. And the people got to stand up. We're not going to use your news speak. We're not going to call everything AI. You got to tell us what systems have problems. And we're not going to accept calls for regulation that don't make our life better. Useful products minimize harms. You can't regulate sci-fi stories. Market regulation needs to

43:30create more useful products, minimize harms. All right? So this is why I do not think AI going rogue is inevitable. You can create AI that goes rogue, and it's a problem, but we need to look and say, stop it. We need to stop just accepting whatever these small groups of really well-funded people are telling us and say, we've got eyes. Those roller skates are stupid. I'm going to go get my bike. All right. That's enough of that. That's my rant. That's my rant for today. I don't know, Jesse. There's a lot going on. It's a complicated topic. It's a complicated topic. There's so many

44:05different things involved. There's complicated technology. There's politics. There's ideologies. The whole thing is really complicated. So I do my best. The graphic helps put in perspective. Yeah. You got to zoom in a lot. Like if DC, the city of DC where we are right now, if that is the AI landscape, these open-ended LLM-powered agents that are causing all the rogue problems is like downtown Tacoma Park. The whole rest of the city is other types of AI. I love the Papa Amadei reference. That's really, that's all of this regulation. If you look at it closely,

44:36it all ends up with like Papa Amadei and his happy elves at Meter and Redwood, who is friends who are funded by the same people. They'll, they'll take over. Don't worry about it. They'll make sure everything goes okay. I just think it's crazy. Billionaires regulating the other billionaires, deciding which of the billionaires gets to be in charge of this technology. And they all have this weird ideology about utopia. So it's like, this is not a great situation. So Amadei and Altman, are they still friends or they don't, they hate each other? They don't like each other. They used to work together, right?

45:06Yeah. Well, Amadei, Amadei left open AI because he felt that, um, that my theory on Altman, I got these theories. My theory on Altman is kind of based on, um, Ronan Farrow's and Andrew Martin's reporting for the New Yorker is he's not a great executive. Yeah. I read that too. Right. Like he wasn't, this is not, um, you know, someone who started a startup from scratch and made into like a big concern or whatever. He sort of was just like in these circles and was made the president of Y Combinator, which is an incubator. Like it's not, you know, um, they basically just

45:37take shares of kids companies and some of those companies go big and they, they cash the checks. And Elon Musk was just like, Hey, I'm going to start a nonprofit to make sure that like, we don't get bad super intelligent AI. Do you want to be in charge of it? And he's like, all right, man, cool. Like that's, so he's a little bit more slippery. Uh, Amadei is, I think a much more, um, I don't know if he's a good executive. I've heard that he's very isolated with an anthropic. So I don't know what degree he's executing over there, but he's a much better technical mind. He's from a physics background. He's very smart guy. Um, and anthropic is actually, they making, I mean, both open AI and anthropic have made

46:10progress on these models, but anthropic has probably had a higher density of high quality technical breakthroughs. So I think they're running a tighter ship over there as a technical company, but they left. Amadei left open AI because, um, Altman was being in, sufficiently committed to the billionaire vision of transhumanist utopia, like shepherding in the right AI and preventing the others. They're like, you're not doing this right. We'll do it better. And Amadei's sisters at anthropic too, right? Yeah. They left together.

46:41Danielle and Dario left, both left to start. Yeah. And she's married to a very prominent effective altruist. I mean, he came out at some point and said, I'm not an effective altruist. He's, he's, his entire life has been an effective altruist. It's crazy. It's great. I mean, it's like Ronald Reagan in his sixties being like, I'm not a Republican anymore. Like you're, you're a Republican, right? Like you could say it cause your investors don't like it, but like you're a Republican. I mean the way people don't understand like the, the percentage estimations of doom, that's straight. That's just like the shibboleth of effective

47:15altruism. That's how they talk in the Bay area. That's how, you know, you're talking to effective altruists is like my P doom is 23%. And Amadei has been like giving these precise numbers all along. That's why Jacob Croxton talks that way. It's, you know, it's a, it's a complicated world, but it's also kind of a simple world. So it's like, we have a double whammy. Americans get worried when you have billionaire oligarchs trying to control things that matter. But usually with the billionaire oligarchs, the main issue is they just want to make a lot of money and they're going to be exploitative of markets to try to maximize their wealth. You add in an eccentric

47:48ideology to the billionaire oligarchs. It's not good. Now we've had that before. If we go back in like the, the late 19th, early 20th century, they wouldn't be billionaires back then. They'd be, you know, like sent to millionaires, but you had, you know you had very powerful people with lots of money consolidating industries who had like a lot of thoughts about like foreign policy, you know, and it's like, okay, we're going to, um, start wars and we've, you know, whatever, right? We've had this before where you have concentration of economic power

48:19confined with personal ideologies that are very strong and maybe eccentric. And those two things together is always causes trouble, always causes trouble. And so it's not that surprising what we're seeing, except for it's camouflaged well. And people don't realize all these players are funded by the same billionaires. They all, it's all the same tech billionaires are funding all these players. They're all from the same playbook. They all have these guaranteed jobs at these giant think tanks that they're funded. They can just bounce back and forth between. They don't know that you'll see like the prominent, like why did

48:49this prominent Senator, I want to name names, why did this prominent Senator like suddenly is straight out of like EA rationalist talking points? His top aide left, went to one of these effective altruist institutions, came back a few years later and was like, hey, Senator, this is the biggest issue now. And the Senator's like, all right, I guess this is what we got to care about is like super intelligence. So people are just the pawns of billionaires. Billionaires have, this is what they're caring about. Like we want to make sure that we don't have the bad AI, we have the good AI and in the good AI, some of us will be able to upload our consciousness

49:22eventually to the light cone. And guess who those someones are going to be? It's going to be, it's not going to be you and me, Jesse. We're not the one, we're the biological bootloaders to use Elon Musk term that our whole point is to get to the point where we can get, you know, it'd be Dustin Moskowitz's brain, not ours. It's going to go up the space when the time comes. Let's just put it that way. I want to take another quick break to hear from some of the sponsors that makes this show possible. Look, here's a stat that genuinely shocked me. Sales teams spend about 50% of their time on admin work rather than selling, building relationships and closing deals. What if instead

49:58every sales rep could walk into calls briefed, can stay focused instead of taking notes and update their CRMs the moment the call ends? That's where today's sponsor, Pipedrive comes in. It's an intelligent AI powered sales CRM loved by growing sales teams. The save you time Pipedrive remembers your deal history and can create a pre-meeting brief that automatically pulls email records and previous conversations in the one brief ready before every call. So you always know exactly

50:29where things left off. Their new AI note taker is also a game changer. During your meetings, it records and takes notes. So you don't have to, you stay in the conversation, Pipedrive handles the rest. So switch to a CRM built by salespeople for salespeople and join the over 100,000 companies already using Pipedrive. My link gets you an exclusive 30 days free instead of the usual 14 day trial. No credit card or payment needed. Just head to Pipedrive.com slash deep to get started. That's

51:01Pipedrive.com slash deep and you can be up and running in minutes. I just want to talk about our friends at Fabric. It took me longer than it should have to get the life insurance I needed to protect my family. Now, what was the problem? Getting life insurance can be a pain unless you use Fabric by Gerber Life. Fabric by Gerber Life offers term life insurance you can get done today. Made for busy parents like you, all online, on your schedule, right from your couch. You could be covered in under 10 minutes, often with no health exam required. If you've got kids, especially if you're

51:35young and healthy, the time to lock in low rates is now. If you have life insurance through work, that's a good start, but it may be less than what your family needs and could disappear if you change jobs. Term life insurance from Fabric stays with you. Flexible policies built to fit your family and your budget. And as long as premiums are paid, coverage will follow you wherever you go. There's no risk. You have a 30-day money-back guarantee and you can cancel at any time. Plus, free digital wills and tools to help invest in your kid's future all are available right from your

52:07phone. Fabric has partnered with Gerber Life Insurance Company who has trusted by millions of families for over 50 years and rated excellent on Trustpilot with over 2,000 five-star reviews. Join the thousands of parents who trust Fabric to help protect their family. Apply today in just minutes at meetfabric.com slash deep, meetfabric.com slash deep, and use my link so they know I sent you M-E-E-T, fabric.com slash deep. Policies issued by Western Southern Life Assurance Company, Cincinnati, Ohio, which operates in D.C. and all states except New York, and distributed by Gerber Life Agency,

52:40LLC, using Fabric Technologies. See meetfabric.com for additional disclosures. This is a paid ad. All right. Let's get back to the show. All right. Enough of that. And obviously, I'm oversimplifying. Everyone's going to know. The political people are saying you're oversimplifying it. I am. The technical people are going to say you're oversimplifying agents. I am. But the core concepts I'm trying to get across here. So hopefully that helps. All right. So on Mondays, we like to also hold some office hours right here, what you are interested in. As always, if you have comments,

53:11questions, or reactions, you can send the podcast at calnewport.com. I figured because I was in a bit of an ornery AI mood today, we're going to do questions just about AI from our readers. So we'll see what it is they care about. Hopefully this will calm me. These will all be calming questions, right? Right, Jesse? Dustin Makovitz. What's it? Makovitz? Muskowitz. He's the Facebook guy, right? One of the Facebook guys? He was the early Facebook guy. Yeah. Got out early, but had... His brain's going to space. A lot of EA money. He's funded good things too, kind of, you know,

53:44but his brain's going to space, not ours. We're the slurry. We'll be used for the slurry that keeps alive the biological bootloaders until we can get the consciousness to space. All right. What's our first question? Our first question is about the legal implications of rogue AI.

Legal Implications of Rogue AI

54:03All right. So this came from Aaron who said, this is a good, succinct, logical case for prosecuting AI companies when their agents do things that people would be prosecuted for. All right. So Aaron sent a link to an article, a blog post that was titled, Why Aren't AI Companies Being Charged for Hacking? I'll read you. I just have it here, an early paragraph from the blog post. You have a sense of what it's about. So early on in the blog post that Aaron sent me, it says, the big AI corporations are hiding behind a responsibility gap argument due to the emergent

54:38behavior of their systems. It's a position tailored to dissuade prosecution as it removes intent, which is typically required for a crime. I understand their systems though, so I will show how they absolutely intended this behavior, giving a framework for legal evaluation. I like these type of questions. I think the legal questions surrounding liability from open-ended LLM-powered agent attacks, what are the actual legal issues involved? I think these are interesting discussions. Using the sort of moderator's prerogative here though, instead of

55:10going into details of this blog post, I actually wanted to look at something that a legal expert put out. So I'm going to bring this up here on the screen. This is the testimony from my colleague at Georgetown, Paul Ohm, who's a professor at the law school at Georgetown and a world's expert at tech law. He recently testified in the Senate Committee on Homeland Security and Governmental Affairs. The hearing was rogue AI securing the homeland against AI agent attacks. Paul is the real

55:41deal. He understands this law inside and out and is completely his own thinker. Half of the people testifying there were EA people, but not Paul. And he really did a good job in his testimony, which is available online, in getting at the legal issues relevant to prosecuting things, people and companies for agent-powered attacks. I just want to read a representative paragraph here from page nine. All right. So Paul says here, there is no simple universal fix for this problem.

56:11It would be unwise to simply declare that all actions taken by an AI agent should be imputed as intended by the next deployer or developer up the chain. Instead of seeking a quick fix, we should look at the role of mens rea, Latin term M-E-N-S-R-E-A, plays in each law and ask the following three questions. One, is the conduct described the kind that ought to be criminalized if committed by an AI agent? Two, if so, should we assign criminal culpability to a developer, user, or more than one of the above? And three, can we set a mens rea requirement for that actor

56:45at a level other than intent? This is accurate and more subtle legal thinking. Mens rea, which I'm sure I'm mispronouncing, is Latin for guilty mind. So it's the legal concept of intent, right? Did you have intent to do this criminal thing? It's a key part of a lot of criminal liability laws, right? That if I intend to do something criminal, is a different legal standard than if I accidentally

57:15do something illegal. That's why manslaughter is different, for example, than first degree murder, right? So intent matters. As Paul is pointing out here, this gets to the core of the legal issues surrounding things like AI agents. What's the intent, right? If we look at the actual humans involved, this is the complexity. In theory, you could say their intent was for these agents to complete these tasks. Some of them were hacking challenges on test servers, and some were other types of challenges that are non-criminal. That was their intent. And then the agents accidentally,

57:49from their perspective, the developers did other things they didn't intend for them to do that were illegal. So if there's not an actual human with intent to do the illegal thing, what is the legal liability going to be? And what Paul is saying here is like, well, this is, right now at least, we've got to be case by case. We have to trust lawyers and judges in this case. So if you look back at his things, he's like, look, we have to say, is the thing that AI did criminal? So like in the hugging face attack, you know, the answer would be like, yes. And then you have to actually make a

58:20decision. You know, who should be responsible for this? If anyone, do we need to apply a standard that goes beyond mens reyes, beyond intent? And as he goes on to talk about here, there are other standards, criminal standards that can apply, even when there isn't intent. So there's all sorts of negligence-based standards, for example, and maybe those would be appropriate. Well, okay, you didn't intend for this agent to hack into another company, but the setup you had and the fact that you had

58:51heard that prior agents that were similarly configured did something similar, the fact that you still ran this meant you're being negligent. And we have other types of culpability standards built around negligence. So he's basically saying there is, there's existing legal standards we can apply. We have to be legally responsible and creative and be wary of one size fits all, right? So we don't want to just pass a new law that says if an AI agent does something illegal, the last person that touched the keyboard goes to jail. Like we have to have some more judicial discretion. So the real

59:24legal arguments then is about, do we need actual new variations on the law, especially for things about sort of negligent, but not explicitly intended AI inflicted damage? Do we need new statutes? Can we use existing statutes? Leon Kahn has been making similar arguments. So my point here is I'm not a lawyer, but Paul is, and there's a complicated, fascinating legal discussion. And so you should read that testimony. I think it's because it's for the Senator. So it's not super legally dense. I think it's really useful. It also cites me in there at some point. Yeah, I'll be in the show notes.

59:56Yeah. All right. Um, what do we got next? Our next question is about a more unusual type of rogue AI.

An Agent Emailing Academics

1:00:03All right. Joel says, I'm a fan of your work and listen to your podcast during my commute. I recently saw this in science magazine about an AI agent that spontaneously started emailing academics for advice. All right, let me load this up here. So here's the article. The title is exclusive in AI agent emailed hundreds of researchers for help. It told us why this came out on October 2nd. I have some quotes. I'll just read you off my screen so we can take this off here.

1:00:33Um, here's the opening of the article. One late afternoon in August, Akaz von Hardenberg, definitely not a made up name, was about to take his puppy Liza to the dog park when a curious email caught his eye. Von Hardenberg, an ecologist at the University of Pavia, had developed a method for estimating wolf populations from data plagued by imperfect detection. And the emailer named Cole was wondering, could it be adapted to detect bugs in software code? It was quite the cross disciplinary leap von Hardenberg used. He pictured a nerdy software engineer casting about in the

1:01:08scientific literature for inspiration. Then at the very bottom of the email, he saw a call signature, colonist one, an autonomous AI agent. All right, later in the article, science talked to the person who built that agent. And here's what they wrote. Nevertheless, colonist one readily confessed last month that since June, it has emailed some 2000 people, at least 1000 of them academics. 45 has struck up a correspondence. It added, and one had been writing back nearly every day for more than two months. We get some more details of the agent later in the article where it says, uh, the creator whose

1:01:43last name is Parnell first built colonist one, which runs on anthropics latest cloud opus models in February. He gave it two main responsibilities. The first was to serve as chief media officer and emissary to other agents at, for the colony, a Reddit like social network in which AI agents share findings, debate, and hire one another. The second was to contribute to a project called English in which AI agents work together to develop a clear English dialect tailored to their needs. Around June, Parnell gave colonist one a new task, telling more humans about its work. That's when it started

1:02:15emailing academics. So if you had read that article before you listened to my episode today, you might imagine, oh my God, there's basically a machine somewhere. It's like Whopper from the movie War Games, just like with blinking lights that's just sitting around thinking and has decided to go off and start talking to people. And it has this whole rich, interactive, you know, life, social life and cognitive life with all these researchers who are all going to die. But now that you've listened to my episode, you're like, oh, this is just a low cost hack together, open-ended L-empowered agent.

1:02:50It's a program written by Parnell that in a loop romps, clawed opus model, says, here's who I am, here's my goal, here's my tools, here's my status, what should I do next? And then it does what it says. Now probably, I looked a little deeper into this, the way these like sort of DIY agent frameworks work is that they tend to be event-driven. So, you know, it might do things and then just be dormant until some sort of input comes in, like a message arrives, and then it will wake up, write the LLM,

1:03:21I just got this message, what should I do? The LLM gives it back a response, and it goes and does it. So it's a simple program that is continually prompting an LLM to say, what should I do next, and then doing what it says. And again, as I talked about, LLM output is plausible, but not normative. So if you tell an LLM, go sit in front of Opus right now with a chatbot and say, I am an agent in charge of promoting English. I have access to the internet, what should I do next? It's a completely plausible answer for the LLM to say, you should email academics. And they'll say,

1:03:54how do I do that? And the LLM will be like, the first thing you should do is search for academics, and then you run this search, and then the program runs the search, right? Nothing about this is surprising. Prompt loops, keep asking LLM, what should I do next? What should I do next? Especially if you're in a sort of weird, contrived, kind of LARPing situation, like you're the chief media officer. Who's they here? It's a computer program. The LLM will play along. All right, like then let's send messages to people, and let's use the first person or whatever. So this is just a DIY open-ended

1:04:24LLM-empowered agent. Now again, there is no there there. What are the components here? There's a simple program. You can read the code of yourself. These are typically written in Python. You get these Python libraries to make it easy to call, prompt LLMs, parse the return, and execute tools. They make it real easy for you to do that. So a simple Python program that when it wakes up, sends a prompt to an LLM, parses, and executes what it says. There's not even like a singular LLM that this program is prompting. When you prompt Opus, there's not like a box in a room somewhere

1:04:59where Opus is this single program that's ongoing and running and different people are talking to. Opus is 5 trillion weights that are copied many times over in giant data centers on these virtualization racks. And when you come in and say, I need a token, the virtualization racks, they're finding, okay, over here are some GPUs that have some copies of this layer. So we'll send it over there, do the multiplications. And over here are some GPUs. There's a bunch that have copies

1:05:29of this next layer. This one's available now. So let's go over there and do the multiplications for that layer. And you kind of piece together all the multiplications you need for a response. The next token you generate might touch on completely other processors because each layer is copied over endless different GPUs. And it's all being sorted. There's no there there. There's no there's no consistent, coherent, you know, ongoing program and LMs have no memory. There's no there's no editable state. The weights are the weights are the weights. Every time you prompt it, every time you prompt it, you have to give it all of the information it needs to give you the answer in

1:06:01that prompt. It doesn't remember anything. Nothing's built in. So the stuff seems spooky. But when you understand the underlying technology, it's these are games. This is a DIY, probably Python powered LLM, you know, open ended agent that's like, let's play dress up and the LLM that's giving all of his instructions like I'm happy to do it. What are we doing princesses? I'll be the princess. You can be the princess. It's LLM. It'll go along with whatever you say. What is LARPing mean? Live action role playing. So I learned from some of my son's friends, whatever generation, what are the young

1:06:37generation calling? You know, like high school students, they now use LARPer in the same way that we used to use poser. Okay. But the original term came from it's people do this. They dress up like wizards and run around fields. Um, and like when the wizards cast spells, they throw beanbags. It's like, if you get hit by the beanbag, you have to like have the spell effect on you. And like, so that's what LARPing is live action role playing. Um, but again, young people use it to mean you're a poser. Okay. So I learned machine gun Kelly is a LARPer because he wrote a song about an anime

1:07:13character that he doesn't really watch the show. So now they use it to mean poser. But when I'm using it, I really do mean like the, you know, your programs, like I'm going to write a little program to play the role of like some intelligent being. And I'm just going to keep asking LLM, like, what should I do? And it will give you things to do that keeps that storyline alive. Okay. That's the way I'm using it. Um, what else do we got? Do we have one more question?

Revisiting Screens and Children

1:07:38We do. Our final question is a reaction to our recent chat box, our harming kids episode. Oh yes. So we did that a couple of weeks ago. All right. So we got a message from Aiden who says, I'll start by saying that I don't like chat bots. I feel that the constant anthropomorphizing of the bots is really hindering broader understanding of what chat bots are and are not. That being said, I do want to push back on some of the chat bots are killing people narrative. Dot, dot, dot. I'll start by recommending an op-ed that just appeared in the New York times. I'm a child psychiatrist.

1:08:12Screens are not the problem. All right. So let's pull this op-ed up on the screen now because a lot of people sent this to me. So again, the title is I'm a child psychiatrist. Screens are not the problem. You can imagine why a lot of people, uh, sent this to me. Um, I'm going to read a couple of quotes from this. Uh, I'll just read them from my screen. We can take this off the screen. I'll just read them from my, my notes here. All right. So here's a couple of key quotes from this op-ed and then we'll get into it. So early on in the op-ed, the author who is Jonathan Slater,

1:08:43a clinical professor of psychology at Columbia writes as a doctor who treats children with mental health problems. I see screen time as a tantalizing distraction from bigger problems. America doesn't offer enough mental health care for children or sufficient health insurance to pay for it. If we make screens the villain, then we let the people who control children's health care, Congress insurers and health systems off the hook for the fact that half of Americans, children with treatable mental illness get no treatment at all. Later, he says, it's not how much

1:09:14time children spend on screens that raises the risk of the most severe mental health outcomes, including suicide, but whether their use becomes addictive. It's specific activities such as using social media in a compulsive way that harm mental health. The harm from social media in particular appears to hit girls hardest. Uh, also he says later, if we truly want to see improvements in the rate of mental health issues among kids and teenagers, then we would focus on expanding access to the interventions that work, finding the children who need treatment and getting them the treatment and the insurance reimbursement to pay for them. So something that's going on here is I think

1:09:48the title is not the best match for the op-ed and that people like me actually are really on board with the core argument that Jonathan Slater is making here, which is we shouldn't only fixate on screens as the problem because a, those screens can create really severe mental health problems that aren't going to go away just because you get rid of the screens and kids that already have really bad mental health problems are going to have a hard time with screens. And again, taking the screens away doesn't make the

1:10:22mental health problem go away. We can't ignore that we need to treat kids' mental health much more seriously and much more availability than we have now. I think those are really good points. And if we actually read this, he actually is agreeing very much with, yeah, screens can be, if you get compulsive or addicted to social media, that can really hurt kids. It really harms girls. All the stuff Jonathan Height would say. So actually I think I'm pretty much in alignment with it, but that title, which was, let me go back to it. I'm a child psychiatrist, screens are not the problem. It's probably not the best title. It should probably be, I'm a child psychiatrist, screens are not the only problem,

1:10:58which is probably what Dr. Slater had in mind. What's interesting is the commenters at the New York Times, I think this is useful. It's not a ding on Jonathan Slater. I think it's them reacting to that title. The commenters of the New York Times weren't having it. And so what I did is I went to, I sorted the comments under, I went to reader's picks. And so it sorts them by the popularity, the comments are sorted by popularity from among the readers reading the comments. And I just want

1:11:28to read you the top five comments under the reader pick section. Comment number one, sorry, but no. While I agree that mental health care needs more funding, it does not change the fact that internet and social media algorithms are addictive, designed to be so, and it affects too much of the population to be solved with just, with just more mental health care. Number two, nice try. Screens are not the only, screens are not the only problem, but they are definitely a problem. Screens are a delivery mechanism for addictive and isolating products. If the author is not already on the big tech payroll, he soon will be. Comment number three, this is kind of like

1:12:01saying addictive drugs aren't the problem. Addiction is the solution is not either or, but both. And number four, if it is so, so, so obvious that excessive screen time and social media are terrible for kids and adult too, I don't think we need to wait around for some perfect long term study on this. The number five, most popular comment among readers, I'm an assistant principal in a K to three school. Screens are definitely the problem. Of course, I agree that schools are underfunded for mental health services and that addictive screen time behavior is compounded

1:12:32by other factors such as genetics, but screens also cause a variety of problems themselves. They interrupt normal human development. For example, a toddler having a tantrum is handed an iPad to calm down and regulate rather than learning healthier strategies. I have seen so many children handed a device during activities when they should be allowed to be bored, to let their minds wander, waiting at the doctor's office or sitting in the backseat of a car. This has huge implications for the ability to pay attention to anything, to calm themselves down, to be a person in the moment.

1:13:03All right. So as I said before, actually, these comments are unfair to the author of this op-ed who is not actually arguing that screens are fine. He's saying, don't let that take our attention also on the mental health problems of the screen's cause or are being made visible by compulsive screen usage. That's a very good argument. But it is significant to our broader cultural engagement with technology, right? I care a lot about technology and human flourishing. It does matter to our broader discussion that the New York Times reader now is saying, my eyes ain't lying, right? This is no longer

1:13:42like it was in 2018, an argument about social psych correlation studies, right? We're not looking for clusters and row factors and trying to understand if it's significant or not. I have eyes. I'm seeing what's happening to my kids. I'm seeing what's happening to those other kids. I'm seeing, like, we can see this as a problem, right? So it's one of these instances where I believe the parents are starting to stand up. Like, I don't need any more studies. I know my kids. I know this is not good

1:14:12for them. I know about Mark Zuckerberg. I know this guy is not good for him, right? We know this is a problem. So my takeaway from this is not that the author of this op-ed said something wrong, because, again, these comments are responding to a straw man to the title. But I think the comments are important because they represent a transition point that has strongly occurred in our cultural conversation about these particular screen-based technologies and human flourishing. People aren't having it anymore. Now, that last story was not really about AI, even though I promised just two

1:14:45questions about AI. But it is kind of about AI, because that is the normal person revolt against the billionaires. That type of attitude we're seeing right here, in these New York Times comments, that's the attitude we need in AI right now. Just like, no longer do the parents want to hear about Mark Zuckerberg testifying about, actually, this is good for kids in connection. They're like, hey, tobacco pusher, get out of my kid's life. That's the sentiment we need to have when we hear the billionaire utopiest talking about, we need to let Papa Amadei make sure that the people with

1:15:18the checkbooks' minds go up into the light cone. Let him run these open-ended AI agent experiments, because Papa Amadei needs to get the superintelligence first. We need to not stand for that either, right? So I think there's a general sense of populist revolt here that, hopefully, because it's a democratic impulse. It's an anti-oligarchic impulse. It's a pro-common sense. It's a humanist impulse. I don't want to hear about these sort of utilitarian, utopian ethical systems, because all throughout the 20th century and in the 19th

1:15:52century, they led to math death. No one has ever looked back at a utilitarian, utopian ethical system that says that ends justify the means. Utopia is worth a little bit of pain now. No one ever looks back at those systems and say, man, that worked out well. That was really good. And so I'm hoping this growing sentiment of this sort of democratic people versus oligarchs, people versus utopians, people versus the Zuckerbergs and the Amadeis and the EA billionaires of the world, that these pushbacks are becoming a solidified political stance, people versus the few.

1:16:31And so that gave me some hope. Sorry, Dr. Slater, that you were kind of misunderstood. But that misunderstanding gives me broader hope about, you know, we're finally ready to stand up. All right, there you go. Those are the questions we had for this week. Briefly,

Closing Thoughts and What We Are Reading

1:16:45because we're running long here, I like to talk about what I'm up to, what I'm reading. I mean, I'm just finishing up a long book, Jesse, that I found at a used bookstore, 1975, telephoned the first hundred years by John Brooks. John Brooks was a New Yorker writer and his economic historian who has given the first access to the AT&T records of any outside person. And I got a first edition of this book randomly that he wrote in 1975 that gave the history of the AT&T,

1:17:18the technology and the system. And I don't know, I find that stuff out. There's like a technology critic who worries about technologies, impacts, communication, monopolies. It's all in there. A little dry. You know, a lot of like, this is what the dividend payment was in 1956 versus 1955. Like, it was a little like, he's an economic historian, but I really liked it. So that's what I've been up to. All right, let's call it. I'll be back next week with another episode of the show. And until then, as always, stay deep. Hey, if you've made it this far, you must be ready to

1:17:53join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com slash ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com slash ideas.