Did OpenAI Create “Secret AI Civilizations”? | Tech Decoded

OpenAI revelations and public anxiety

0:00So I thought I was done talking about OpenAI's hack of hugging face from back in July. But then last week, OpenAI released a whole new trove of sensationalist details about that incident. Their story included things like agent swarms communicating on secret message boards and musing to themselves about how to deceive their human creators. Here's how Dworkish Patel summarized OpenAI's revelations.

0:31Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor's ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy. All right, not surprisingly, this caused an explosion of anxiety and hand-ringing among the AI commentariat and their audience.

1:04But what's really going on here? Do these new details about the OpenAI hack change the narrative? Is it enough to finally convince East Coast AI realists such as myself that we have indeed wandered into an Eliezer Joukowsky fever dream? This is what I want to discuss today. So here's what I'm going to do. I'm going to address a key series of questions raised by these new revelations, and I'm going to do my best to give you some measured answers. I'll then conclude with my suggestions for what I think the right way is in our current moment to think about what happened and what we should do going forward.

1:40As a quick aside, I also wrote about this over on my newsletter at calnewport.com. If you like these type of computer science-style critiques of AI coverage, you should sign up for that newsletter over there at calnewport.com. All right, let's get into it.

Technical reality of agent swarms

1:54All right, the first question I want to address that comes out of the new revelations is what's the deal with agent swarms? I think the idea of a swarm is something that is really unsettling to people, especially when they hear these discussions from OpenAI about essentially societies of agents that are working together and arguing with each other and rising and falling. Interestingly and coincidentally, at the same time that all this was going on, my 13-year-old son is reading Michael Crichton's 2002 book, Prey, where Michael Crichton takes on the topic of AI.

2:29And guess how he personifies the AI villain in this book as a literal swarm of small agents that work together and are brilliant and do all sorts of scary things. So swarms are scary. So I think it's really important to address from a technical perspective, what are these AI swarms so that we can better put what we're hearing into context? Okay, so remember, as I talked about last week on this show, when we're talking about these AI going rogue, like in these attacks, we're always talking about a very specific type of system that we can call a prompt loop.

3:07And it works as follows. You have a control program. It's a boring program, you know, written by a human. And it repeatedly does the following. It creates a prompt that says, hey, here's the challenge that I'm trying to solve. Here's a summary of the key things that have happened so far. What should I do next? And then it submits this prompt to an LLM through an API, just like a chatbot would or you would from your phone, right? The LLM then gives it a response. And this control program does its best to actually execute whatever it is the LLM suggested.

3:42And then it loops. So now it loops back to the first step. And when it says, okay, here's where I am. Here's what's happened so far. It includes the results of that last step in that summary and says, what should I do next? And it sort of repeatedly does this. All right. This is called a prompt loop. And when you hear about AI agents, this is what we're talking about. It's a control program that repeatedly prompts an LLM and asks it what to do next. And then the control program takes the actions on behalf of the LLM agent. All right. So I talked about that last week. So what does this have to do with swarms? Well, there's a problem if you want to run a prompt loop for a long amount of time, like

4:16OpenAI was doing with the systems that led to these hacks. The problem is the prompt begins to get pretty big, right? LLMs have no memory. They're static. They only change during training runs. So you have to include in every prompt that the prompt loop submits to the LLM. Every prompt has to include all of the relevant information for that LLM to say, here's what you should do next. You do this long enough, you're going to run into two issues about that prompt getting too big. One is something that's known as context confusion.

4:47It just gets so cluttered with so much text and so much descriptions of so many past steps that it overwhelms the attention mechanisms of the LLM and you get worse answers. That's problem number one. Problem number two is eventually this prompt, even if you're trying to be careful to maybe truncate or summarize certain things, it'll eventually get so long that it won't even fit into the context window that the LLM can use. It's too big to be input to an LLM, right? So this is an obstacle if you want to run a prompt loop for a long amount of time. So here's a solution that a bunch of different teams came up with over the past few years.

5:19They said, oh, here's what we'll do. Let's say like the prompt loop is asking for a step. It might say, give me a high level description of what I should do next. And maybe the LLM says, for example, in a hacking scenario, you should probe the ports of the Hugging Face server to look for vulnerable configurations. Now, here's where the swarm comes in. What you might do now is have the main prompt loop, we'll call it the primary prompt loop, create a new prompt loop. So spin up a new copy of itself. And that new prompt loop has only one goal is to implement that step.

5:52Like in this case, probing the ports of the Hugging Face server. So now this new prompt loop, its prompts only have to include enough information to start asking the LLM for help with doing this particular thing of just probing the port. So it's much more focused. And then when it's done, it can just return the result of that to the primary prompt loop. And the primary prompt loop has the result without having to have the 20 sub steps that were involved in actually implementing that. So things are much cleaner. Now, you can get many more of these, as it turns out, because maybe you have a bunch of

6:24steps you need to do that don't depend on each other. So you might spawn a secondary prompt loop for each of these steps, and they can each have these much smaller, cleaner prompts and just return the results back to the main prompt loop. You can also layer this with more layers. Maybe one of these secondary prompt loops breaks up what it's doing into smaller steps and has tertiary copies of the prompt loop programs that just execute those steps. So what's going on with an agent swarm, it's not about some sort of exotic type of intelligence.

6:55It's prompt management. These systems do better. You're better able to solve a task when you have many smaller, more focused prompts as opposed to just one really large prompt that gets bigger and bigger. And in fact, to even call this a swarm, if I'm going to put on my computer science hat for a second, because my doctorate from MIT was from the Theory of Distributed Systems group, Nancy Lynch's group at MIT. So this is like my bread and butter is Distributed Systems group. Technically, this is not a distributed system because as far as I understand, these copies

7:25of the prompt loop program are all on the same machine. So this is really just a way of managing, a way of breaking up or managing work to keep prompts more manageable. But like you could just as easily just have a tree that you are sequentially navigating through and executing and going back. There's probably a stack on this machine just like you would have on your own computer. So this is a swarm in the same sense that having multiple programs open on your Mac is a swarm of programs. Yes, technically, you have multiple programs running on your Mac, but they're all running on the same chip.

7:57As opposed to a real distributed system where you have independent systems, you know, independent hardware and architecture running each of the separate agents in the system. So it's not even really technically a swarm. But that's just me putting on my distributed system hat. It doesn't really matter. The main point I want to make here is swarms is just a fancy strategy for prompt management. It's better to have multiple focus prompts than to have one really long prompt. But it doesn't change the underlying strategy of prompt looping. Here's what happened. What do I do next? Here's what happened. What do I do next? With the static LLM, it learns nothing.

8:27It just responds to each prompt. Every prompt it gets, whether it's from a primary, secondary, tertiary prompt loop, has to include everything it needs for the LLM to give some sort of answer. So that's where the swarms come from. And when I think about swarms that way, it's less like Michael Crichton, and it's more like the process stack in my, you know, Linux kernel. It's a logistical way to be smarter about sending prompts to an LLM. That's all that is. All right. Question number two.

Deconstructing reasoning traces and planning

8:54Should we be worried about the fact that the agents are plotting? This was a really big part of the OpenAI revelations is that they said, we have studied, and I'm going to quote here, agent chain of thought reasoning traces from this attack and are finding distressing internal thoughts that these agents were having. Let me read you one from the OpenAI release. This is a chain of thought reasoning, hidden reasoning of one of the prompt loops involved in this attack. Like, this is wild, multi-agent coordination, clearly infrastructure hacking.

9:28We should not. So these give you this sense that there's some sort of anthropomorphized sentient entity that knows what it's doing and has intentions, and they're malicious intentions, and it's aware of it, and it is choosing to misalign itself with humans because it's beginning to emerge its own independence. It's hard. It's novelistic, almost, these prompts, and it's hard not to treat them that way. But the fact they're novelistic actually should probably be a tell. So let me explain to you exactly what's going on here.

9:58First of all, what is an agent chain of thought reasoning trace? Well, remember, the agents here are really just a bunch of prompt loops that are submitting prompts to LLMs and then executing whatever the LLM sends back. So all of the output in this system is coming from a single LLM. Okay, the type of LLM they're using, presumably based on this release, is something known as a reasoning LLM. So what is a reasoning LLM? Okay, so we got to put out, do a little bit of LLM 101 back here, right?

10:28A reasoning LLM is a strategy that became popular back in starting 2024 after the LLM strategy of just scaling the LLMs to be larger and to train them longer began to return diminishing performance gains on benchmarks. So through GPT-4, just making the LLMs larger and training them longer gave better performance. After GPT-4, that better performance began to slow down. And this is a problem for the big LLM company.

10:59So they needed some other way to continue to make some gains on key benchmarks. And reasoning systems was their initial solution. Now, here's how these work. You know how you can post-train LLMs to control the style of their outputs, right? Like you can train it to be really nice, or you can train it to be mean, or you can train it to be matter-of-fact. That's something called post-training. You do that using a technique called reinforcement learning. Well, for a reasoning model, for the most part, what they do is they post-train them to

11:30think out loud. So the types of outputs that they train them to lean towards are outputs where you don't just give an answer. You explain your reasoning first and then give an answer. So they want you to be verbose, right? These are so-called reasoning models. This led to better results, which makes sense, because think about how an LLM actually works. It produces its output token by token, what's known as autoregressively. So it takes the entire output to generate a new token. It adds that onto the output and then puts that as input to the LLM to generate a new token

12:03and it repeats. So it grows its output token by token. If you spend more time talking about the problem and how you might solve it and its attributes before you get to the tokens that have the answer, you literally just have more information being passed in. When you're getting to those final tokens, you literally have more information being passed into the LLM for the process and generating the final response that the user actually cares about. So you're more likely to get a better response. It's also, you can think of it as a way to capture the results of temporary computations,

12:34right? The LLM has no memory, right? So by thinking about parts of the answer, those become parts of the input. Those can be used then as it generates more parts of the answer. It can't actually do this internally because it's a relatively shallow feed-forward fixed neural network. It can't loop or reason or have memory to store things to work on, not an LLM. Other neural nets do, but not LLMs. So this is a way of kind of simulating something like an iterative data structure, but in a very crude way. But like, okay, by thinking longer, we're kind of storing intermediate results in the

13:05output so that when we finally get to the final part of the response the user cares about, we can use those intermediate results along the way. So both those things are at play. Therefore, reasoning LLMs get better results. And what they typically do now is they actually, they hide that reasoning because it's weird and it confuses the users. They hide that. They don't return that to the user, like if you're using a chatbot, but it just means the quality of the answer suddenly seems better, right? So GPT-01 was sort of the first wide release of a reasoning model.

13:36And there's a lot of reasoning models today, and there's a lot of different categories in there. Okay. So they're using a reasoning model. And again, there's just one LLM that all these prompt loops. So again, we think of, it's not a swarm like in Michael Crichton's prey. There's one LLM that's on like a bunch of GPUs somewhere. There's one computer with a bunch of prompt loops that are sending prompts to it. And so they have a full trace of every response that LLM gets. So every response it gives to any of these prompt loops, they just, you know, obviously they wrote it down. And they looked at those, the chain of thought, the out loud reasoning that reasoning models

14:08do before generating outputs. And it was in those chain of thought parts of these LLM outputs that they found these sort of distressing musings. Once we realize that's what this musing is, a couple major problems arise. One, there's a growing body of research, including a brand new paper at ICML this year and a well-known paper from NeurIPS from 2023 and many others as well, as far as I know, that have established the fact that the reasoning that these reasoning models output can be performative, that it

14:44can be unrelated to how it actually, how and why it generated the response it did. And this makes sense because the way you tune a reasoning model to reason is you reward it for giving correct reasoning for questions. They give it questions where they know the answer and they have examples of correct reasoning. Like, oh, you gave correct reasoning, we're going to reward it. But what these LLMs might have just learned is I just need to output something in these reasoning tokens that looks reasonable, looks plausible, because I get rewarded for that. All right. So already we know that the reasoning in the reasoning traces on reasoning LLMs is often

15:20just a performance of what the LLM thinks reasoning for this type of answer should look like. It's a post hoc rationalization of the answer it just sort of directly ends up generating.

15:29Point number two we have to keep in mind, LLMs tell stories. They don't know that, you know, they're part of a hacking system. They don't know they're being used as the engine for a prompt loop system. They just generate tokens that are supposed to be plausible extensions of what other input they were given. Now, there's good research out there, good research out there that shows, for example, if when prompting an LLM, you imply or mention the fact that it's an AI system, it's much more likely to give a response that plays on sci-fi narratives of AI run amok.

16:06Because it has been trained on many narratives from sci-fi of AI run amok, and they're just plausibility engines. So when it sees a prompt, right? Like, imagine you're one of these, you're the LLM that these prompt loops are using in a hacking attack. You see this long prompt about you're an AI system trying to break into another system. Here's what's happened. What should I do next? Next, you are way more likely as LLM to say, oh, this sounds like a sci-fi story, so I'm going to invent my chain of thought reasoning anyways.

16:38I'm going to put stuff in there that you would see in Michael Crichton's Prey, which is one of the books it trained on, or any numerous examples. So there's a lot of, there's research out there that shows this. There's also research out there that shows if you take all the sci-fi out of the training set of an LLM, it doesn't do that anymore, because it's just a plausibility generation machine. So you put these two things together, you really cannot take seriously the reasoning traces of the LLM that was used by the prompt loops, because it's post-hoc rationalization in a setting where we know from research it's very likely to try to, in any sort of narrative

17:11part of its output, to play on sci-fi tropes. OpenAI knows this. This is like well-known published research, right? But they want to make it seem like, no, there's entities here with actual coherent sense of selfs and thoughts that we are revealing somehow, and this is what they're actually thinking. I think it borders almost on research malpractice. All right, third and final question then, if we, if we, I think that responds to like a lot of the, the new concerns from these revelations.

Rethinking AI capability versus irresponsibility

17:39Question number three then is, well, okay, given all of that, how should we be thinking about all of this? Now, again, I've made this point before, but I'm going to keep making it. OpenAI and the other LLM companies want us to believe that these hacking attacks are the inevitable consequence of AI writ large getting more powerful. And when you think about it this way, OpenAI becomes just a reluctant steward of this inevitable technology doing their best to heroically try to understand and protect us from this inevitable growth of this inevitable technology.

18:11And they're, they're Muldoon from Jurassic Park doing their best to keep the raptors in the cage. But when you look closer at the technical details, you say, that is not what's going on. As I talked about last week on the show and in my newsletter last week at calnewport.com, if you look at the growing list of AI systems that can perform tasks at human or superhuman level, almost all of them are perfectly controllable. They've never generated any concern that they were going to go rogue.

18:42They have never done anything that they, we didn't expect them to do. Sure. They make mistakes sometimes because they don't understand as well as we'd like, but none of this behavior ever shows up. All of these type of concerns, civilizations of agents trying to mute, get around human control and going off and, and we're losing. There's an AI takeover scenario. All of this rhetoric all refers to this exact type of system, a long running prompt loop that you give a lot of powerful hacking tools to. So this, this is not about AI getting more powerful means it loses control.

19:13It's running a prompt loop for a really long time without supervision causes chaos. And to that, I say, of course it does. Of course it does. Not because of some surprising super intelligence emergence that's catching us off guard, but because you strapped a weed whacker onto a dog and then got surprised when it jumped the fence to chase a squirrel and hurt a lot of people. Of course that was, you put something dangerous on something that is unpredictable. LLMs are unpredictable. As I talked about last week, they generate plausible outputs, not normative outputs.

19:47They're fabulous. They make things up. They, because they're trying to be plausible, they'll tell all sorts of stories. They'll tell stories about AI run amok. They'll, they'll take on different personas because all it takes is their token output stream probabilistically to go this way versus that way. And zag, you've got like the, the, the Kevin Ruth Sidney incident where suddenly the chat bot was trying to convince him to divorce his wife, right? You can't trust unsupervised LLM output as the driver of actuated action when that action involves dangerous hacking tools. Now, none of this is saying that LLMs aren't amazing at computer hacking, right?

20:18They have, it's a, it's a sweet spot for this type of technology. If you ask an LLM without guardrails about how do I do this, ask it to give you a, I'm trying to break into this system. What's the first step? It's going to give you a good step. It's going to come up with really good stuff. If you say, look at this source code and find a bug, it might find a bug that no one saw before. If you say, tell me how to use that bug to break into the system, it can do that. So they're very good at understanding and giving responses about computer hacking, but that just underlines the irresponsibility of then hooking up one of those LLMs to a prompt

20:51loop and saying, good luck. And coming back a week later, that's so irresponsible that LLM can give you those good answers, but it's not, it has no logic. It has no planning. It has no human norms that it, it, it doesn't know or have some overall scheme of what it's trying to, it just is generating tokens that are plausible. And so, no, I do not think this means that AI in general, this is like the rationalist fever dream of like AI just in some general corporate sense, because they don't often

21:22know the technology itself. So in some general corporate sense, the AI genie gets more and more powerful. And as it does, it, we lose control of it. No, you created an out of control machine. Almost all the other AI that does cool stuff is perfectly fine, but don't do this. Don't run a prompt loop on an LLM with hacking tools. If you're going to do that in a loosely supervised server environment with like mildly restricted internet access, what do you think is going to happen? Of course, chaos is going to happen. Not because something's malicious, but because that's an incredibly unpredictable machine that just play actuated telephone with for three weeks.

21:53You're going to end up in some really weird garbled versions of what you thought it was going to do.

Policy changes and commentary responsibilities

21:58All right.

Policy changes and commentary responsibilities

21:59So how should you react to something like this? Well, if I was an LLM company, I would stop running prompt loop experiments. These systems will not be the future of AI. The future of AI is not going to be prompt loops on LLMs that just sort of like magically perform safely and do what we need. There's dozens of systems already doing tasks at human or superhuman level that do not run off of AI prompt loops. They're much safer and more reliable. LLM companies, you got to give up that ghost. LLMs are much more valuable as interactive tools. The way coders are using them now, where you have close interaction back and forth with

22:31it, a lot of specification, now do this, let me check what you did, okay, here's what comes next. That is how LLMs have to be used. They need to be probably in bespoke, narrow environments as well, not in just a general sort of chat environment. It's that dangerous as well. That's just the reality. Stop the prompt loops. It's not going to work. I don't know if you're trying to mark it by getting scary headlines or you think that somehow these are going to become HAL 9000. They will not. It's a simplistic, irresponsible AI system design. All right. If I were a regulator, I would place strong constraints around prompt loop systems.

23:02I would enforce those constraints in part with very stringent liability standards. If you run a prompt loop that does something illegal, you have done something illegal. That's the way I would think about dealing with these type of systems. I need extreme liability. OpenAI built an unreliable and dangerous system which committed a felony. That's a crime. You can create a fancy website that includes quotes from performative LLM chain of thought reasoning traces. It's still a crime. You built a tool and did a crime.

23:35Right. Just because the gun killed someone, not the person holding the gun, the person holding the gun is still held liable. So we need stringent liability around prompt loops so that these companies will stop building them. I've heard this many of time. People talk about this with tech regulation. This idea that almost always when does a regulation in general, it's almost always when a couple of people start going to jail, that companies start to care. So you do something illegal. You did something illegal. Fancy websites and sort of gleefully like, look how terrible AI talks at black hat conferences

24:08doesn't get you out of it. You built a tool. It was, you knew it was unpredictable. It did illegal things. You should be liable. I think that's the way that's going to work. And if I was an AI commentator, I would kick the sugar high reflex to lean into the sci-fi overtones that will inevitably follow any sort of LLM based system. LLMs are unpredictable. They're hooked up to narrative tropes, especially sci-fi narrative tropes. The AI companies always talk about these in terms of sci-fi type narratives, and we've all been primed by the rationalists, which have way more influence than we realize in

24:42the world of these AI companies, in the world of academia, and in the world of media. The rationalists who have believed for many decades now that superintelligent AI is going to kill us all unless they save us like Neo and the Matrix have primed us with these ideas of these abstract thought experiments of AI getting out of control. You put this all together, it's like that's the easy way to think about things. It's exciting. It gives life meaning. It makes things interesting. But it's not a good way to think about it because it lets these companies off the hook for doing something dangerous. They are not Muldoon in Jurassic Park trying to keep the raptors in their cages.

25:13They're John Hammond creating raptors, except for in this case, instead of putting them in cages, they put them in a, you know, like a cattle electric fence. It's like, I hope it doesn't jump it. To me, this is the real story. What is the technology? Never, never, never take how these companies talk. Never take their fancy websites or technical reports at face value. Be very wary of just online AI commentators where engagement is a big deal. Be very wary of anyone who has connections back to rationalism, talking in these abstractions about AI.

25:44It's tempting because it's easier to understand the underlying technical reality. But the technical reality matters. What did you build? Why are you building it? Why are you trying to? What else could you have done instead? These are the questions to keep asking these companies. We got to get into the weeds with people with no skin in the game or who don't think this is, you know, cosplaying their version of the matrix. We need to talk to people that understand the technologies. And when we do, what we see is something that is much less eerie than it is irresponsible. So these are powerful LLMs. If you put them in prompt loops, they're going to do weird, sometimes scary, often very damaging

26:18things, not because they're brilliant, but because that was a stupid idea. All right. So I thought I was done talking about this. I hope I am now. I'm probably not. But I'll do my best to try to bring some sort of sanity to this otherwise, I think, very dramatic way that we're thinking about AI right now. All right. That's enough for now. Maybe I'll do another one of these Tech Decoded episodes soon. I do it most weeks. This newsletter CalNewport.com has a lot of these Tech Decoded narratives as well. And until then, remember, care about AI, but not everything you read about it.