An Insider's Guide to AI Doom

Mapping the AI harm debate

0:00The conversation surrounding AI harms has really heated up in recent weeks, but it's also at the same time become really confusing and hard to follow. There's many different groups involved, and it can be difficult to try to figure out who belongs to what group, what group believes what, how do they feel about each other. I don't know how you can follow the current AI discussions about harms if you do not understand this landscape of different players.

0:30So that's what I want to help you figure out today. This is a short episode I'm going to record where I'm going to lay out the three major categories of people involved in the AI harm debate. I'll tell you who's in each category, what they believe, and how they feel about each other. So if you are interested in the current discussion on this technology, you need to understand how this landscape is actually configured. All right, so let's get into it.

1:00All right, so let me give you two caveats before we get started. One, I'm going to make some mistakes here. I'm basically just winging this episode, and the boundaries between these different groups I'm going to talk about are a little bit porous, and there's some people that fall in between and some people that won't like the characterization. So I apologize in advance, but I believe the landscape I'm about to present to you is more or less correct and will definitely help you better understand the current discourse on AI harms. My second caveat is I'm going to try to keep my personal opinion out of it. That's not the point of this episode.

1:31I really do want to just give you, to the best of my understanding, the current lay of the land in the sort of AI discourse that's happening at the moment. Okay, so let's get into this. I'm going to bring up my tablet here because this gets complicated enough that we need to actually probably see how these different groups connect.

The rise of the doomers

1:50I'm going to start over here by talking about the first of the three major groups that's involved right now in the current AI discourse, and we're going to call these the Doomer group. Maybe I'll pluralize it. Let's call it the Doomers. All right. This is an important group in the current AI harm debate. Who is in this group? Well, I'm going to start by putting rats and EA down here. These two philosophical movements give the intellectual underpinning for the group that I'm calling the Doomers.

2:26So let me give you the quick history on this. You have to go back to Silicon Valley futurist movements in the 90s into the 2000s. In particular, there is a movement called the Extropians that was futurist and very utopian. It believed that technology was advancing at such a rate that it was going to deliver us in the future a utopia, probably a transhumanist utopia, where we merge with the machines in such a way that we could have infinite life or whatever else we think is critical to our utopian visions.

3:01There was two main technological threads that the Extropians wove together. One was nanotechnology, which was just becoming big in the 80s and 90s when a lot of the Extropian movement got started. They were influenced by a thinker named Eric Drexler who wrote about this for both academic and public audiences. So they thought nanomachines would be able to repair our bodies, reverse aging, reverse disease, etc. The other major technological thread that the Extropians held on to was AI. They said maybe we're going to have super intelligent AI that could act like essentially a benevolent digital deity.

3:39So this was going to be the other technological move that was going to deliver us this utopia. All right. So that's the Extropians. They're pretty positive. Then you have a particular Extropian. This was a young man named Eliezer Yudkowsky who dropped out of school after the eighth grade, was reading a lot of science fiction, found his way to the Extropian mailing list, got really into these ideas, was writing a lot for these email lists, eventually got funding to start his own sort of techno-utopian institute. He came to believe as time went on that though super intelligent AI could deliver a utopia, it could also just as easily kill us all, and that it was actually more eschatological than the Extropians realized.

4:25That super intelligent AI was inevitable, but it's not inevitable that it was going to deliver good things. And so he began to write and think quite a bit about what would be required to prevent super intelligent AI from killing us all and to help make sure that people understood this danger before we created this technology. So Yudkowsky's group became known as the Rationalist, right? So that's why I have rats written here under Dumerism.

4:56It's a shorthand for the Rationalist. All right. But I also wrote EA under there. That stands for effective altruist. Now, this is a movement you've also probably heard a lot about recently. This is a little bit complicated, but I'll summarize it really briefly. So the origins of the effective altruist is William McCaskill and Toby Ord, who are two philosophers at Oxford in the 2000-2010s. They were heavily influenced, among others, by the Princeton ethicist Peter Singer to apply logic and rationality to being altruistic.

5:27And so they're trying to be very effective at altruism. So early on, the effective altruists, for example, were really big on rating charities, right? Because they figured you want to give your money to a charity that's going to do the most good per dollar invested. So they worked on projects to help rate charities. They also had another campaign called GiveWell, which said, look, if you have a high-paying job, the best thing you can do for the world is keep that job but give a significant portion of your high income towards really effective philanthropy.

6:02So, like, these are the type of things the effective altruists were thinking about. But pretty soon, they also became, as a general movement, very worried about superintelligent AI. There's lots of different points of influence between the rationalists and the effective altruists. A key one is Nick Bostrom. He was also a philosopher at Oxford, but he was also heavily involved on the extropian mailing list in the 90s into the early 2000s. And he was very convinced, especially by Yudkowsky-style arguments, that superintelligent AI was inevitable and without care was going to kill us all.

6:40And so he was one of several voices in effective altruism that moved the movement away from just thinking about effective giving and towards what was known as a long-termerism view that said, wait, the most altruistic thing you could do is prevent the extinction of humanity. And so he started an institute at Oxford that looked at different existential risks. But over time, superintelligent AI killing us all became the dominant risk they cared about. So we're in a world now where effective altruism is a very large movement.

7:12It's incredibly well-funded, mainly by tech centermillionaires and tech billionaires. It deals with lots of things. It's not only focused on superintelligent AI in the way that rationalism really is, but it's become a dominant theme. William McCaskill gave this well-known speech, I believe it was at Google, where he effectively said, look, if we're going to be super rational, we have to think long-term. And effectively, why worry too much about hundreds of millions of lives right now when we really, if we want the biggest return on our efforts, need to think about preventing human extinction, which would prevent tens of trillions of future lives from ever being born.

7:52So they pulled on this logical thread long enough, and a lot of effective altruists said, preventing superintelligent AI from killing us all, that is what effective altruism should focus. Now, again, not everyone in EA is focusing on that. Dustin Moskowitz, for example, recently gave a huge amount of money to foreign aid after USAID was effectively denuded by the Doge effort. So it's a bigger movement than the rationalist, but superintelligent AI is a big part of their portfolio.

8:22I've heard from a lot of people that in the Bay Area in particular, the rationalists and the effective altruists are sort of indistinguishable. This is why, for example, if you read the really good reporting by Cade Metz, the lead technology reporter for the New York Times, he often puts those two words together. He will say EA slash rationalist circles are saying X because in the Bay Area they become more or less the same thing. Okay, so to summarize, the rationalist and the effective altruist think superintelligent AI is inevitable.

8:54They've been thinking this long before LLMs. They think that if we're not really careful, it will kill us all. But if we're careful and do this just right, it will deliver us a transhuman utopia that is like the ultimate good and therefore this should be a major focus of our altruistic instincts. Okay, so I want to return now to our diagram.

OpenAI and Anthropic's roots

9:18And I'm going to add – this is where things get a little bit more complicated. I'm going to add anthropic and open AI under the Doomer category. Okay, so this is really important. There are a lot of AI companies in the world and in America that have hyperscaled LLMs or are working on big projects. But those two labs in particular, anthropic and open AI, are distinguished by the reality that they come out of the rationalist, effective, altruist intellectual tradition.

9:50They were started in large part to try to fulfill the rationalist, effective, altruist idea that we have to carefully align superintelligent AI to prevent extinction and to instead get us closer to a transhumanist utopia. That's why open AI was founded. You have to remember Elon Musk originally funded open AI because DeepMind, which also came out of these circles, was bought by Google and they said we can't trust a big company to shepherd superintelligent AI the right way.

10:22So we'll start open AI to do that. And then anthropic was breakaway employees from open AI. They broke away two years later after the for-profit subsidiary was declared an open AI. They broke away and said open AI is not sufficiently pursuing the rationalist, effective, altruist goals of steering away from bad, superintelligent AI and towards good, superintelligent AI. Anthropic in particular is just completely incestuously connected with these movements.

10:54I mean, many of their employees, Dario Amadei himself, he used to live in a group house with other really major effective altruists. His sister, who helped start Anthropic Wisdom, married one of them. A lot of the initial funding for Anthropic came from effective altruist individuals like Sam Bankman-Fried and effective altruist investment funds. Anyways, there's a lot of connections, which I don't want to go through now, but I would recommend you look at my New York Times op-ed about this from a few weeks ago or Cade Metz did a really good appearance on The Daily, the New York Times Daily podcast a couple weeks ago where he walked through this history.

11:29Michelle Goldberg did a good column on this history. The Wall Street Journal last weekend had a good column on this history. Look at Adam Becker's book is very good on this. Listen to my podcast with Adam Becker and Ed Zitron where we went through this. So there's a lot of sources out there detailing these connections. But what I want us to focus on here is this idea that Anthropic and OpenAI share the same intellectual taproot as the rationalist and the AI-focused effective altruist. Now, what makes this complicated is that the relationship between these two frontier labs and the rationalist and effective altruist who are not part of those labs is strained right now.

12:10So you have some people from these circles, right, like Eliezer Yachowsky from The Rationalist, who are completely upset, Ananthropic and OpenAI. They say, you guys are moving too fast. You're going to end up with the bad superintelligent AI. They all share the belief that eventually we want a transhumanist utopia delivered by AI. But like Yachowsky would say, you guys are going too fast. We're not ready to do that right. I think you're caring too much about money or winning. And so like Yachowsky is openly hostile towards these two labs.

12:42You have a bunch of effective altruists who are sort of in the middle, who maybe are a little bit on board with Dario Amadei and saying like, yeah, we need to have regulation that will allow Amadei to set the pace because he'll do it right. And we need to prevent competitors from going too fast because they're not going to do it right. And then you have a bunch of effective altruists who are out there lobbying hard to say like, we have to basically shut down these labs. They're doing it wrong. So it's complicated because they all believe the same general things. Superintelligent AI is inevitable.

13:13It will either be a good god or a bad god. They differ, however, in whether they believe we can get the good god right now or whether we're not ready and we need to wait. And so the people at the labs think we can deliver the good god. And a lot of the people not at the labs are saying we're not ready yet. Now, the reason why that's confusing is that these feel like they're opposed parties, right? So when you see Nate Soros and other – and then some effective altruist figures surrounding Bernie Sanders as he talks about legislation to try to prevent superintelligent AI, you would assume that these are from a completely different mindset than the frontier labs that this legislation would apply to.

13:54But they all come from the same doomer mentality. They just differ on where we are on this timeline that they both agree about.

The accelerationist perspective

14:02All right.

The accelerationist perspective

14:03So that's the AI doomers. But that's only one of three parties. So let me draw another one up here. Another key party right now, I'm just going to write down the word – I'm going to abbreviate accelerationist. So the accelerationist – and I don't know. They would call themselves other things. The accelerationist are a group that believes that the doomers are weird and that we need to go as fast as possible because there's a lot of benefits from AI. Why slow that down? And also there's an economic race occurring with China that we want to win.

14:34So who is under the accelerationist rubric? Well, you have a bunch of venture capitalists in here, which makes sense. They want to invest in a lot of AI companies and have them not be impeded and trying to make a return on investment. So you'll have Mark Andreessen as a well-known accelerationist that would be in here. The sort of the host of the All In podcast would probably put themselves in that category as well. They're all investors or venture capitalists who are not happy with what the doomers are doing. You also have, interestingly, other AI firms.

15:07I'll write that down. So Anthropic and OpenAI are the two – along with DeepMind, they're kind of a – they're a subunit of Google that's kind of being reorganized. But Anthropic and OpenAI, these are the two firms working on hyperscaled LLMs that are fully in the doomer philosophy that comes out of rationalism, effective altruism. A lot of the other companies, major AI companies in the U.S. and abroad are not so directly connected to doomer philosophies, and they think the doomers are nuts.

15:38And so now there's kind of a feud between OpenAI and Anthropic and all the other AI companies. This is why Jensen Wong, for example, is coming out and saying, what's with all this sci-fi stuff? This is crazy. Just don't build unsafe things. This is a fairy tale. Well, Mark Zuckerberg a couple weeks ago had a tweet that was really pushing back. I'm paraphrasing here, but he was like, why is Anthropic obsessed with recursive self-improvement? Why don't you take that effort and instead put it towards building useful products, sort of calling out that sort of doomer mindset of we have to race towards superintelligent AI.

16:12The other figure over here that's very important is the White House. So the executive branch is pretty firmly in the accelerationist camp right now as well. So President Trump's rhetoric about AI and what we should be doing is straight up sort of accelerationist style rhetoric.

16:33You shouldn't fully trust the accelerationist. They have an obvious motive here, which is money. So if you're a venture capitalist, you don't want impediments on you being able to, you know, your startups to be able to grow and return on investment. If you're Jensen Wong, you don't want chip exports bans, right, because you want to sell as many chips as possible everywhere you can in the world. And the White House just has a lot of connections to the sort of right-aligned corners of Silicon Valley, and they sort of like them better, and they're sort of going along with what they have to say.

17:09Right, so the doomers are very influenced. We don't understand biases. The doomers are very influenced by these philosophies that go back to the 80s, these sort of futurist transhumanist philosophies. The accelerationist are motivated by money, money and power. We're kind of comfortable with the accelerationist story because we're used to this. Like, yeah, you know, obviously, like, companies want to make more money, and they don't want stuff to stop them. The doomer story is a little bit more complicated, and I think it's sort of thrown a lot of people off. This idea that they have an eschatological, futurist, utopian vision that is driving them is way stranger in the American experience than just being driven by, I want to get a return on investment.

17:48I want my company to grow. But those are two major groups.

The centrist position on regulation

17:51So what's the third? So I'll put the third in the middle here, and I'm going to call this the centrist.

18:01And I'm only calling this group the centrist because they're literally in the center of my diagram between the doomers and the accelerationist. Okay, so the centrist, this is a group that is concerned about AI, largely is in favor of the idea of we need some sort of controls or safeguards or regulation of AI. So they disagree with the accelerationists who say, let's just let her rip. But they also have no interest in the doomer philosophies, which they feel to be anti-humanist and, quite frankly, kind of strange.

18:37And so they want to regulate AI like a normal technology. We want to regulate this like we would regulate any other technology that we're worried about being unsafe. And if we think companies are acting weirdly, then we need to investigate and find out why. So they share the doomer's interest in regulation, but the style of regulation and the motivation is different. And they don't – they agree with the accelerationists' rejection of a lot of the ideological underpinnings of the doomers, but do not agree with the accelerationists' let-a-rip philosophy of like let's just let it grow.

19:10So all we're doing is holding back good – anything we do to slow down is just holding back possible good outcomes. So who are the centrists? There are sort of three very related groups. I'll write them as three separate ones. We have the AI ethics crowd. We have the – I'm writing NTCS here. This is the normal technology computer scientists. So the computer scientists who understand enough about the technology and enough about the ideologies to be like, look, this is a powerful technology but normal. We've got to treat it normal, very wary of the hype.

19:42And then we have cybersecurity experts – below that I'll just write the word cyber – who are worried in particular about the computer hacking ability or the cybersecurity implications of language models that are very good at producing computer hacks. So those are the centrists. All of these groups are related. Like for example, some of the AI ethicists are also these normal technology computer scientists but not all of them. Some of these cybersecurity people also come from the computer science world but not all of them. Some of them are from industry. So they're overlaps but they're not all the same.

20:16AI ethics is a term of art that – it's a field that's been around for a while. It definitely predates obviously like large language models and open AI and anthropic. The AI ethicists have traditionally focused on proximate harms. Like we're worried about this thing that AI is doing now or could immediately enable, right? So they're very worried about proximate harms. They would talk about things like bias and models. Like, hey, cultural biases that we don't like as a culture are in the training data sets.

20:48And if you train a model on this data set, it might have these cultural biases. And then if you're using that model to make decisions, it might be making biased decisions in ways we don't want it to do. But they're also worried about other applications of AI like for surveillance, for example, which is happening in China right now. Man, they're using these AI tools to hunt down and persecute in terrible ways minority groups that they don't like. So the AI ethicists have – they care about the proximate cause. They were also the first – I want to give them credit – they were the first to really point out early on the unsavory aspects of the futurist ideologies that drives the doomer.

21:30So if you've heard terms like Tescriel, which is an acronym that captures a lot of related philosophies, the R in there is rationalism. The E in there is EA. But they have transhumanism, long-termism. There's a bunch of sort of related and sub-branches of these various thoughts. But Tescriel, this was – Timnet Garibu and others worked on this. Emel Torres as well, I believe. They're arguing, hey, you've got to look closer at these philosophies because there's some deep anti-humanist aspects to it. It's almost eugenicist. And this idea of what matters is it doesn't matter how many people die along the way if we can get these like idealized transhumanist individuals to survive and be the apotheosis of the human race.

22:13And so they were kind of early at be wary about these philosophies. So it's something that they've been at. I have a foot – I'm in the normal technology computer science circle. You would also have like Arvin Narayanan is in here. I would put Gary Marcus in here as well. I'm in here as well. It's like, look, we look at the actual technology. We're less worried about superintelligence than we are about good old-fashioned straight-up harms that it could or couldn't cause right now. I do a lot of work, for example, on the specific types of long-horizon LL-empowered agent experiments that OpenAI and Anthropic are running that are causing problems.

22:53Other AI companies are not. They're running these experiments, I believe, because it's part of their race to get to superintelligence first. And this is why I've advocated we need to get in there. We need to investigate. We might need to regulate. Not because I'm trying to stop the bad god from coming but because I think these are bad actors, that their motivations are leading to dangerous behavior and we got to get in there. So I think the normal technology computer scientists, I'm one of them. But I also have a foot in the AI ethics world. I'm a part of the Center for Digital Ethics at Georgetown. I also see these things through an ethical frame.

23:24The cybersecurity world, these are experts at computer hacking and cybersecurity. Here, I'll take this off the screen. Which is not my fundamental expertise, but if you talk to the cybersecurity centrist, they'll say two things. One, this is a fairy tale, the superintelligence stuff. Two, these hacking tools are really dangerous and we got to stop advancing these just so you can do well on some benchmarks or reach some vision you have about where you should be on the route, the superintelligence. It's like these tools are very dangerous for very realistic, pragmatic reasons, all right?

24:00So again, I'll go back here. This is our landscape then. This is what you have to understand about the debate right now about AI harms is that it's these three circles. And the accelerationists don't like the doomers. The doomers don't like the accelerationists. The centrists have issues with both and agree with parts of both. Inside the doomers, you have civil wars and interscene battles where they all believe the same general thing. But this group, the rationalists, the effective altruists, often think that these two doomer companies, Anthropic and OpenAI, are going too fast or they're going to get it wrong.

24:31So it is a confusing picture when you see little aspects of it, little corners of the picture in isolation. But when you zoom out, it makes more sense. So I want to give this lay of the land. I mean I would say like probably the most confusing thing about our current moment is the fact that there's a call for regulation. And I think the most confusing thing at the moment is that the centrists and the doomers both are calling for regulation. But they're very different visions and motivations for what that means. And I would say if you're observing this, and I'm hearing this directly from congressional staffers, Congress that's looking towards regulation, so it's mainly the Democrats right now but also some Republicans as well, are being very influenced by the doomers, especially the effective altruist AI doomers, are having a huge impact on the types of regulations that Congress is thinking about.

25:24They have a lot of money and they're aggressively lobbying Congress, but also I've been told by, again, from staffers on the Hill right now that the effective altruist through the years and through these sort of fellowship programs have been placing – specifically this is a plan. They've been placing effective altruist onto congressional staff. So a lot of these congressional staffs have their own staffers in the ear of the reps and the senators saying superintelligence is the biggest worry. This is what we have to stop.

25:54Why is this a problem from the centrist perspective is that the concern is that intervention or regulation inspired from a doomer philosophy is probably not what we need. We need normal technology intervention. What are you doing? Why are you doing it? What's your motivation? And that we could actually have much more effective interventions that are going to apply, to be honest, way more to the two doomer labs than anyone else. The concern about just focusing on superintelligence or recursive self-improvement, which is an idea that comes out of the futurist circles, not computer science.

26:26I think we're going to miss actual, meaningful, probably workable-style interventions, which needs to be much more, in my opinion, focused on the specific technologies and techniques with specific types of safety guards. And so we're roughly aligned with the doomers who are saying we need to intervene, but we're really not aligned on why we need to intervene and what that should look like. So I think that's the most complicated part of the picture right now is you have multiple groups arguing for intervention, but they're not all the same people, and we really have to pull that apart.

26:58So let me show this picture one more time. Let this sort of sear into your understanding. This is the landscape of the AI harm conversation. So whatever you're reading, whether it is an article or it is a tweet or blue sky thread or it's a podcast or it's a buddy of yours at a dinner party, you got to first categorize that person somewhere in here to understand what they believe, why, and what their agenda is.

27:29Listen, I'm not going to tell you right now which of these you should agree with. Obviously, I'm a centrist, but that doesn't mean I'm right. It doesn't mean I'm right. I don't even really want to be a part of this debate, to be honest. I find it very stressful. But I do think clarity really matters. So if you're in one of these versus the other or you're listening to someone in one of these versus the other, you need to know where they are and why so that you can properly contextualize it, whether that ends up you rejecting their thoughts or embracing them even harder.

27:59All right.

Newsletter and closing thoughts

28:00That's my best attempt to explain this landscape. Hopefully, you found that useful. See you next time. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com slash ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life.

28:31Sign up today, calnewport.com slash ideas.