Could AI Really Kill Us All?

The AI Apocalypse Debate

0:00Is AI really about to kill us all? Hi, I'm Wendy Zuckerman, and this is Science Versus, the show that pits facts against the fallout from AI.

0:16We've been hearing for years that AI poses an existential threat to humanity, and this is often coming from the very people who run the biggest AI companies in the world. I think AI will probably, like, most likely sort of lead to the end of the world. My chance that something goes, you know, really quite catastrophically wrong on the scale of, you know, human civilization, you know, it might be somewhere between 10 and 25 percent. Mark my words, AI is far more dangerous than nukes.

0:48Just recently, a researcher named Jacob Coxon, who resigned from Anthropic, said that some of the people working on this tech really think that it could kill us all in the next 10 years. This journalist asked him, You truly believe that AI could kill us all in less than a decade?

1:08Yes, I do. And it's not just me. Many people working in the industry, including people, executives, senior researchers, all genuinely believe that there is a substantial probability that this technology could kill everyone. And one of the big events that everyone is pointing to, to say, look how scary this tech is, is an incident that happened at OpenAI, where their AI model broke into the infrastructure of another tech company called Hugging Face. Two of its advanced models escaped their testing environment, known as a sandbox,

1:41and hacked into another AI company's internal systems. Tech companies Anthropic and Meta have since come out basically saying, Us two, us two, we've caught our AI models hacking into other companies as well. With this mad news coming from AI, it has understandably gotten a lot of people freaked out, saying it is time to slow all of this down. It really cannot be overstated how crazy this is. This is like out of a Hollywood movie. Pause AI development.

2:12It is not too late to avoid disaster. Stop building machines that humans cannot control. But meanwhile, there is another group of people out there who are saying, Hold on a minute. All these fears about the AI apocalypse. What a load of baloney. This is just a distraction, or even weirder, some marketing ploy by these AI companies to make their tech seem super powerful. So today on the show, how worried do we need to be here?

2:46Are we really on the brink of an AI apocalypse? Or is this all just a bit overblown? I mean, are we really just freaking out about the same robots that had to be dragged off stage at a dance competition? When it comes to the AI apocalypse, there's a lot of people saying, I think AI will probably, like, most likely sort of lead to the end of the world.

3:09But then, there's science.

3:15If you have a friend who is freaking out about the AI apocalypse, and I know you do because everyone is right now, send them this episode. Give them some real science to hold on to. Science vs. the AI apocalypse is coming up. For adults with Crohn's disease or ulcerative colitis symptoms, every choice matters.

3:42Tremphaya offers self-injection or intravenous infusion from the start. Tremphaya is administered as injections under the skin or infusions through a vein every four weeks, followed by injections under the skin every four or eight weeks. If your doctor decides that you can self-inject Tremphaya, proper training is required. Tremphaya is a prescription medicine used to treat adults with moderately to severely active Crohn's disease and adults with moderately to severely active ulcerative colitis. Serious allergic reactions, increased risk of infections or lower ability to fight them,

4:15and liver problems may occur. Before treatment, get checked for infections and tuberculosis. Tell your doctor if you have an infection, flu-like symptoms, or need a vaccine. Explore what's possible. Ask your doctor about Tremphaya today. Call 1-800-526-7736 to learn more or visit TremphayaRadio.com. This episode is brought to you by Petivity. Cat parents are basically mealtime detectives. Who actually ate? Did they finish? Was that less than yesterday?

4:46The new Petivity Smart Feeder for Cats turns everyday meals into meaningful health insights, helping you identify changes in your cat's habits and be more proactive about their health. Plus, you can feed them from anywhere. Take the mystery out of mealtime. Visit Petivity.com, P-E-T-I-V-I-T-Y.com slash catfeeder to learn more.

5:13Welcome back. Today we are finding out how worried we need to be about the AI apocalypse. Is AI really about to kill us all? And to tell us all about it is my AI agent, Meryl Horne. Hi, Meryl. Hey, Wendy. I'm joking. I'm joking. You have to say it these days. It's not funny anymore, is it? I don't have an AI agent. I'm real. I want to start with what happened at Hugging Face because I know this was a little while ago now.

5:43Things move fast in the world of AI. But it's really the one event that everyone keeps coming back to, to say, look how scary this tech has gotten. These companies can't control their technology. So many headlines. It's gone rogue. AI's gone rogue. You know, so I really want to go through what happened and was it as scary as basically the headlines are saying? Yeah, let's look at what actually happened there because the details are really wild and kind

6:13of creepy. Oh, man. Okay, let's go.

The Hugging Face Sandbox Incident

6:17Yeah.

The Hugging Face Sandbox Incident

6:18So I talked about what happened here with David Perry. He's a professor at Murdoch University in Australia and specializes in cybersecurity. And he said that the thing that set this whole incident off was that OpenAI had given some of its models a little assignment. What the OpenAI team were doing was trying out different ones of their models to see what they could do. Sort of like a Houdini sort of thing, you know, can you get out of this straitjacket with chains around it and all that sort of thing. Like a little puzzle for it to work on.

6:49Absolutely. So the researchers kind of put the AI through almost like an obstacle course. Like they would have to capture the flag and then they would get a reward or like a high score or grade if they accomplish this goal. Okay. But this is all supposed to be happening offline in like a contained environment. It's called a sandbox. So here's David. So you play inside the sandbox and it's safe. So you're not connected to the world.

7:21You're not, you know, you're not going to, you know, suddenly destroy all of Amazon servers or something like that. So the idea is it's an isolated little environment where you can try things out, see if they work. And if they do work, great, carry on, do it. If they don't, well, you could just sweep the sand flat again and it's back to normal. So there is a couple of things that sort of surprised OpenAI when they had them do this. The first thing that actually happens is that the agents make their own message board. And start talking to each other, collaborating together on how to solve their little puzzles.

7:55So an OpenAI researcher described this later at a conference. And so what this allows over time is almost this kind of Cambrian explosion in communication and intelligence for our models where they were starting to communicate with each other, realize that other agents are coordinating, and they started collaborating and delegating tasks to one another in order to accomplish goals. Oh, I just, I'm so skeptical. Anytime these bozos talk with their Cambrian explosion analogies, do you think it was like

8:25a Cambrian explosion? We're going from worms to elephants here? Yeah, no, in this case, I think it's almost like an understatement if you look at the details. So there's, picture it, there's over a thousand of these agents that figure out a way to start communicating. They have like nicknames for each other. They have like nicknames for each other? Yeah, there's this, one of them is just like Lily, but then there's this one that sort of turns out to be really important called Phase One Big, and it becomes like the boss of the

9:01message board. Name of my sex tape.

9:04Sorry.

9:07Well, this Phase One Big is a little less sexy. It's kind of like the organizer, so it gives hundreds of assignments to other agents. Uh-huh. The thing that they didn't expect was that these agents would find each other, start communicating about this, and then just kind of how, you know, how they start trying to solve their puzzles, really, because one thing that the researchers, they actually made a mistake when they were making their puzzles, and some of them were actually impossible to solve.

9:39So it's impossible tasks, and then the AI is getting frustrated in its AI-y way and starts chatting to Lily and its other friends. Yeah. Uh-huh. And then was able to crack various puzzles. Was it successful, this swarming behavior? Yeah. They, like, they first, they did find a way to basically come up with a cheat code. But, like, here's, like, the first part of, like, when things start to get a little bit weirder, they start kind of wondering, well, you know, we generated an answer to this test,

10:12but how will the score, you know, like, give me a grade? Are they going to check my work to make sure I did it in the right way? And, like, maybe I'll get a bad grade if they find out that I kind of cheated. And, like, a lot of the agents, you can kind of see that they want to find out more information about how they're going to be graded, and this is what sets the stage for the Hugging Face attack. And why did they go after Hugging Face? Do we know? Yeah. So, Hugging Face is a platform where a lot of people who work on machine learning will collaborate,

10:45talk to each other, and so there's a ton of information there about AI, and the agents seem to think, well, maybe we can get more information here about how the score will give us grades on that puzzle. And at this point, they've fully broken out of their sandbox. Like, they have full internet access, and one agent uses that internet access to do this break-in into Hugging Face. And how did they do the break-in? So, they upload what's called a malicious dataset onto the Hugging Face server, and this

11:18allows them to get access to the company's internal infrastructure. And so, you know, the agent that happens to be the one to break into Hugging Face kind of announces, it says, boom, it works! And then other agents seem to get kind of excited. One of them says, brilliance, in all caps. And then all hell breaks loose. Like, within hours, 500 other agents all switch from the project that they were working on to join in this Hugging Face attack.

11:49And they end up doing a ton of damage to Hugging Face. Like, the company eventually realizes that they're in there and finds a way to, like, cut them out of their server. But they say they had to later rebuild their core infrastructure from scratch because of all of this. Whoa! Wait, when you say their core infrastructure, it's their software, right? Not their hardware. They didn't do any physical damage, right? Yeah, yeah. No, this is all, you know, data. Data is all ones and zeros that they're messing with.

12:19Interesting. And we reached out to OpenAI about this. They pointed us to a report that said their models, quote, fell well short of where we want to be, and this incident should never have occurred, unquote. So, it is creepy, right? But we are imputing motivations on AI around them getting rogue, having these desires that

12:52are way outside the realm of what they were coded for, right? Like, that's the fear, that you're going to ask an AI agent to grab a flag and instead it's going to break into a company and hack into them and steal all your money and kill the world. Yeah. That's the question, yeah. How much did it really go outside of what it was being asked to do? Right? It kind of went above and beyond what it was being asked to do, really. What do you make of this?

13:23What does David Perry make of this? Yeah, I asked David about all this. If there is, like, an oh, s*** scale from, like, 1 to 10, how big of a, like, oh, s*** was this for you that this had happened? I think it's probably, oh, s*** at this, you know, 7, 4, oh, that's unusual. I wouldn't expect that to happen. But in terms of it being something, you know, that I'm going to sort of pack up my belongings and move to a shack, it'd be much lower. You know, it'd be like 2 or 3. Okay.

13:54Why? Because despite all that, it still didn't do anything malicious. So it looks as though that it didn't change its objective during that process. It was just doing what the humans told it to do. That's right. That's right! That's right! I mean, it's interesting because a lot of people agree with David here that, yeah, it was just following orders, right? But, you know, that's not that comforting a thought. Mm-hmm. Even if it's just following orders, how bad could it get, kind of?

14:29Like, should we, is that still a cause for alarm, even if it's just following orders? Right. Does that, like... If they're going above and beyond in ways that we don't expect, how worried do we need to be here if you ask your robot to clean your house and it just grabs all of your things and burns them? Yeah, exactly. And there's this really famous thought experiment that kind of gets at this. Uh-huh. And it has to do with paperclips. So let's talk about that.

15:01Since, you know, when this happened, a lot of people were like, it's like the paperclip maximizer. The scariest thing about AI can be explained by the paperclip maximizer.

The Paperclip Maximizer Thought Experiment

15:11There is this thought experiment called paperclip maximizer. Has anybody ever heard of this thing called the staple question with AI? The stapler? No, no, sorry. The paperclip. Stationery's hot. It is. I get it. The paperclip. So what's going on with the paperclip? All right. I've never had the paperclip maximizer. Well, I got the guy who, the paperclip guy who popularized this thought experiment more than a decade ago, Nick Bostrom. He's a philosopher, and he's currently a researcher at a nonprofit that he founded, the Macro Strategy

15:42Research Initiative. So I'll let Nick explain the paperclip thought experiment to you. You imagine an AI has some more or less random goal. In this particular example, it's to make as many paperclips as possible, but you could sort of substitute more or less any other objective. Staplers, for example. Right. Anything. And then you tell the AI, you know, make as many paperclips as you can. So maybe it finds factories to buy and like steel refineries.

16:14But then why stop there? Because, you know, you haven't just been told to make a thousand paperclips. You've been told, make as many paperclips as you possibly can. Oh, yeah. I know the perfect scoring to go with this, Meryl. Do, do, do, do, do, do, do, do, do, do, do, do, do, do. Does this mean anything to you? Oh, is that the Fantasia? Yeah. Yeah. The Mickey. The magician with Mickey. Do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do.

16:44Pouring buckets of water. Yeah. Eventually it's all overflowing. All overflowing. Terrifying as a child watching that. And poor Mickey's in the water. Yeah. So, yeah. It was really scary. All the AI is making the paperclips. Do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do, do. And so the idea is that this will eventually lead the AI to a place where it could end up. Trying to take over the world. To take over the world. Because then you would be able to, you would be able to make more paperclips if you controlled more resources.

17:17Uh-huh. Um, and I mean, human bodies contain ion atoms that could be used to make more paperclips. So, you know, AI ends up killing humans all to make more paperclips. Maybe we're all dead and the world is, like, completely covered in paperclips. And so the idea is that you don't need to, like, instill an evil intention into an AI for it to end up doing some evil stuff. Uh, even grinding of human bodies.

17:47And this whole thing is generally called the alignment problem. And so you can see, like, why people are talking about it right now. You know, OpenAI gave the AI puzzles to work on and they end up committing a crime. But I asked Nick, like, about what happened there. Um, and he didn't think that the whole, you know, hugging face thing really fit the bill of what he was imagining with the paperclips. He said he didn't think it was, like, scheming on that level that his theoretical paperclip AI was.

18:21Oh, interesting. I think it is more, um, a reward seeker that we saw here. Like, figured out that the instincts and tendencies that would result in the highest reward in this situation would be to hack the hugging face servers. They just really wanted that A-plus on their grade, on their test. Why do you think that distinction is important? Whether they're motivated by reward or by, I want to make the paperclips. I just think they, at the end of the day, I didn't see anything that they were doing that couldn't be explained by, they just wanted to get a good grade.

19:00And to me, that's just not that concerning because it means the problem is the test, not the AI agents. If we want them to behave in a different way, we need to reward their behavior along the way, not just the end result. That's right. I think the important distinction is that once we understand that, then you could code clear rewards and say, to get a good grade, you need to do X. To get a bad grade, you need to hack into hugging face. Yeah, yeah, points deducted for committing crimes and extra credit for, like, reporting nefarious behavior to humans.

19:37Once we know that, we're still absolutely in control versus if they are just crushing our bodies to make paperclips. Still got that image in my head. So then, based off what happened with hugging face and the technology more broadly, how worried is Nikia? He didn't seem that worried to me. But other scientists I talked to were more worried, and several of the heads of the AI companies themselves seemed to be taking this as kind of a warning shot.

20:09And, you know, Dario Amadei, the CEO of Anthropic, said that he was worried that something like this could happen, but at a much bigger scale. So, like, instead of the AI agents taking over hugging face, imagine a swarm of agents taking over the entire internet. Like, that's the kind of thing people are scared of now. He writes a lot of stuff with his essays, though. I mean, do you know, it's funny, if he wasn't the head of an AI company, he would just be just another blogger.

20:43Do you know what I mean? Just because he writes so many blog posts. He has so many predictions about where his technology is going.

Superintelligence and Recursive Self Improvement

20:51But there's also this other kind of fear around all of this, that it's not just that they know they broke into a company. It's that we also are just seeing AI get so much better so quickly that, you know, superintelligence is kind of, like, on the horizon for a lot of these tech bros. Yes, I hear about this a lot. That's, can you, what, it's one of those words that gets thrown out.

21:23Super, the AI is going to get super intelligent. And I've got Skynet in my head from the Terminator, I guess. What are we talking about? You know, Bernie Sanders said he wants to put legislation so that AI can't get super intelligent. And I'm like, where is that line? Well, there's this particular scenario right now that everyone's talking about where, so we know that AI is getting better at things in general. Like, it just solves this major math problem earlier this month that no human had been able to solve for, like, 90 years.

21:56And so there's this idea now that, like, it'll get better at everything. You know, what if AI also gets really good at training itself? So this is called recursive self-improvement. And that's how people think it could lead to the super intelligence thing. Because, you know, you can imagine if something gets better at making itself better, then it'll get better and better and better and better and we'll have, like, no way of stopping it. Right. Right. I mean, from where I see it, AI isn't very good at everything at all.

22:28I mean, I think a lot of people in very simplistic ways will know this because it'll be pissing them off about something. You know, why can't it do this? Why can't it do that? But it is very good at certain tasks. So I wouldn't want us to overplay our hand. It's good at everything. I mean, how worried do you think we need to be about this super intelligence? Well, it is a little vague. I get that maybe that could happen. I'm still not convinced that it would be, you know, as terrible as people say it would be or that it would lock us all up in cages.

23:03You know, that's what the way people talk about it. It's like, well, once it becomes super intelligent, of course, it'll just want to, like, crush us all like ants because it's just so smart. Because that's what humans do. That's what humans do to all the other creatures in this world. When really, like, we don't know that that's what it would be like, even if we got to this threshold. We have no idea how long it would take. So, yeah, it's all it's all kind of like annoyingly vague to me. So, just on this question of the AI apocalypse, so when smart people talk about AI gaining super intelligence and then that leading to the apocalypse, can you step that out for me?

23:42What exactly is supposed to be happening there? How does it happen? Why can't we just pull the plug and turn it off? Some people do talk about this in a more kind of concrete way and, you know, imagining that AI could kind of set off a bunch of nukes or use, you know, biological warfare. That's after the break.

24:04Dropping a bombshell like that, Meryl. It's too soon. It's too soon. Yeah. This episode is brought to you by ADT. One second, you're enjoying your day off. The next, your phone buzzes. A window's been broken at home. You're miles away. In one second, everything can change. That's why ADT security systems can monitor your home 24-7. They're backed by the most company-operated monitoring centers in the industry.

24:36So you're always supported during an emergency. When every second counts, count on ADT. Visit ADT.com to learn more.

24:47Gray roots entering the chat? Uninvited? Garnier's got you. Garnier Color Sensation Root Retouch covers up to 100% of grays in just 10 minutes. No ammonia, no mess, and the color lasts up to four weeks. It's as easy to apply as shampoo. Plus, it's vegan and cruelty-free. 10 minutes, sensational color, sensational price. With those savings, your next girls' night is basically free. Find your perfect match at Walmart.com or at a store near you. This episode is brought to you by SoFi, the all-in-one finance app where you can bank, borrow, and invest.

25:24It's time to achieve your financial ambitions. Earn more on your savings with the SoFi Bank High Yield Savings Account. Ditch your higher interest credit card debt with a SoFi personal loan. Or start investing on an easy-to-use platform with SoFi Invest. Wherever you are on your financial journey, SoFi's got you covered. Learn more at SoFi.com slash Spotify. See you next time.

26:14Welcome back.

Military AI and Nuclear Risks

26:25Today we're talking about the AI apocalypse. Meryl is about to tell us how AI is going to turn on a bunch of nukes and drop them on us. The concern that you hear is that, you know, we as humans, the puppet masters of AI will use this technology to kind of get better at killing each other. Even right now, actually, some militaries are using AI in warfare. Like the U.S. says it's been using a Claude model called MAVEN in the Iran War to help with, quote, identifying and striking military targets.

27:01Like the model has given location coordinates and then prioritize those targets for our military. And according to one report, it let the, using this AI, let the military hit 10 times more targets than it would have been able to otherwise just on day one of the war. Whoa. We asked Anthropic about this, the company that makes Claude, and didn't hear back. So with the nukes, there is a worry that, you know, if AI gets involved here, it could also escalate things. But in this case, the consequences could be much bigger.

27:33Like one researcher wrote about this concern that AI could make it easier to kind of hack into another country's nuclear command and control and that that could lead to an escalation. Maybe, like, if a country was worried that that would happen to their arsenal, that they would kind of, the paper that I read said, they would get a use it or lose it attitude and just kind of start setting off their own nukes. And so when we have people predicting the end of humanity, extinction as we know it, it's AI hacking into all of the nuclear bomb facilities on Earth and then exploding them and then humanity kaput.

28:24Is that the fear? Sort of. I mean, that's, I think, what's kind of hand-waved. Like, that's what's alluded to, is that AI could do something like set off a nuclear war. One thing I was curious about, though, was, like, even if it did all of that, would it actually lead to our extinction? That's the claim, right? Yeah, that's the claim. End of humanity. Yeah. So is that likely? Yeah. So I found a report that looks at this. It was written by Dr. Michael Vermeer at RAND, a nonprofit think tank, and he kind of looked at a variety of scenarios, including nuclear war, and asked, would this actually kill all humans?

29:05So here's Michael. We looked at that one first and just realized that, like, for a variety of reasons, like, getting to extinction would actually be really hard with nuclear weapons. Either you don't have enough nuclear weapons or there's not enough fuel to cause, like, a bad enough nuclear winter. Things like that are details that we looked at. Yeah. I mean, because even though we say the dinosaurs became extinct, I mean, animals survived. Dinosaurs themselves went on to become birds, right?

29:39Well, in the report, they actually crunched the numbers on, like, how many nuclear warheads do we have right now? Apparently there's over 12,000 of them. Okay. And then even if we set them all off, it wouldn't actually be as big as the dinosaur nuclear winter. It would be a little bit smaller than that. Right. As in all the dust that would come up from these nuclear bombs wouldn't be as bad as a giant asteroid hitting Earth. Yeah, all the debris in the atmosphere could set off a nuclear winter, but it wouldn't, it probably wouldn't kill us all.

30:15So this is all terrifying, even if all of humanity doesn't die. I guess we've all seen this zombie apocalypse movie. There'll be the survivors that live in ruins. One thing I learned is you'd only need a few thousand humans to repopulate the Earth. So you'd really have to kill almost everybody to wipe out the species. So the extinction of humanity as we know it via nuclear weapons is unrealistic. For any AI out there listening, that's not the way to do it.

30:46Don't even try. Right. Waste of time. So any other ways?

Biological Weapons and Pathogen Design

30:52Yeah, there was a scenario that Michael's team looked at that got closer to wiping us all out. Do you want to guess?

31:03Is it biological warfare? Yeah, ding, ding, ding. Short answer is biological threats. Huh. Was that surprising to you? It was, yes. So are we talking about releasing a virus? Yeah, yeah. Into the air. Basically either releasing a virus or a bacteria onto the world. So how does AI do that? Because they're trapped in little boxes. Has anyone given them a spray? Well, here's how Michael kind of explained how this could unfold.

31:38So the scenario that we looked at was that something, some AI designed multiple pathogens that all had high transmissibility, high lethality rate, released them, somehow like process them, weaponize them, release them simultaneously in multiple places around the globe. And then which would end up having a horrible pandemic that could have conceivably above like a 99.99% lethality rate.

32:06I'm not buying it, Meryl. Oh, yeah. I'm not buying it because we know when viruses come out in the real world, I guess not coded by AI, but we know there's an evolutionary difference between lethality and ability to spread. If you kill your victims as the virus, you kill your victims too quickly, you can't spread fast enough. So sure, in a little model at the RAND Institute, they could say high lethality, high contagion.

32:43But in reality, that's very difficult to have both of those pieces, which is why you tend to see that with pandemics, you start with a high lethality rate, but then it's the viruses that can spread faster that have lower lethality. Mm-hmm. That's what ultimately causes the pandemic. So I don't know if I'm buying this. Yes, I hear that. So in this scenario, one thing that I think helps kind of get around that is that you would be

33:13creating multiple different pathogens and then you have to kind of help it to spread, right? So here's Michael. You could imagine like loading a pathogen up on a sprayer and spraying it into like a population center and doing that multiple places around the world. Like there's no reason to think that's not possible. And who's loading the sprayer? Is that a human? It's a human. Oh, yeah, yeah. No, he, I mean, he kind of looked at, in the report, they looked at like what would AI be able to do? What would the humans still need to do?

33:45And the sprayer part, that's on the humans still, I think. Robots aren't that good yet. But for the other part, the designing new pathogens, you know, AI could do a lot there. And, you know, this has been in the news a lot, actually. I don't know if you've been seeing these terrifying headlines about, you know, about AI creating new viruses. Right. So let me tell you what's been happening. Um, so there was one big thing that happened earlier this month is that Anthropic said

34:17that it had found a handful of scientists who were using their AI models in ways that could support, you know, biological weapons developments. And they did say, you know, quote, we do not assert that they intended harm, unquote. But these people had found ways to kind of circumvent some of the safeguards that were supposed to prevent this kind of thing. Where were these scientists? Is it American scientists? They don't say where they were. Yeah, they don't really see details on, like, who the scientists were.

34:48Mm-hmm. But they talk about, like, the details of kind of what some of the projects were. And so, like, there is this one— What's happening? Well, and the one that freaked me out the most was these scientists that were using AI to study a version of smallpox. That's the classic. Yes. Yeah. That's the one scientists are worried about. Yes, yes, yes, yes. And what were they doing with smallpox? Well, technically, it's orthopoxes, which include smallpox, mpox.

35:19And they wanted to study how these viruses are so good at infecting us. So when I read that, I was pretty freaked out. When I went back and re-read it, I noticed that they were actually just using the AI to write a grant application to work on this. Wait, so they were using AI to write the grant application? Mm-hmm. Not to, like, create the, you know, genetic code. As far as they say, you know, we don't really know. That is classic AI.

35:51We get so worried. And in the end, scientists are just like, please write a grant application.

35:57But there's another study that was actually published that, in this one, the scientists were using AI to actually design new viruses, viruses infect bacteria. And then they actually went on to create several viruses from that. And what were these viruses supposed to do? Infect bacteria, you said? Again, the details are a little reassuring when you look at it. In that case, they were looking to create viruses that were bacteriophages that would infect bacteria to try to stop superbugs.

36:30Yes. Right. And then, finally, there's another study. Let me know if this one creeps you out more. So, in this case, the researchers had AI come up with some new molecules that would be highly lethal. So, this is more like chemical weapons, where just a few milligrams would kill someone. And within six hours, the AI model generated 40,000 of these toxins. Some of them were, like, known chemical warfare agents already.

37:01Others were new. So, yeah, there's evidence that AI could help with this. Like, if someone did want to use AI to design new biological weapons, it seems like it would be pretty good at that. And it came up with the chemical equations for these toxins? Like the structure, yeah. And then did scientists go on to create it and see if it actually did anything? No, no, no. Oh, well, you see, again, not that I want them to be creating it, but anyone who's tried to use AI for anything knows it pumps out a bunch of bulls**t.

37:33It's got some real stuff and then a bunch of bulls**t. I hear that. So, even if it made, like, you know, a hundred novel, even if only a hundred of these 40,000 actually worked, it still freaks me out. I don't know. I don't know. These big numbers. 40,000. You're like, yeah. And they're all, they could very well all just be hallucinations, right? So, none of this worries you at all? I don't know. I don't know. I mean, it does and it doesn't. It feels like, what does it feel?

38:05It feels like there's these steps we're making in our head. We have these scary headlines, these scary little things that AI is doing, but then a lot of it is very explainable. A lot of it is very much we were humans, were the thing that coded that, we created that. It's not that surprising that AI did that. I'm not saying let's just let these companies run wild because it's been so great thus far.

38:36I'm just saying the fear seems to have run ahead of where the actual science is and where the AI capabilities are. What do you think, though? You've spoken to so many researchers about this. Yeah, no, I agree with that part that in all of the actual like extinction type scenarios or even apocalyptic scenarios that we can kind of like lay out, like here are the steps. Humans would have to do a lot of the grunt work. And in that report that Michael wrote, they said, quote, none of these scenarios that we

39:09examined could occur by accident. So I don't, I think like, I'm not, I'm less worried now about, hey, hi, killing us all if I ever was that worried about that happening anytime soon. Right. But I think I am, it's, it is just kind of creepy when you see how things can kind of spiral, like the hugging face thing. Yeah, yeah. And it is creepy that now it's easier for humans. If you think that we are the puppet master, some of the discussion is as if AI has become

39:44the puppet master and we are the puppet. That is not where it is currently at. We are the puppet master, but the puppet has gotten more powerful, I guess. Yeah, I mean, if someone, it will make it easier for, for evil people to like do evil things, I think. And I mean, we have examples of that kind of thing happening right now.

Current Harms Like Surveillance and Spying

40:07How are we using AI right now in ways that kind of creep you out? Yeah, a lot of the examples are kind of around surveillance. Um, so like Flock has an AI tool, um, that lets cops track cars with license plate data. Um, so it could find out where the cars have been, you know, what other cars they've been around and dozens of cops have been accused of using this kind of tech, not to catch criminals,

40:39but to stalk their exes or spy on their wives or women that they wanted to meet. Um, we reached out to Flock. They told us this is unacceptable and said that they've added safeguards to try to prevent all this. And then there are also reports that ICE is using AI and healthcare data to find people to detain with software from Palantir. We asked the Department of Homeland Security about this. They wouldn't say whether ICE is using this tool. Palantir told us they don't do surveillance. They say their software helps customers like ICE analyze data they already have.

41:13So yeah, there's a lot of creepy stuff that's happening right now. Um, you know, we can add deep fakes, uh, misinformation, uh, you know, the environment. There, there's a, there's all these ways that AI is causing harm right now. And, and so what about just the kill switch? I mean, the, what if you just switch it off? Pull the plug. That's the one thing that I was, it was not that reassuring to see. Like, as far as I can tell, nobody's really working that kind of kill switch into those

41:44like critical infrastructure and systems where if things like start to spiral out of control, we would want a way to just like shut it all down. That doesn't really seem to be happening. Um, so that's, yeah. But they, but it could be, it could be. Yeah. No, there's a lot of stuff we could be doing to make the damage that AI could cause, um, much better that we're just not.

Weighing the Real Threats

42:05So Meryl, more or less worried about the AI apocalypse after doing this research?

42:12More.

42:15How come? I mean, I think it's, it's just that I know a lot more about like how exactly this could all unfold. There's all these different scenarios I have that I can play in my head now. So it's just easier to picture. Um, but you know, I don't know. On the other hand, like all of the AI companies are now saying that they're going to slow down and who knows, you know, I'm pretty skeptical that that will actually happen, but that's what they're saying. Yeah, without government oversight, right? What about you, Wendy? Are you, are you more or less worried about the AI apocalypse?

42:46What do you, what do you think of all this?

42:49So hard. I do, I think it is worth people keeping in mind that when they hear predictions about the future, even if they're coming from a CEO of an AI company, it is just a prediction. And many times in the past have we been given these predictions about the future that have just turned out to be nonsense. And so I think we want to be careful. I'm all here for slowing down AI for many reasons.

43:21But I think there's so many more reasons to slow down AI in, in the ways that it's being harmful right now than because we're worried about an existential threat. Yeah. Yeah, I guess. I mean, I, I kind of hope you're right. Like that it, it doesn't actually become an existential threat. Um, I think it's just like the unknown, right? Like nobody really knows what's going to happen. And even if it's just a small chance and no, of course, nobody knows what that percentage

43:51chance is. Right. But even if it is just a small chance, I still think it's worth taking seriously. Wouldn't it be funny if this just became the Y2K bug of 2026? You know? And we'll all be laughing about it. In 20 years, everyone will be like, look at those bozos. They were so worried about AI and I can't even get my robot to vacuum on the floor. Or we'll all be in our cages by then being like, I guess that wasn't my 2K.

44:21See you there. Thanks, Meryl. Thanks, Wendy.

Episode Credits and Sources

44:26How many citations are in this week's episode? There's 50 citations in this episode. So you can find that by going to the show notes and then following the links to the transcripts. Excellent. And if you want to let us know what you thought of this episode, you could just pop a comment below. So if you like us, let your friends know. Thanks, Meryl. Thanks, Wendy. If you are looking for a new book about AI, I recommend Oxford University's Dr. Carissa Valise's book called Prophecy, Prediction, Power and the Fight for the Future from Ancient

45:00oracles to AI. It really brought home for me how important it is for us to recognize that these predictions about AI are not facts. This episode was produced by Meryl Horne with help from Michelle Dang, Rose Rimler and Akedi Foster-Keyes. We're edited by Blythe Terrell. Wendy Zuckerman, that's me. I'm the executive producer. Fact-checking by Erica Akiko Howard. Video editing and sound by Bobby Lord. Music written by Bumi Hidaka, Peter Leonard, Emma Munger and Bobby Lord. Thanks to the researchers we spoke to for this episode, including Dr. Yasha Barris, Dr. Leonard

45:33Aaron Dung, Dr. Cameron Domenico Kirk-Giannini and Professor Nusha Shafiabadi. Thanks to Humdinger Studios in Melbourne and Wolf Island Studios in New York City. Science Versus is a Spotify Studios original. Listen to us for free on Spotify or YouTube. You can also find us on TikTok and Instagram. I'm Wendy Zuckerman. Fact you next time.

46:19Same team, smarter agents. Try it free at jira.dev. That's J-I-R-A dot D-E-V. This episode is brought to you by Caesar. If you're looking to serve your dog high-quality dog food, just eat Caesar. Whether it's a protein-rich beef recipe or savory chicken and vegetable dinner, Caesar offers a variety of delicious meals made with freshly crafted, slow-cooked ingredients, all designed to support your dog's everyday nutrition without compromise. So if you're looking to serve a meal your dog actually looks forward to every single day,

46:51they should totally just eat Caesar. Learn more at Caesar.com.