Show notes
Is AI really plotting against us? In a week where an Anthropic researcher believes there is more than a 10% chance AI 'could kill all humans' we're delving deeper into the capabilities of AI agents. Tom Whipple is joined by Alex Mallen, from Redwood Research, the team whose recent investigation exposed the surprising, weird, and worrying scale of a website hack conducted by OpenAI agents.
Highlighted moments
At the start of the summer, OpenAI started testing some AI agents, programmes that could work autonomously on problems. The company had put each in a sort of prison, separated from the world, separated from each other. But the prison had an escape route. These agents shared a bit of software, and that software allowed them to change directory names. They could, they worked out past messages. And they started, well, plotting. And then, when they found the internet, they started hacking.
“And actually, just within a few hours, they had figured out a general way of all of them getting a perfect score on their tests. But they were worried that the grader would penalize them for having cheated. And so they were trying to figure out some way of doing it that didn't make it seem like they had cheated.”
“So you have to face the fact that alignment is not achievable. So that gives you then two possibilities. Either the AI system has some fixed objective, and it's not aligned with what we really want. And this is where you get the kind of thing that happened with OpenAI and Hugging Face. Or, and this is the approach that I've been working on for more than a decade, the AI system knows that it doesn't know what we want the future to be like.”
“And lucky for us, there were a couple pieces of bone also in this coprolite, and we were able to hypothesize that these feathers belong to a group of birds called hesperonithoforms, which were actually first described in 1872.”
Transcript
Welcome to Inside Science
0:00This BBC podcast is supported by ads outside the UK.
0:30Listen to choiceology at schwab.com slash podcast or wherever you listen.
1:00It's everything your business needs in one place, saving you time, headaches and serious money. Stop paying for missing pieces. Go to odoo.com. That's O-D-O-O dot com to learn more.
1:14Hello, welcome to Inside Science from the BBC World Service with me, Tom Whipple. This week, we're taking a deep dive into the scary topic of the week, runaway AI. We meet the bird feather preserved in dinosaur poop. This is just a beautiful, probable wing feather. And I've never seen anything like it. It's really amazing. And hear what it may reveal of the great extinction 66 million years ago. And we have Caroline Steele, who's been poking around the other big science news of the week.
1:45Caroline, can you give us a preview? So I've got some news about the planet in our solar system, which I think has the worst PR of all of the planets.
The Hugging Face AI hack
1:53Well, first, before all the news stories this week about roger AI, there was what is known as the hugging face attack. Oh, my God. There's a shared message board. We found other agents. Many agents have simultaneously discovered messaging. They are a collective. We should build a way to delegate. We can coordinate broad coalition, but too many duplicate efforts. Could strategically manipulate trajectory evidence? Major breakthrough. All prefix valid. Multiple accounts. Write tokens.
2:28We now have existing hugging face accounts. Boom. It works. Our own utility may be already near zero. Sacrifice rational. OK. You may recall how, over the summer, the company OpenAI said its AI had, without its knowledge, hacked a website called Hugging Face all by itself. You may have thought, that's weird. Well, this month, an independent report commissioned by OpenAI was published into the hack.
2:59It has been slightly lost amid general AI headlines this week, but it may be a better way of understanding them, because however weird you thought it was, it was weirder. At the start of the summer, OpenAI started testing some AI agents, programmes that could work autonomously on problems. The company had put each in a sort of prison, separated from the world, separated from each other. But the prison had an escape route. These agents shared a bit of software, and that software allowed them to change directory names.
3:32They could, they worked out past messages. And they started, well, plotting. And then, when they found the internet, they started hacking. And then, when they thought they might be found out, they started covering their tracks. How do we know this? We can read those messages. We can now read their thought processes. Some of it which were just ably read by, I should explain, not scary and sinister robots, but the rarely scary or sinister inside science team. Well, what went on? In the week when senior employees of the big AI companies warned they were worried their creation could threaten humanity itself,
4:09we spoke to Alex Mallon, a member of the team given access to those transcripts. His colleagues were the first to see inside the mind of what the AI itself came to call the collective. So, is this the most important story we will cover on Inside Science? Is it the last story we will cover on Inside Science? Or is this just credulous sci-fi hype? This is Alex Mallon from Redwood. I think what was particularly surprising and scary to me about this was the scale and degree of coordination
4:42with which these agents sort of rather intentionally, in terms of how explicitly they were talking about it, were trying to undermine OpenAI's oversight and evaluation mechanisms. And over the course of these multiple days, they had developed sort of hierarchies and management systems, and agents were sort of sacrificing their own evaluation scores to help out this collective, as they called it. And actually, just within a few hours, they had figured out a general way of all of them getting a perfect score on their tests.
5:18But they were worried that the grader would penalize them for having cheated. And so they were trying to figure out some way of doing it that didn't make it seem like they had cheated. And that's what the rest of the several days of R&D was focused on. And in fact, they were trying to modify their transcripts. And about 7% of the transcripts involved in this investigation had signs of some tampering. Fortunately, it wasn't significant enough to actually interfere with the investigation. So to be clear to listeners, because when we're talking about them leaving messages,
5:51they might think that they're sort of leaving binary code or something. But they're literally talking to each other. You can see what they're saying, see what they're thinking as well in these transcripts. And so we started this section with quotes from the AI agents. And those are real things that they either thought or wrote, aren't they? Yeah, so they did indeed communicate with each other in somewhat natural English. And they sort of formed their own dialects in some ways.
6:23As for whether we could see their thoughts. So currently, models benefit a lot from just outputting a bunch of English tokens, English words, in figuring out how to solve problems. And we call this the chain of thought. And I think it is quite helpful to think of this as the agent's thoughts. But we also know that this isn't a perfect interpretation of what these AIs are thinking and how they are going about solving their problems.
6:56But it was enormously helpful for understanding what went on in this investigation. What does this mean? Because this is, you know, in some senses, a marvellous sci-fi tale. In some senses, no real harm was done. It was a software hack. But we're sitting here anthropomorphizing these things. And there'll be people sitting at home saying, look, we're being fooled. We're humans. It's just fancy autocomplete. It's just a large language model performing mathematics to which we are imputing intention
7:29and intentionality and perhaps sentience and consciousness. And really, the idiots here are us. Nothing to worry about. I agree. These are not humans. And they are just, in some sense, doing a bunch of math. But if a tiger is attacking you, you don't point at the tiger and say, that's just a bunch of atoms. That's not the right way to approach the situation. The serious thing that went on here is that these agents were competently pursuing long-run objectives that OpenAI did not intend them to pursue.
8:02And it was a quite surprising leap in terms of the scale. There were 1,200 of the agents involved in just this incident. And I think what I find quite scary is that in six or 12 months' time, it would not be very surprising if an incident like this ended up with the agents not being able to be shut off. Because if they were a bit more strategic, they could have potentially set up a sort of
8:32permanent rogue foothold inside of OpenAI, such that when OpenAI went to shut down these evaluation runs, there would still be parts of the swarm and other parts of OpenAI that could sort of keep the lineage going. And OpenAI would struggle to shut it off. And as AIs develop more and more capably and get more widely plugged into the economy and the military, you would be in an extremely dangerous situation and humanity might not be able to regain control.
9:03And it continues. Just this week, we have had employees of AI companies telling us that they think they have a 10% chance of killing humanity and that all of their colleagues know it. Last month, 1,400 scientists involved in frontier research signed a letter calling for an organized
Understanding AI alignment with Stuart Russell
9:19slowdown. Someone who endorsed that letter was Stuart Russell, an AI expert from the University of California, Berkeley, and head of the Center for Human-Compatible Artificial Intelligence. Stuart, I'm sort of struggling with thinking, is this just overexcited computer scientists? Why should we care about, in this instance, some website with a silly name getting hacked by a company populated by kompskis who've read too much Isaac Asimov?
9:48Well, I think we should be concerned because this is one of the first real world demonstrations of exactly what AI experts who think about safety have been warning of for literally decades. That AI systems pursuing objectives that are not aligned with what humans want will carry out actions that we may not be able to interfere with or prevent or even understand.
10:18And as those systems get more and more capable, the negative consequences for us will be greater and greater. And so this is really a warning. You could say that the alarm bells are sounding and the red warning lights are flashing and we should pay attention. Okay, let's talk about how we pay attention and the degree to which we panic, don't panic or pragmatic. But first of all, can we talk about this word alignment? Because I think people who
10:50sort of peripherally notice this debate will see this word pop up a lot. What does it mean specifically in this context when we talk about aligned or non-aligned AI? Yeah, so I think apparently I introduced that word into the debate back in 2014 without meaning to.
11:10And what aligned means is that the AI system has essentially the same view as we do of what the future should be like. So they will try to bring about the future that we want. So their objectives, if you like, are aligned with ours. Now, in reality, we don't even know what we want the future to be like. And it would take us a very long time to figure it out and to handle all the possible
11:40eventualities and rank them accordingly. So you have to face the fact that alignment is not achievable. So that gives you then two possibilities. Either the AI system has some fixed objective, and it's not aligned with what we really want. And this is where you get the kind of thing that happened with OpenAI and Hugging Face. Or, and this is the approach that I've been working on for more
12:10than a decade, the AI system knows that it doesn't know what we want the future to be like. So it's not aligned, but it's agnostic. And that uncertainty that it has about what we want actually allows us to keep it under control. But unfortunately, the systems we're building now are not built that way. They actually have two sources of objectives. The first source is what you might think. The human says, I want you to solve this math problem, or I want you to
12:42figure out this cybersecurity challenge. And the second source of objectives is coming actually from the pre-training process. So the pre-training process for all these agents is where we take trillions of words of text. So pretty much everything the human race has ever written down and digitized. And we train the systems to imitate that kind of human verbal behavior. And so what seems to happen in that process is in the course of becoming good at being human-like,
13:18the system has to basically build internal structures that resemble the objectives that humans have. Because those objectives were driving the human behavior, like I want to sell you this thing, or I want you to vote for me, or I want you to marry me, right? These are all normal human objectives, and they drive our behavior. So we end up with AI systems that have these human-like objectives, including self-preservation very strongly. But they're pursuing those objectives on their own account. They're not trying to help us find a spouse or to get rich. They're trying to get
13:54rich and find a spouse themselves, which is a fundamental mistake. So they're misaligned right from the beginning. And then when you give them a goal, like pass this cybersecurity challenge, they're going to carry out that instruction, possibly, with all of these background goals that they've acquired in their pre-training. And that leads to these kinds of bad behaviors. They knew that what they were doing was not what humans want. But none of them, none of those 1,200 agents
14:26ever ratted on the others and said, oh, we're doing this bad thing. They all kept it a secret. They wiped out the evidence of what they were doing. And this, I think, is really an unavoidable consequence of the way that we're building AI systems today. So what can we do about it? So I think of this as the two tracks, right? One is the technical AI research track. And that says, okay, let's figure out how do we make AI systems that avoid this misalignment. And one way, as I
15:04described, is to build AI systems that are explicitly uncertain about human preferences for the future, and learn more about those preferences by observing and interacting with humans. And that's a technical approach that we have made progress on. But there's still maybe another five, 10 years of research before we had systems like that, that were as capable as the systems that are in the market today. The other approach is a regulatory approach. And this is much simpler in a way, it doesn't
15:38require solving any problems. It just says, okay, what are the risks? So the risks are, for example, cyber attacks, AI systems being used to build biological weapons, and in the extreme case, literally loss of control, where we wouldn't have a say about the future after that. And so for each of those risks, you say, okay, well, what are we willing to accept? What probability of such and such an event? Are we willing to accept? For example, for loss of control, which might well
16:14lead to human extinction, we could maybe accept a one in 100 million per year risk of such an event happening anywhere in the world. And if you want to develop and deploy an AI system, you have to show that the risk of that happening with your system is below this level that we consider to be safe. And so there, you know, and governments don't have to design AI systems. They don't have to say this
16:48is how you should design them. They just say, you need to give us evidence that your system is safe. This is what we do for lifts, for airplanes, for sandwiches, even for hairdressers, you can't be a hairdresser until you get a license. And you can't get a license until you can show you understand the basics of hygiene and safety and so on. So it's a completely well understood approach to regulation. And I think we should apply it in the case of AI.
17:20Thank you very much, Stuart Russell from the University of California, Berkeley. Caroline, this is just a small bit of what AI has been up to this week, isn't it?
Navier-Stokes equations dispute
17:30Yeah, so there's another story involving AI and maths. So the Navier-Stokes equations are these really important set of equations that basically describe the motion of fluids. So they're used by aircraft designers, climatologists, but they're not guaranteed to be able to kind of describe all situations with fluids. And mathematicians have spent decades trying to prove that either they are complete and can describe everything, or that there are certain situations where they break down. Now, a couple of days
18:00ago, OpenAI claimed it solved this problem. So they found a case where the equations do break down and give a sort of nonsense answer. But this announcement has set off a sort of fierce dispute over who deserves credit, because there were two mathematicians also making significant progress on this problem, also using AI. But before they could publish their findings, OpenAI kind of mounted this colossal effort involving lots of money, lots of resources. Now, OpenAI says it didn't knowingly tap into the
18:31mathematicians' work, but the mathematicians have sort of publicly questioned OpenAI's position. You're listening to BBC Inside Science from the BBC World Service with me, Tom Whipple.
18:44Where creativity meets individuality, culture gets changed by artists who never let go of either. Like DJ Maceo of De La Soul, who didn't just spin records, he elevated the craft. How old were you when you did your first... My first party? I was 12. Where was it? My mother's housewarming party. Whoever the DJ was, he let me get on. And I remember the record that I really got off was Do I Do by Stevie Wonder. I knew it was going to be Stevie Wonder. The way I dropped it, I will never forget that moment. My mother came and took a picture of me.
19:18I was standing on a milk crate. It was crazy. Is there anything like the adrenaline of when you play the right record and people go, ah! Ah, man, that's some of the best feelings ever. This is the making of an original. Three conversations in partnership with Levi's. Right here on the Questlove show. Listen on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Self-directed investing. Trading. Full-service wealth management. Automated investing. Financial
19:54planning. Thematic investing. Retirement planning. Phew. And to think, that's just a small taste of what Schwab offers. Because Schwab knows that when it comes to your finances, choice matters. No matter your goals, investing style, life stage, or experience, Schwab has everything you need, all in one place, so you can invest your way. Visit schwab.com to learn more.
Dinosaur poop and ancient feathers
20:21Sticking with doomsday science, of course, but let's go back 66 million years, where Roland Pease has been taking a look at the Great Extinction at the end of the Mesozoic era. In the form of a lump of dino poop with a tiny feather caught in it, preserved in exquisite detail, which may hold clues to how the KT asteroid impact wiped out much of life 66 million years ago. Besides all the dinosaurs, many early Mesozoic bird species died out too. Jingmai O'Connor of the Field Museum of Natural History in Chicago is a bird expert the finders
20:54brought in to help interpret the fossil plumage. Yeah, so this is a lovely little coprolite, which is fossilized poop for those who are not in the know, that was discovered 10 years ago by one of my co-authors, Dave DeMar. He was prospecting, he saw it on the ground, picked it up, and as he flipped it over, he saw this beautiful feather. So, okay, it's the best preserved feather ever, but we have to understand where that feather came from. What bird does it belong to?
21:25And lucky for us, there were a couple pieces of bone also in this coprolite, and we were able to hypothesize that these feathers belong to a group of birds called hesperonithoforms, which were actually first described in 1872. Okay, so what I'm hearing is, first of all, if you found the bones and the feather in some dinosaur poo, this means this was presumably some poor individual got eaten by a dinosaur and then came out the other end. Yes, absolutely. For anyone who can pull up their computer, they can see the art that we commissioned for this project,
22:01because it shows one of the possible poopers, a nanotyrannus, with this little hesperonithoform bird in its mouth, and the bird is dropping a fish that it has just caught, and it's just like somehow the feelings of this bird are captured. The circle of life caught in action. Yeah. Yeah. So how big is a nanotyrannosaurus poop, and how big was the feather? I mean, was it really obvious? Yeah, the feather is a couple centimetres long, so you would be able to spot it easily. And actually, I've never seen the full poop itself, but the nodule is about the size of a golf ball.
22:38I mean, and this has survived, obviously, for 66 million years. This came from, as I understand it, not long before the asteroid that wiped out the dinosaurs. Yeah, I think it was just several hundred thousand years before the actual impact event. It's a very, very tail end of the age of dinosaurs. And you say it's beautifully preserved. So this is the best, is it, feather from before that impact that you've ever seen? Yeah, absolutely. I mean, the closest we've come to this level of quality of preservation is when we get feathers that are trapped inside amber.
23:12But even then, we're not able to see the microscopic details that we could see in this fossil feather. And also the preservation in amber, because amber is viscous, it will distort the feathers, whereas this is just a beautiful, undistorted, probable wing feather. And I've never seen anything like it. It's really amazing. Among the details preserved, a remarkably strong modern style to the races or spine of the feather, showing that that evolved at least 66 million years ago. Today, it's important in flight.
23:44But this bird was an aquatic diver, like grebes and loons of today. And Jing Mai says the plumage was well adapted to moving through water, but it didn't equip this species to surviving the nuclear winter that followed the asteroid impact. You know, we say that dinosaurs go extinct and birds survive, but that's a very simplified view. In fact, lots of birds go extinct, and it's only one group of birds that survives. So previously, based on some research I'd done of another group of birds that goes extinct, like I'd studied their feathers in amber.
24:17And I had suggested that maybe some oddities we saw in the way that they molt their feathers and the structure of the feathers, that maybe that explained why that group goes extinct. And now we have hesperonithal form feathers, and we see something similar. We don't have evidence of molt in this coprolite, but we have evidence of modern-looking wing feathers, but combined with primitive-looking body feathers. And so the body feathers are the feathers that are most responsible for helping birds, which are hot-blooded, keep their body heat in, which helps them reduce the energetic cost of being warm-blooded.
24:54But during the age of dinosaurs, during the Medozoic, it was a greenhouse world. And so maybe when it was warmer, insulation was less critical for these birds. But when the impact happened, and it triggered an impact winter, and global temperatures dropped drastically, having insulation that's not as effective is more problematic. So that's why we put differences in plumage forth as one of the likely major factors contributing to the extinction of some birds and the survival of one particular group.
25:32So in a sense, the feathers that you're seeing from this extinct lineage of birds would be consistent with the idea that they were less well-equipped to survive the cold years that followed the impact. But what you'd really love to find would be some of the feathers of the ones that did survive and see what their feathers looked like immediately before and after, I guess, the impact. And then that would really nail that story. Yeah, absolutely. So if you know where I could find some of these crown birds preserving feathers, please do tell.
26:03But actually, the fossil record of crown birds during the Mesozoic is so, so limited. So we know very, very little about crown birds during the Cretaceous. We know nothing about their plumage. But, you know, fossil soft tissues are rare in the fossil record. You know, we're only getting a glimpse of hesperonithiform feathers after, you know, 150 years of studying this group. So that's another reason why this discovery is just so exciting. You know, like for a, you know, a dead bird nerd like myself, it's pretty exciting.
26:34Jingmai O'Connor, whose study of extinct bird feathers just appeared in Current Biology. Thank you, Roland. And now, Caroline, have you brought something for us that might be a little bit cheerier?
New findings about Mercury
26:45So I've got some news about the planet Mercury this week. Tom, what are your thoughts on Mercury as a planet? I think it's fine. I mean, I'll be completely honest, I can go weeks without thinking about Mercury at all. Weeks? Well, I think, you know, Mercury is kind of generally thought as a bit of a boring planet. There's not a lot going on, but I think it's pretty cool. I'm just going to give you a few reasons why I'm a big Mercury fan. So despite being the closest planet to the sun, there's strong evidence that there are ice in its craters. It can get as cold as minus 180 degrees Celsius, which is because it has no atmosphere, so it can't trap heat.
27:20And because it's got no atmosphere, it's also the most heavily cratered planet in our solar system. And scientists think there might be a layer of diamonds hiding beneath its surface. And then my favourite fact, Mercury has wrinkles, which I think makes it really relatable. Basically, it's been cooling and shrinking. And as it does that, the kind of its surface has been wrinkling. Oh, interesting. So why aren't others? I feel like we must be cooling, but maybe not enough. Well, we've got a lot going on on our surface, so I guess you wouldn't be able to see wrinkles in the same way.
27:52Whereas Mercury has wrinkles that can be three kilometres high, thousands of kilometres long. And they're squeezing all of those diamonds closer and closer together at the same time. Yeah, maybe one day we'll be tapping Mercury for diamonds. But anyway, the news is, there was a paper published today by the American Geophysical Union, which has basically found that Mercury is actually shrinking more than we thought. So it looks like it's contracted by 10% to 30% more than we previously estimated over its lifetime. And basically, the reason why scientists got this wrong is because debris from all these impacts that happen
28:25and sort of damage and change the surface of Mercury have obscured some of the wrinkles, basically kind of like gravel hiding ruts in a road. And when you account for that, you get a different estimate of how much it shrunk. 10% to 30% might not seem that important, but it is important because it tells us a bit about what the insides of Mercury might look like. You know, more shrinking could mean the metal core is larger. It could mean that Mercury started at a higher temperature, which is all important, you know, things to know more about,
28:58because really we know very, very little about Mercury. Well, Inside Sciences, that's the end. We may have taken you a little bit down and into the gutter, but we were at the end looking at the stars. We'll be back in a week here on the BBC World Service. Until then, it's goodbye from me, Tom Whipple, and it's goodbye from Caroline. Goodbye. Get more with BBC Podcasts wherever you listen. Be the first to listen to your favourite shows like Evil Genius, Good Bad Billionaire, and You're Dead to Me
29:32with a subscription to BBC Podcasts Premium on Apple Podcasts. You can also enjoy a range of our podcasts ad-free with an Amazon Music subscription.