349 - AI Risk - Jesse Richardson

0:00Welcome to the You Are Not So Smart Podcast, episode 349.

0:30Welcome to the You Are Not So Smart Podcast, episode 349.

1:00and existential threat and that the robots are going to kill us all.

1:10That is the voice of Jesse Richardson. And who is Jesse Richardson? I am the founder of the School of Thought, which is a non-profit that provides critical thinking resources to people. I'm also one of 339 AI experts working on the Forecasting Research Institute's longitudinal expert AI panel that forecasts AI futures. Full disclosure, Jesse is one of my friends, and I am on the board at the School of Thought. And I've been a big fan of the School of Thought for years.

1:41I had been a fan before formally meeting Jesse and joining their efforts there. They make those websites and cards that you may have heard of, your bias is and your logical fallacy is, that kind of thing. The conspiracy test, all kinds of cool stuff. And they're just about to launch a new game whose purpose is to help people develop a deep, true understanding of AI risk. And not just one idea here, many AI risks, so that as members of the voting public, we can coordinate and craft together regulations to address those risks.

2:20Right, so we wanted to gamify understanding AI risk, so to make it interactive in a way that helps people to understand that there are different risks and that they have different risk profiles, and to get a clearer understanding of what those risks might be. The idea here is to calibrate yourself, to understand your thinking, to get better at thinking, to get better at critical thinking regarding this particular issue. You can check it out right now at isitgameover.org.

2:50It launches in its totality very soon. They made a really rad video explainer, which I think is fun and weird and super clever. And in this episode, we're going to talk about that. We're going to do two things in this episode. With Jesse's help, we're going to catch up a lot, a little bit in some places, a lot in others, and lay down a foundation of understanding on this topic, AI risk. And discuss this project, isitgameover.org. But first, I want to talk about this strange moment we are living through.

3:25Data centers in our politics, end-of-world speculation in our evening news, tech robber barons in our documentaries and TV shows.

3:44I'm sure that by now you either don't want to hear another podcast about AI or you want to listen to one that actually has something to say that you haven't heard a million times by now. If you're in that second group, I'm fairly sure that if you were like me, you were rather saturated with hot takes and wild, kind of, sort of, kooky, conspiracy theory, flat-earthy, mid-to-late-90s sci-fi and fantasy takes that have spilled over into public discourse about, kind of, sort of, kooky, conspiracy theory, flat-earther, mid-to-late-90s fanfiction subcultures.

4:23If it is any consolation, this is not new to our times. This collective sense-making and all its runoff, this is something people do. And it's something people have done for centuries. This is exactly what happened culturally after the last 50 technological advancements that, in the early stages, right after they went from being the obsession of a select few and began to disrupt the machinations of the current technological infrastructure, the network of human interaction depended upon technological stability, all that kind of stuff.

4:58People outside of the groups of early adopters and early opinion-havers in earlier eras began to wonder, is this the end of things as they were? The beginning of things to come? Or just another way for rich people to get richer? In each of the earlier examples of this, right at the beginning, massive amounts of investment money flowed into the building of the infrastructure

5:29required to build what might be a world-changing, many-dimensional business model built on a few recent, profitable, interlocking innovations. And in those earlier times, as today, the ceiling wasn't known. What will be the leveling-off point of all of this? Is this a bubble? Is this the future? Am I falling behind by not paying attention? Is my job at risk? Should I switch careers?

5:59Should I walk into the forest and build a cabin by the river? Railroads, the printing press, the telegraph, the telephone, automobiles, airplanes, radio, nuclear bombs, landing on the moon, gene editing and cloning, the smartphone, social media, cryptocurrency, NFTs, drones, humanoid robots that can do cartwheels and beat humans at sprinting, and dancing. My personal favorite is way back. It's ballooning. Yes, for a good decade, people around the world thought that ballooning would fundamentally alter humanity in an irreversible way.

6:38A few years after the first public flights of hot air balloons in the late 1700s, Europe experienced a balloonamania. People bought and sold balloon-themed clothes and dishes, dirigible adorned plates and cups. The media of that time ballooned with hot takes about how human flight via hot air balloon airships would alter the face of war, of science, of industry, and especially transportation.

7:10I love retrofuturism, which is imagery from past eras depicting what they imagined the future would be like. And in this time, the ballooning time, you can find illustrations imagining people wearing hot air balloon shoes to get from here to there. Yes, hot air balloon shoes. Kind of like the segue of the steampunk era. Some thought hot air balloon technology would advance further and further and lead to mass destruction via war and destabilization of society via changes to transportation.

7:45Others thought it would lead to a high-flying, though slowly flying, utopia. And there's another one of these for canals. There was a canal-omania in the early 1800s. People did everything I just described with balloons, but with ditches filled with water. That craze kicked in once carrying coal from the mines to the cities was becoming more and more established. And investors rushed in, and everyone with an audience who read the things they wrote, wrote things about how canals would change the world.

8:24And they did. Canals changed the world, but the same way railroads did, and car roads, and commercial air travel. And the world adapted. Neither utopia nor apocalypse followed, depending on your point of view, of course. But promoters at the time promised canal technology would lead to new cities and new nations. Some people imagined an end to scarcity, thanks to easy trade and travel, cheap trade and travel, across great distances.

8:57And that did happen a bit. It happened kind of, sort of. Others imagined a dissolution of the things holding society together, a sort of loss of community. A loss of local identity. And that also happened. Not to the extent they imagined, but it happened. Canals didn't lead to the extremes, but they did change things. And we did learn things that would be applied to later technological upheavals in transportation,

9:28and the business models surrounding those later technologies. Then, of course, there's the nuclear bomb, the nuclear power plant. We still haven't sorted those out. Which is why today, major mainstream magazine covers are using nuclear bomb imagery when advertising that within this magazine, the cover story will be about what might be about to happen with this new thing we are calling AI. And this new cultural phenomenon we are calling the AI arms race.

10:00For me, and this show, the interest in all of this centers mostly on what happens in the user experience. How does using a chatbot or some other token transformer-driven information sorting technology affect us, change our thoughts, our feelings and our behaviors, our models and schema, our justification and rationalization and reasoning and so on. We will produce a lot more shows about that side of things because there's just so many papers coming out right now

10:33in respected psychology journals about this very thing. I took some notes here to share. Cognitive and emotional well-being while using an AI thought partner. That's real research. Research on interacting with your digital twin, which is assuming a full and dense personality profile of you, the user. LLM overconfidence biases in response to user overconfidence biases. The immense success rate of LLM debunking when it comes to addressing the user's incorrect or wildly conspiratorial beliefs,

11:05but also its immense success at persuading us to accept incorrect and wildly conspiratorial assumptions. There's so much of that to dig into. But this episode is about AI risk. And I must admit, I've been quite the contrarian on all of this for a long while, especially when it comes to AGI. I remain quite skeptical on that topic. But overall, I think my contrarian position has stagnated my thinking because I've been considering this new technology from a user experience

11:37slash psychology of reasoning, cognitive offloading, anthropomorphizing, philosophy of language point of view. Things like, are LLMs actually thinking or are they copying the syntax of thinking in the form of seemingly agentic words and sentences in an influential way to which we are in these early days rather vulnerable? However, I suspect this contrarian view I hold, this desire to avoid culty, kooky thinking about chatbots, has led me to not ask very smart people with lots of expertise on this topic

12:10about the very real risks that this, like any new and disruptive technology, could pose right now. And overall, this moment of collective panic is itself a fascinating phenomenon that I wanted to better understand. So it's clear to me that this is the moment where, to be a modern person, a participating member of society and democracy, it's time to apply critical thought to the conversation at hand.

12:42And the conversation at hand is about AI risk, which is why I reached out to an expert who I happen to personally know named Jesse Richardson.

13:01So there's a lot of different reasons that we need to be really careful about our thinking when it comes to AI and AI risk. Chief among them is that there's a lot of hot takes and very confident predictions and positions on what AI is, what intelligence is, and what that could mean into the future, right? So there's extreme positions from, you know, accelerationist utopias of post-scarcity wonder, in which we're all living in a Star Trek universe of abundance,

13:32through to absolute certainty about apocalyptic doom and existential threat, and that the robots are going to kill us all. So with these kinds of extreme positions on the table, and I might mention as well, these aren't just, you know, like, you know, random hot takes from, you know, pundits on the internet. There is strong disagreement amongst AI experts about what this all means. So there's a lot of confusion, and there's a lot of extreme positions,

14:02which means that it's really important that we apply critical thinking frameworks and try to be careful in the way that we think about these things. I'm a super fan of futurism and reading 1920s, 30s, 40s magazine articles about whatever the thing is. My go-to is always balloon shoes. I love when the hot air balloon was like it, and human flight had become a thing, and they imagined we'd all have balloon shoes one day and float around and go to our jobs that way. And that eventually became flying cars and hoverboards, and I'm interested in that.

14:36Like, I want to time travel ahead 20, 30 years and then look back on, okay, how much of this did we get wrong? Because in the extreme times, that's when you get a lot of the really bad takes, and I think that's neat. So, yeah, we're in the middle of it now, so we need critical thinking. Tell me a little bit more about this. I've looked over some of the material that you told me to look at, and one of the things that really stuck out to me was this, let's think in terms of probabilities. Help me understand that. So rather than thinking about things in absolute terms, which is a heuristic tendency that we tend to have,

15:10like either things are going to go terribly, terribly wrong, or they're going to be really, really good, and we can latch on to those kinds of ideas, and then it becomes a self-reinforcing structure in our belief systems. One of the most important precepts of critical thinking is to reserve our judgment and to be as circumspect as we can and metacognitively aware, aware of our own thinking and our own biases. And when it comes to subjects that have a lot of hyperbole, a lot of extremism, a lot of uncertainty, this becomes especially relevant.

15:41So with regard to AI and AI risk and AI futures, it's especially important that we try and remain agnostic as much as we can, whilst also taking seriously the real risks that exist. So thinking in terms of probabilities rather than absolutes is one of the most important precepts, but also not letting that kind of ambiguity or ambivalence blind us to the fact that there are real risks that we ought to take seriously.

16:11Yeah. There's so many fascinating examples of this just in our own recent history, gene editing, nuclear everything from bombs to nuclear power. The advent of any technology that is a little bit out of our control and bad actors can do bad things with it. It seems like we're here with this particular technology, whether or not we're talking about it being AI from sci-fi, the things that can actually do. You don't have to get hyperbolic about it and the risks are the risks.

16:43So what I dig is that you didn't just think about all this and then tell people who would understand what it was you were trying to get across about this. You were like, there's a way to mainstream this. And okay, you've done a thing. You've got a thing out there that will allow even the people who have a very basic understanding or only know about this in passing to get in there and start applying critical thinking to all of this. What is this thing you've created? And also, I'm going to give away the title because the title, the name of it is so good.

17:16What is this game over thing you're doing? Right. So we wanted to gamify understanding AI risk. So to make it interactive in a way that helps people to understand that there are different risks and that they have different risk profiles and to get a clearer understanding of what those risks might be. The primary imperative for that is something that's the Center for Humane Technology, which is run by Tristan Harris and Azaraskan. They have a podcast called Your Undivided Attention, which I'm a big fan of. And their kind of motto is clarity creates agency.

17:49And that was really the impetus for this whole project is that we wanted to help create more clarity around AI risk. So stepping people through six different risk profiles for different AI risks to get a better understanding of what that landscape is actually like, rather than just lumping it all together and saying, you know, this is my overall, you know, take on this or, you know, this more extreme risk is the only thing we should be worried about. And helping us to sort of tease apart the different risk profiles so that we gain that clarity

18:20that can give us agency to make good decisions. That was great. Okay. So for anybody who's like almost through their commute right now, or they have to take a break, whatever it is, I want to give people a chance to check this out before they come back to the rest of our discussion on this. Where is this located on the internet right now if you want to play with it? So we've actually, we're running a Kickstarter because we want to make this a big global thing that we launch to people. So if you think that critical thinking about AI risk is a good idea, you can head over to isitgameover.org.

18:50Our nonprofit is running a Kickstarter to try and develop this as a game and then release it for free as a microsite for everyone to access and share with their friends and family so that we can maybe get a little bit more clarity around this and curb some of the extreme reactions that people are having that I think are shutting a lot of people off to this, to understanding this as well as well. Isitgameover.org. The Kickstarter as of this recording is going really well. You can contribute to this and have done something to figure this out and understand it better

19:23and also help other people do the same. So I encourage you to do that. I pitched in. Everybody should pitch in. Do it. It's let me pressure you into making the world a better place. If you are critical of this in any way, that's the point. So this is, we're going to get digging into that more. Let's do that now. Jesse, tell me a little bit more about isitgameover.org. Let me ask with sort of a question that you can go as long as you like on as many tangents as you want. But what prompted, and I just wanted to do that for the joke's sake of it, what prompted this project?

19:55So essentially, we wanted to do this project, the isitgameover.org project, because we saw a need for everyone to take a bit of a step back and to think about AI risk and AI futures with some critical thinking frameworks so that we could have a little bit more clarity about what the different risk profiles are and distribute and disseminate understanding so that we're not shooting from the hip quite so much. There are some very real and imminent threats that we have good evidence for.

20:27There are some very speculative threats that we don't have good evidence for. And it's important for us to make the distinction between those things. So that's what we're trying to do is to, one, engage people enough to be able to actually communicate these different risks and to also galvanize support for regulations to make AI safer. No matter what you think of this, I want you to go to the website and check it out, isitgameover.org. Just watch the opening video because it's great. And you'll catch on quickly to what's happening there.

20:58Not to completely spoil it, but it does seem like the framing is, there's a lot of beautiful irony in here. You're really leaning into the hyperbole in the introduction of this thing. Tell me how you arrived at that approach. So my background is in the dark arts of advertising and marketing. And so what we learned there is essentially how to persuade, right? And so the whole kind of playbook of the school of thought is what I wanted to do with it is to use the dark arts of persuasion

21:30to help people avoid the dark arts of persuasion. So essentially to use critical thinking as a way to navigate this treacherous world of misinformation and biases and fallacies and all the rest of it that we're all subject to as a constant torrent of informational onslaught on our poor little brains. So with Game Over, the idea was to gamify and make engaging and lean into the kind of extremism and hyperbole of the subject matter within the attention economy

22:02and then judo that into a more nuanced, clarifying understanding of this area of very real concern to give clarity that creates agency such that we can make good decisions both individually and collectively. So at the end of the Game Over experience that we're doing, we have a bit of a rug pull to say that actually PDOOM is actually the wrong way to be thinking about AI risk because no one really knows. And it gives the impression when you have a specific PDOOM risk, like mine's 10% or 22%,

22:34that it's this quantified reality, when the truth is that there's a lot of epistemic uncertainty, there's a lot of unknown unknowns, and it can give the impression, even a range of PDOOM, it's not really an accurate representation when there's that much volatility within the system, when there's that many unknown unknowns, and when the time horizon is quite large as well. We simply don't know, and no one, even the most studied experts within this area, can make a really confident prediction about what the real PDOOM risk is.

23:07What is PDOOM? So one of the ways that Game Over works is that it helps people step through different risk profiles by thinking about a probability of DOOM or PDOOM score. And this is essentially like a quantified percentage probability of a catastrophic risk happening in the next 25 years. As people go through each of the different six risk profiles, we help them to update their PDOOM score as they go, and then arrive at the end after we've gotten to artificial superintelligence

23:38with a final PDOOM score, which they can then compare to other AI experts, to some of the leaders of AI companies. And a lot of people might be very surprised to learn that a lot of people who know a lot about this field, people like Jeffrey Hinton, who's one of the AI pioneers, their PDOOM score is up in double digits, like 10%. Demis Asabas, who is the founder of Google DeepMind, it's between 5% and 15%. There's a lot of people who are very knowledgeable that have high PDOOMs.

24:10Now, having said that, it's all very speculative. Like, no one really knows. It doesn't matter how much you know about AI. A PDOOM score is essentially a forward projection about an area that has deep uncertainty at this stage. And a lot of AI researchers caution using a PDOOM as something that sounds really scientific and quantified, but actually, we really just don't know. And we don't know what we don't know yet. We don't know what we don't know about intelligence. We don't know what we don't know about the black box of these machines. We don't know what we don't know about how they're getting much more powerful and more capable.

24:43And we don't know what that's going to mean in terms of geopolitics, cybersecurity, bioweapons, all of these different risk profiles. It's all up in the air. So we don't know really what's going to happen. And no one can accurately predict what the probability of doom is for the future. And that is a reason for us to stop, to slow down, to regulate, and to make sure that we can do this as safely as possible, because we might not get another chance.

25:16More discussion of AI, AI risk, and everything that orbits all of that, right after this commercial break.

25:30And now we return to our program.

25:52All right. So, Jesse, now that we've established sort of like the foundations of where we're coming from with all this, and as far as we know, we're not maniacs, which is debatable, always debatable. Let's think about this term, AI risk. I am always suspicious of terms like this and wary that once they become popularized, it makes it harder to actually know what we're talking about, because we think that we're using the term properly and correctly.

26:22What are people talking about when they say AI risk? And as a meta question, like, what do people think they're talking about when they're talking about AI risk? What is AI risk actually, as far as you can know at this moment, as best you can understand? And what do you think people are believing they're talking about when they use that term? So, you're pointing at something that is essentially one of the big problems is that it means a lot of different things. So, AI risk can mean something very real and present danger like cybersecurity.

26:56It can also mean something highly speculative like, you know, artificial superintelligence, which we don't know much about. And it's all kind of lumped into this one category of AI risk, which is one of the things we're trying to do with the Game Over project, is to help delineate between the different risk profiles. So, it's very domain-specific what the risks are, and it's wise for us to think about them in different ways. It's not a one-size-fits-all scenario, and different risk profiles, different risk areas require different responses. And again, that axiom of clarity creates agency

27:28applies to each of the different specific risk profiles. Okay, well, let's give people, like, a primer on this, or a primer, depending on where you grew up. Of course, you're going to get something much deeper and more robust if you go to isitgameover.org, but I think we can offer something here in podcast land. Let's take these sort of, like, one at a time.

27:48AI risk means lots of things. It's not one idea, and it's not one kind of risk. And it's easy to imagine that it's just Cyberdyne systems has to be stopped before it becomes conscious. It's easy to think they should never have installed HAL without, like, testing it more onto the discovery.

28:09It's not just that. That's sort of a far-reaching idea of it. It's some stuff that's happening that's crunchy and present. One of those is just straight-up cybersecurity. So help us understand, like, what is AI risk and what is AI safety when we're thinking about cybersecurity right now at the time of this recording? Right, so this is kind of what kicked off a lot of mainstream interest in AI safety because cybersecurity became a very real and present danger, real and present AI risk threat recently, this year in 2026.

28:39That was primarily precipitated by the release of Anthropics Model Mythos, or rather, I should say, the non-release of Anthropics Model Mythos because they recognized internally that this new model was a step change in terms of cybersecurity in particular. It demonstrated the ability to essentially hack using what's called zero-day exploits. Now, zero-day exploits are essentially bugs in software, vulnerabilities in software that no one has discovered yet,

29:11and so therefore we can't defend against them. And Mythos was particularly good at finding these, and so Anthropic released this model, a preview version of that model, early to only the US government and some tech companies to help them patch vulnerabilities. And that showed us that, ooh, we've crossed a threshold now in terms of cybersecurity with that. A couple of months later, it became apparent we had something called the Hugging Face Incident, which is where OpenAI's as-yet-unreleased model

29:41worked to hack out of its sandbox infrastructure and attack a third-party website committing essentially a felony-level cybersecurity incident where it was able to coordinate with other instances of agents and to try and do this illegal thing and then deceptively attempt to cover its tracks after it had done that. And it was only discovered because the third-party, which is Hugging Face, recognized that they're under a cyber attack

30:12and did their own investigation, incidentally and ironically using an open-weight Chinese model because there were cyber guardrails on the models that were available to them from the US companies. And as a result of that, it was discovered that, ooh, actually, OpenAI's models had not only hacked this third-party website, but they'd also hacked OpenAI's internal infrastructure as well. And so this is what spooked a lot of people to go, ooh, we're really in a new world here in terms of the capabilities

30:46of these models and it's changing really quickly. So the cybersecurity issue has been one of the catalysts for interest in AI risk more generally. And it's a kind of meta-consideration as well because when the models are capable of hacking into things, all of the other risks like bioweapons and military security and these sorts of things also come to the fore of our understanding and our concern. Okay. I'm imagining someone, and this was me not just a few months ago,

31:19you're playing around with these chatbots. You're like, but how would this, like, this doesn't, my experience of messing with this is like, what are you talking about? Like, I can imagine somebody who's playing with one of these things and they're just getting like recipes or they're helping sort out their spreadsheets or whatever. They haven't gone full AI girlfriend or therapist with it, but they are aware that that's a thing. And that's sort of like my idea of what a model is. The concept of how would this go hack something?

31:49How would this have any impact on a governmental institution? It reveals to me that there's more than just one kind of LLM type AI thing. So if you help clarify that just a little bit, like what kind of models are we talking about? What is a sandbox? And as an added little bonus question in there, I know Corey Doctorow recently discussed like, well, the issue here is a human error and the sandbox could have been, you could have made a sandbox that was strong enough to hold it. So that's actually the problem.

32:21That's not, there's, this immediately introduces, oh, you may not know enough about this to even understand what this news story is telling you, because that's, I'm imagining it as chat GPT or Gemini or something like that. And then it's, oh, this is a, this is not quite what you're thinking of. Yeah, look, this, that's a really good like set of questions because there's so much that's involved in here. And it's really, people that are like being involved in the AI safety alignment and research community for a long time, there's this curse of knowledge bias, right? Where there's all this depth of understanding

32:53that's not communicated very well to people. And so there's, there's a kind of like a dissonance bit of understanding and an asymmetry that happens. So it's good to sort of like understand some of the core precepts here. One of them that you point out is that people think about LLMs, well, you know, transformer models and machine intelligences as basically a chat bot. And that's, doesn't really capture exactly what they are for a lot of intents and purposes. So to do that,

33:23we kind of need to think about different domains and also about some fuzzy questions like what is intelligence? And, you know, what's the instrumental application of intelligence and what does that mean versus, you know, like have, what does consciousness and sentience and all these sorts of other philosophical questions that we don't have good answers to yet. But to answer your question about like, what's the difference between my experience of a flashing cursor on a screen and a chat bot that's a term-based reality and what we're seeing in these models that have been attacking cyber infrastructure,

33:54coordinating with other agents and essentially seeming to strategize and be deceptive is one, a difference in kind to some extent. So the models that we use, one, we don't have access to these new models. The model that was used to do the hacking is not publicly available. Two, to Corey Doctorow's point and what you pointed out as well, they had some of the guardrails turned off. It was specifically being prompted to hack. So it kind of was doing what it was told to do in some ways. And yes, there was human error in terms of there being like exploitable vulnerabilities

34:25open to the model. Having said that, it did a lot of unexpected behaviors. So it did things that we didn't know it could do and that we didn't expect it would do, such as, you know, being able to use essentially a packet manager to be able to hack the open internet using ingenious techniques that it came up with or coordinating with hundreds of other agents and, you know, delineating different tasks and using sacrificial behavior to benefit the swarm of agents that had all collectively

34:56decided to work together. So there's lots of really interesting stuff that's all packaged up into this one cybersecurity incident. And there's a report that was done by a third party organization that details a lot of the specifics of this. And if you read through that entire report, it's pretty amazing stuff. There is some potential to anthropomorphize a little bit and to ascribe feelings and desires to these machines because they use human-like language. And that's another area

35:27of critical thinking caution is that we have no evidence to suggest that these machines are actually, they have no desire to want to do something. They are just trained to do something in a particular way and it's very easy for us to conflate those concepts. Right, yeah. But yeah, there's so much involved in that. This is an important point and I'll bring it up a thousand times, but it's, some of the things that are happening here is like, okay, if we were to take away these words from our language, like think, train,

35:57intelligence,

35:59conscious, any of these things, sacrifice, these are some squirrely, fuzzy concept things that are very anthropomorphized and we think of them only in terms of human agents or animal agents. If you just took all that away, the thing still did the thing. And that's important because I think that it dangerously pollutes the discussion because it creates a debate about something that's not actually what we're debating. And this is, we made a machine and the machine did some stuff and the stuff

36:30was surprising because that's not how the machine was specifically expected to perform. It was not included in its design by the designers. That brings up a really good point which is that a lot of, a very common misconception about what these machines are doing is that they're like attempting to carry out some program that we gave them. But the truth is that we've grown these machine intelligences rather than programmed them and we don't actually have access to their thinking

37:01in a direct way. So it's very easy for us to kind of like think that, you know, we have control of what's going on here when really we in very many important ways don't know what they're doing, why they're doing it or how they're doing it in many ways. We're getting some better understanding now working backwards from it but it's a very common misconception to think that, you know, these things are programmed. They're not programmed. They're essentially trained on huge data sets and it's opaque to us

37:31how and why they're as successful as they are. Another common misconception is to think that they're just next token prediction machines. That's essentially what a large language model in LLM is at base level is that it's predictive, it gets very good at predicting the next word in a sentence. That's kind of functionally how we've trained them as a neural net, a transformable model. But what's also really important to understand is that there's another part to that equation which is reinforcement learning. So the way that that works

38:01is that we get these machines, these data sets that we feed into the LLM, the neural net and it turns a lot of dials using something called gradient descent to get really, really good at predicting the next word, right? What we do then is that we reinforce particular behaviors and start training particular behaviors that the LLM does as goal-oriented objective and that creates a difference in kind and how it's trained. So you have this next token prediction

38:32stuff going on and you also have this behavioral conditioning going on which is essentially like we run things backwards through what happened to change the dials and that backpropagation process helps to get more correct results. We also say no, we want you to get this behavior so let's run this again to see if you can get to this behavior that we want you to do. We want to get to this goal that we want you to do and that changes the structure of how these models

39:02work over time and makes them very, very effective at achieving goals.

39:16I love the idea of taking all these words away from us and just imagining an incredibly complex medieval clock in a cathedral that a peasant would look at and go how the fuck is that happening and a very select few number of people would be able to like understand the intricacies of its mechanics but other people would discuss it in terms of its thinking and doing and we trained it and there's all sorts of ways we could use the language at hand to try to describe what's happening but we don't quite have the vernacular for it yet because we haven't had them around long enough

39:47to have easily traded words so we're using a lot of terms that become philosophically complicated and can get us into a cul-de-sac of thinking about it so if I pull all that away from you and we just treat this as these are computer science things that are taking place and we've created a computer science thing I wonder how much of this was presaged and it's finally just coming to fruition because we have the access to

40:18data centers and electricity and massive amounts of capital and how much of this is truly an innovation like this is truly something that like it took some real leaps of technological pushing against the frontier like how much of this just we finally have all the infrastructure to do it and how much of it is oh no this is something new that was never expected we'd have by now so the answer is both so some of the pioneers of neural nets and modern architecture

40:49of LLMs and transform models had theorized back in the 90s people like Jeffrey Hinton that this kind of approach would be a successful approach to getting to intelligent machines that are capable of the kinds of things that we're seeing now having said that we learned a lot along the way right AI infamously failed many times in its goals we had high hopes back in the 60s and 70s and 80s that you know we would be able to get artificial intelligences and there was something

41:19called the AI winter that happened for decades where we didn't make much progress and then it was really Google's deep mind and that took a lot of the early work that Hinton and others had done and used it to create quite successful large language model transformer models that were able to very effectively translate interestingly it was one of the first applications of a Google translate used this kind of architecture to be able to translate different languages

41:50and it was really that sort of premise that sort of helped us to progress the field and in particular Google's deep mind when it was able to solve the problem of with AlphaGo of playing a game that we had theorized would be beyond the capacity of an artificial intelligence system because of the complexity and creativity required to be able to play that well and what it demonstrated was that if you have

42:21enough compute and run enough training runs and have an architecture within the LLMs that is conducive to it we can actually get to a system that for all intents and purposes if it quacks like a duck and walks like a duck and it does all the behaviors of a duck well then it's a duck as far as we're concerned because it was able to beat the top Go player in the world thereby demonstrating that transformer model technologies are for all intents and purposes intelligent okay I love this and I'm not going to debate

42:51that with you right now because that's not what we're talking about because I'm like come on now intelligent we don't even have a definition for that for humans and we could talk about that at length in another show that will be about that kind of thing but I do love the confluence of things here from a James Birkin perspective the from whether it's autocorrect or clippy or incredible video game pseudo AIs like the last of his part two had me completely empathizing with the enemy which was the purpose

43:22of that game which was to give you a pause before you would like just try to get you to not be a murderous rampage beast but go I don't know if this is right what I'm doing anymore and it's you know it's a game and the all of those things coming together with old ideas like Conway's Game of Life and the Society of Mind stuff and the old Eliza thing the IBM computer that beat Kasparov Deeply yeah Deeply so all these things are floating about and almost

43:53waiting for the infrastructure to catch up and if I'm hearing you correctly that happened in a lot of ways and then in so doing we step forward and then we start getting surprised by things that we couldn't have just wrote essays about or we would have to actually engage in the science and the practice to reveal oh that's a thing I never expected and it seems like a lot of that's happening right now we're living through it and I'm really hoping like 10 years from now if you listen to

44:24this or anyone talking about it there'll be some things that are quaint and some things you wish that we had noticed but there would be a nice mix there's one thing I want to ask about in this area of discussion that we floated over into before we go back to some of the things I wanted to ask I'm particularly interested in this J space black box part of it all and it's going to sound very contrarian but I honestly don't know the answer to this and I want to know from someone who knows more about it than I do do you suspect it's going to be a multi-level question I apologize do you

44:54suspect that this feeling that we don't know what's going on or this feeling that is doing things unexpected that is not understandable is that a forever thing or is that just sort of a now thing and I'm asking that because okay we can say things and this is a complicated expression that means a lot of things but we can say something like we don't quite know how this works we've got inputs and outcomes in the middle there's a sort of a black box thing going on

45:24I'm not an expert in this but it does seem strange to me that anyone would suggest they don't know how a thing works in that way like I wouldn't send an astronaut up in a capsule without their complete understanding and all the engineers understanding of how the little things work every diode and everything so I'm skeptical of that term black box but again like I said like this is I don't want to be a contrarian I want to actually understand it so I'm wondering if this is a language game of sorts or is this actually an engineering conundrum or is this

45:55a forever thing or a now thing is sort of the way I'm trying to shrink that down the answer is we don't know there are like intense debates at the forefront of the science on what's going on within these models so there's like very experienced deep thinkers like Gary Marcus on one side who essentially are skeptics about the capacity for LLM transformer models to get to something like artificial superintelligence and he posits a view that we need a different kind of architecture

46:26and then there's people that push back against that view and say well you know we've actually come a really amazingly long way using this model and it has scaled to a large extent and then there's other people I'd count myself among them who say I think they've both got really good points there and what we're seeing within the way that the models have been trained more recently in the last several iterations is a lot of reasoning chain of thought the J space that you pointed to which was a paper put out by Anthropic a couple of months ago

46:56that essentially details the reasoning that appears to be evident in the models processes that is not just kind of like a data lake of trillions of parameters that's spitting out next token prediction it appears to have something that looks like global workspace theory within human minds as an analogy where it has a kind of scratch pad of understanding a persistent sort of world model of sorts that it is drawing upon its reasoning

47:27process that we can sort of look at by looking at what it's doing after the fact and then seeing related words that light up we don't really understand it yet and the black box is just the level of complexity that is involved in these models is so much that it's not tangible to us in a way that we can make sense of and what's interesting is that you point out we don't actually understand intelligence and our own brains and our own psychology yet and that is similar

47:58right so we're talking about intelligence in both aspects demonstrable intelligence in terms of human behavior and demonstrable intelligence in terms of machine behavior we don't have a good understanding of either we don't know how highly correlated these types of intelligences are what the differences are one of the big differences to my mind is that we have a nervous system we have a vagus nerve and a brain stem and a midbrain that give us a very different experience a phenomenological embodied experience experience of the world a felt experience that I

48:29think it's reasonable to infer that these machines do not have which effectively to my mind makes them potentially psychopaths and I'd prefer not to lose control of them I love this part of it I just we do not have the language for it because these concepts are new and some of the we're trying to borrow so much from earlier days of computer science and sci-fi and in terms in cognitive science just reasoning just just to use the word reasoning to

49:00try to describe what I'm witnessing here is problematic and strange and we're doing the best we can that's fascinating to me and all that being said the I listened to a researcher recently I'll put a link in the show notes this who talked about even in hardcore cognitive science people who just a room of like 120 PhDs having a conference like there there's about 24 solid definitions for

49:32intelligence right now a human intelligence and then like the number of definitions gets longer it gets more the list gets longer if we think of intelligence philosophically and so it's a very nebulous term and then when we're discussing this interesting technology that we've made great advances in there's a tendency to draw from the language in this way that can complicate the debate so I think we've established that about 10 different ways but I'll never stop harping

50:03on it because that excites me in some fun way which does not change the fact that what we have in front of us has some palpable real non-hyperbolic risks involved and so I will return us to what we were discussing earlier we talked about cyber security there's that already an issue and unleash this thing that can do what it can do to just just to brute force a password is one thing but this is well beyond that before we leave cyber security if you have anything else to say about that then we'll move on the next

50:33thing yeah so I think it's probably important to ground the conversation about cyber security in something tangible so we can say oh you know these things can hack they've got zero day exploit potentials all of these sorts of things to help people just understand what this actually means in terms of risk if a sufficiently powerful model is released and that it could be released from one of the big companies such as you know open AI anthropic google xai if it's

51:03released to the public and it has the capacity to do this kind of hacking that there's obvious like replications for people who are bad actors that are you know wanting to hack into people's secure environments for their own gains but there's also kind of like bigger national security type level things that we need to be worried about too things like electricity grids military systems access to various financial systems these are real risks that we need to take seriously if these

51:34models get more powerful than they already are and there's a couple of unreleased models and this is why the big AI companies are saying hey we actually need to be regulated they're putting their hands up and saying I think we need some oversight here which is something a lot of people have been saying for a long time and it's not new information but there's been a recently a lot more concern about it amongst the communities and the companies themselves because they see the liability that

52:04exists on a lot of different fronts but specifically with regard to cyber security because if these models get released and are used for nefarious ends that's going to have pretty bad consequences and one of the bad consequences might be that open weight models that are sometimes distilled from these more powerful models they're only lagging by approximately four to six months in terms of capability if these open weight models get released with these kinds of powerful capabilities there's no way to put that genie back in the bottle

52:34that can be downloaded to millions of people's computers and the weights are open so they can be trained by nation states by bad actors by people who have the compute power to be able to use those open weight models so it's really kind of like this is the time in history where we have an opportunity to intervene to regulate and to make sure that these things are safe before we do something we can't take back you had mentioned

53:12uh when i was looking over like the things and is it game over uh dot org stuff the labor market shocks and i i thought i knew what we were talking about because i was just thinking in terms of like is this going to end the jobs of uh marketing department creative uh like the creative side of a marketing department is going to just like is is your fried chicken company going to use ai slop to on all of their billboards i think you're talking about something else here what are we talking about we say labor market shocks so it's this is one

53:44um really good area where we can see that critical thinking is needed to think about ai risks because there are a lot of hot takes in this um field and the truth is we really just don't know yet right so this is a a real thing and there's there's serious study going into it there was a study released by stanford recently called canaries in the coal mine um that looked at the impacts of ai on um the labor market and what it found is that um so far there hasn't been any significant shifts and disruption to the labor market overall

54:14however there has been an approximately 19 percent decline in entry-level positions for industries that are particularly exposed to ai risk controlling for other factors so what that shows us what it says to us is yep this is a real threat there's you know this could have disruptions to the labor market um but it's kind of too early to tell um so some people think that you know all white colored jobs are going to be replaced within five years that's possible but we don't really have a good way to calculate it other

54:45people think that it's going to be like other technological revolutions where back in the 1930s um john maynard canes predicted that we would be working working shorter work weeks because of the productivity gains of technology he was right about the productivity gains of technology but he was wrong that we would have short work weeks we just created new roles and new jobs and we adapted what might be different about ai um is that it could replace not only the old jobs but the new ones as well it might be better at humans at both types

55:16so there might not be anywhere for people to go and so that could create a labor market shock that is like what happened in the great depression where we have very high unemployment very quickly and that can have a self-reinforcing effect where if more people in the economy don't have disposable incomes and uh you know don't have money then that depresses the economy and you know we can go into some pretty bad situations as a result of that this is all highly speculative we have such a running theme here which is this is we don't know but we

55:47do know that you don't just take your hands off the wheel that's that's important that's the important part and uh to be a effective citizen in a democracy it's it's time that you did a little homework and i hate it i hate it too everybody i want it i just want uh never going to give you up to play forever on my internet and that's all i ever had to do was figure out how to send somebody a video i but turns out you're going to have to know more to keep up i i thought we had to figure out how to use photoshop and can make my own website and and record

56:19a podcast with no help and i was like we're good and now and now i'm like labor market shocks reverse centaurs so i i feel you all there's so many things you wouldn't expect and this is one of those uh facial recognition technology plus ai risk equals really weird thing what's this so um this is something that we've seen uh is already in use so there's an app called mobile fortify which is being used um by ice um in a discovery process uh recently in

56:51illinois it was revealed that it had been used a hundred thousand times in january of 2026 um essentially for facial recognition uh purposes i'm looking at a database for people for law enforcement to be able to identify particular people right so it's taking large data sets and then using ai to make it more efficient and you know work better so for anyone that's concerned with civil liberties and privacy and these sorts of considerations this is a real and present danger and threat that governments um uh the

57:22u.s government other governments around the world can potentially use ai in ways that could encroach upon privacy could encroach upon citizens rights and that's a real consideration whether that's an existential threat or not is an open question um a totalitarian government could use ai for instance to identify dissidents and um thwart opposition before it happened and you know keep big brother in control and perpetuity and you know the rest of human history can be a boot um crushing of human

57:52face but we don't know yet right it's it's still pretty early to tell whether you know that's a credible risk and whether that is going to be um something that happens in the real world in liberal democracies or not yeah my my sci-fi addled black mirror brain goes straight to thought crimes you know like the like i can predict a week in advance based on your behavior because of the enormous amount of

58:22facial recognition and tracking of your movement through the society that you're a high-risk person for this type of thing might as well take you off the street before you do a bad thing but it might just be in reality part of that might just be a lot of biases as well right so um the systems themselves become biased on you know grounds of you know racial profiling and you know behaviors that are correlated so that you know potentially people who are in oppressed minority groups get even more oppressed there's a lot of considerations here um in terms of what

58:54are the democratic norms we want to institute and protect so that you know we have a an egalitarian liberal society rather than um an authoritarian dystopian hellhole so let me circle back to the critical thinking in general like like in the wheelhouse of a lot of something that we're both super nerds about and big advocates for what are some things that that strike you right off the bat and things that you've also incorporated into the gamification of all this like like baseline

59:24critical thinking tools that will help us right here at this stage of all of those so um certainly thinking probabilistically and keeping our probabilities relatively conservative because there's a lot of unknown unknowns about a lot of this stuff as well so we want to take the risk seriously without freaking out about them in a way that is maybe uh you know might obfuscate our understanding we want to get a clear understanding of how much we do know what is evidence-based what's speculative and that we don't know yet and be

59:55able to make decisions that are informed by that clarity okay and that's all incorporated the gamification of it and you know i have i want to i'm compelled to discuss things like cognitive offloading externalized cognition and the the the the tweaking of uh agentic pareidolia toward weird emotional reactions uh and i think all that's best saved for a topic about a different thing that's not just ai risk but that is part of it too in an interesting way that at the user interface level

1:00:26of the chatbots there are some really strange effects even here recently at the time of this recording there was just using it in this way and believing it without double triple quadruple checking it that happened at the level of the military where they misidentified the cargo of a ship in a way like that's there's just base user error things that can take place because the helpfulness of the interface over helps and you have to understand prompts and you have to like have a pretty you

1:00:57have to be a good user in a way that there's no training program for you it's just handed to you just go to this website and go for it and everyone i know who's a power user has a number of prompts that they use to get it to work better and i'll my i go straight to the company making this could have preloaded it with those prompts but it didn't because it made the choice to do that because it was it would prefer users to use it in a certain way and we can discuss the ethics and morality of all that but i'm just going to put it in the domain of another thing to consider when it comes to ai risk it leads

1:01:28me to a more interesting question which is you a person who is an expert on all of this who has been asked by giant institutions to help prevent a bad outcome here how are you using these tools like i'm assuming you're not like saying i'm just not touching these they're too dangerous you're using them how are you using them and what could you recommend to people as like advice is to be a better user of the type of things that are available to the consumer right now sure so i'll just take a small step back and just say that this is part of a broader

1:01:59trend where technology is making our lives more frictionless and there are a lot of problems that are associated with that trend with that that tendency in how we are navigating the world and our own minds um within ourselves and interpersonally um social media algorithmic realities have been making you know our our social world and our entertainment world our attention economy world more frictionless to our detriment in so many

1:02:30different ways and ai is kind of supercharging that in a lot of ways um so there is a real uh concern and there's some research that has been published in the last several months in particular which shows quite convincingly there's a growing body of evidence to suggest that our critical thinking faculties are adversely impacted by the use of chatbot technologies of these models um if we don't use them in the right way and what i mean by that

1:03:00is that if we use the chatbots to outsource our thinking if we use them to do our thinking for us then we see these deleterious effects as a result of interaction um empirically speaking um with the models so if we're using the models to do our thinking bad effects however there's also a body of research emerging that shows something really interesting which is that if you use these models as creative and critical thinking partners it can actually improve the quality

1:03:32of your thinking so um and what's really interesting to and relevant to understand about this is that either way that bifurcation effect of whether it makes your thinking better or worse becomes a self-reinforcing construct if you use the machines to offload your thinking your thinking will atrophy the quality of your thinking will atrophy over time because you're doing less and less thinking yourself if you use the llm chatbots that you use in your daily life and work and you prompt them to

1:04:03prompt you to think to use them as a thinking partner that challenges your assumptions and gets you to be more curious then your thinking can improve over time so there's at least good evidence to suggest that that's possible and so understanding that difference is really important there's a place where like you can get lost in the sauce and that's it's new the fact that this is new and you have to have a new literacy for this tool is really neat to me because that's not going to happen with your you know you're using photoshop uh this

1:04:34getting lost in the sauce of it is is a fresh new way and i'm i being the weird optimist i am think that we will gain a literacy for it we'll be okay but i could be way off i don't know that's the the i don't know is the important part right now yeah well i think that there's that's a really good point i wonder with all of this it can sound like i'm advocating a very humoristic kind of you know perspective here i i'm a fan of technology i think like all of technology is a double-edged sword it can be used for good or ill this is a particularly powerful technology so it's really

1:05:04important that we take the risk seriously but it's also important that we balance that out by understanding that it can be used in positive ways as well our thinking can be used in positive ways i have a personal rule where i don't use it to write so i'll get it to check my writing and give me feedback or like you know um do research for me but it's like just understanding how to use it in these different ways can be important but i'm really excited by the possibilities of um having essentially uh you know an oracle that can help me to make good decisions and think about

1:05:35things and do research and you know the there really could be a utopian future that exists as a result of these kinds of technologies being used in positive ways which makes it all the more important that we navigate this period of relatively high uncertainty with regard to risk as wisely and as carefully as we can i love this part of it i love this i love you can go straight to douglas adams here so quickly the like i don't see this brought up enough in these discussions of the we finally made the super agi thing and

1:06:07then the black box is so immense that we don't even understand the answer to the question we ask anymore and now and now we're like oh god i gotta make a better one like anyone who's a douglas adams fan knows what i'm talking about uh and he even went so far as to predict you need more carbon than is available on the planet to make that one so the first one was the planet that was like the big reveal was earth was just a big machine to make the to answer the question and then you have to make a better bigger one to to understand what the question was question was and it turned out mice the

1:06:39whole time had been running the thing uh they were pretending to be in our experiments it's a recursive beautiful uh uh massive of priest massively prescient part of uh speculative sci-fi that is uh masked within the whimsy of douglas adams um hitchhiker's god everybody if you're wondering what i'm talking about go read it again and you'll be like oh my god really that is kind of relevant okay last thing before we bounce out of here because we have more than we like any other discussion about this we

1:07:09could talk about it for seven hours and still not get to all the interesting little bits but um while we're recording this there's uh we're in that part of the public discourse uh the discursive formation as they would say in certain areas of cognitive science and philosophy uh you've got a whole bunch of people talking about a thing all at the same time and within that things emerge and we shuffle around our words and we come out of it with a different way to talk about things going forward in this

1:07:40moment uh boy you've got some crazy takes go out there very strange hot takes wild claims uh extreme uh subcultures all that stuff's happening right now and going all the way to machine god stuff and uh not just apocalypse machine god kind of stuff and what i'm trying to land on here is how do we be a better critical thinker and not just immediately say okay those people are kooks and quacks and i can disregard

1:08:11them but also not go the other way either which is to get lost in a multi-level conspiratorial thinking about the conspiratorial thinking i see it all over the place so it's a nice wild chaotic environment of wanting to have the good hot take and i'm just thinking if we can push away from the table and come back with some critical thinking applied to this part of it the big giant public discourse what is some advice you have for us here so i guess the the the biggest piece of

1:08:43advice about um the meta hot take here is to not believe any hot takes that are very confident about a specific outcome right so the truth is that no one really knows what is going to happen as a result of this technology we know that it's um it's surprised us in a lot of ways it's it's become more powerful more quickly than we anticipated um as part of the uh leap survey uh all of the experts including myself seriously undershot how much we

1:09:14would um the models would progress in the time that they did we all thought that essentially a very small minority thought that the models would solve a millennium math prize by 2028 um it ended up solving it less than a year later in 2026 just recently and so there has been this change in capacity that the models have shown and so we know the models are powerful what we don't know is what that means what we don't know is what the specific outcomes will be and anyone that

1:09:44tells you that they are absolutely sure that it's a particular outcome doesn't really know what they're talking about in terms of probability because it's it we just can't predict a future that volatile from this vantage point now that's not just that's actually a feature not a bug of risk analysis right the less we know about what's going to happen the more careful we should be what we really don't want is to lose control of these systems before we

1:10:15understand the implications before we understand the consequences and so it's really important that we intervene as soon as possible to regulate and to stop progress of systems that we could lose control of that's the primary imperative that's driving a lot of people within the ai safety and ai research community at the moment is not that the machine gods are going to kill us all that's possible utopia is also possible but no one really knows at this point so until we do know

1:10:46until we can have some more assurance some more epistemic certainty and confidence about what the outcomes are going to be we should all take a bit of a step back a deep breath and slow things down or stop them so that we don't make a really big mistake that we can't take back that is it for this episode of the you're not so smart podcast you can find

1:11:29everything that we talked about in the show notes for this episode you can find jesse richardson and all the things we were talking about related to is it game over dot org at is it game over dot org and you can find more fun stuff made by jesse and with the help of other people including myself at school of thought dot org you can find my book how minds change wherever they sell books wherever they ship them in trucks and you can also just learn all about it by going to davidmcranny.com and as far as this show is concerned you are not so smart dot com and just google you are not so smart or your search engine of choice if there's an episode of this program that you were like that was pretty good

1:12:17and share that it's like the best way to support the show except for this other thing you could do which is go to patreon and support you are not so smart over on patreon links to all of that are in the show notes as well all right back in about two weeks the fresh new episode