Stopping The AI Safety Cult ft. Adam Becker & Cal Newport

Welcome to Better Offline

0:00This is an iHeart Podcast. Guaranteed Human.

0:05Hey, it's Kelly Rowland. You may not know this, but I have eczema, so I get how it can steal your time. But why let eczema take over when you can talk to your doctor about ebglis? Ebglis Lubricizumab LBKZ, a 250 milligram per two milliliter injection, is a prescription medicine used to treat adults and children 12 years of age and older who weigh at least 88 pounds or 40 kilograms with moderate to severe eczema, also called atopic dermatitis that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies. Ebglis can be used with or without

0:37topical corticosteroids. Don't use if you are allergic to ebglis. Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with ebglis. Before starting ebglis, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about ebglis and visit ebglis.com or call 1-800-LILLY-RX or 1-800-545-5979. We believe in starting with your financial goals, not a formula. At Oppenheimer, we put the full

1:11strength of our longstanding expertise to work, understanding your life and your ambitions, and designing the precise strategies that build and protect your wealth with confidence. across this generation and the next. Put the power of Oppenheimer thinking to work for you. Wealth Management, Capital Markets, Investment Banking.

1:34A big part of parenting today is figuring out which technology will help your kid and which might hurt them, especially when it comes to learning. Brilliant is the online math tutor that's visual and interactive. Unlike other tools, it never hands your kid the answer. It helps them think for themselves, with personalized support when they need it. It's helped millions of students excel in math, from fractions to calculus. Sign up today at brilliant.org slash tutor and get 20% off for back to school. That's brilliant.org slash tutor. Nothing quite pops like the fun freeze-dried crunch of M&M's Popped Caramel.

2:09Except maybe a pop concert performed live with you and a friend in the crowd. That's why M&M's Popped Caramel is teaming up with iHeartRadio, who's giving away a trip for two to the iHeartRadio Music Festival in Vegas this September. Enter for your chance to win at iHeartRadio.com slash MMS. M&M's. It's more fun together. Available at your local retailer. No purchase necessary. Ends 8-20-20-26. Must be 18-plus. 50 U.S. and D.C. resident only to enter. See official rules at iHeartRadio.com slash MMS. Cash flow crunch? OnDeck's small business line of credit gives your business immediate access to funds

2:40up to $200,000 right when you need it. Cover seasonal dips, manage payroll, restock inventory, or tackle unexpected expenses without missing a beat. With flexible draws, transparent pricing, and control over repayment, get funded quickly and confidently. Apply today at OnDeck.com. Funds could be available as soon as tomorrow. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval. This summer, Prime Video is the place for steamy romances, addictive love stories,

3:11and the book-to-screen favorites you've already read twice. The Love Hypothesis. L. The Devil's Mouth. Sterling Point. The Last Sunrise. And Off Campus. The Slow Burns. The Second Chances. Chemistry you can actually feel through the screen. It's all here. Obsession is in session. Get ready to obsess on Prime. This is Iris Palmer from Against All Odds with Iris Palmer. A flawless makeup look starts with the right foundation shade.

3:42Sounds simple, right? But finding your perfect match can honestly be one of the hardest parts of your routine. But there is good news. Sephora has all the best brands in one place with so many shades, finishes, and coverage options to choose from. Plus, their beauty advisors, shade matching tools, and free samples make it so much easier to find the one that's just right for you. Visit Sephora and find your match.

Introducing the guests

4:06CallZone Media. My name's Ed Zitron, and this is Better Offline.

4:20Better Offline.

4:23I am back, and I'm announcing that we will be speeding up the process of researching powerful and dangerous podcasting technology. My researchers are spending over $200 million a day on GPU compute, trying to create and generate a recursive self-improving Zitron, and no amount of pressure from world governments will stop me. But before my greatest work is complete, I must continue to record. And today, we've got an absolute barn burner. I'm joined by the clavículas of computer science, by which I mean Professor Cal Newport, an astrophysicist and author of More Everything Forever. Adam Becker, fellas, welcome back to the show.

4:55Thanks for having us. Yeah, I'm terrified of the idea of recursively self-improved. That means your newsletters will be 200,000 words long now? Is that what we'll expect? Yeah, they'll either be 200,000 words or 100 words long. Oh, maybe it's not the worst thing then. All right, keep going. When the perfect matrix of Zitron is complete.

Origins of the rationalist movement

5:14Well, today, though, this was inspired by your wonderful piece, Cal, in New York Times, about the rationalists. And we are going to start very simple. Who are these people and why are they suddenly notable? Well, I'm probably going to actually throw the baton to Adam here in a second to give the history of this group because he and his book was one of my primary sources in trying to understand them. But let me just set the stage for why I was writing about them. I don't think people realize just how weird some of the lead figures are who are at, in particular, OpenAI, Anthropic, and DeepMind.

5:52And I don't think people realize that until last week when you had a whole group of 20-somethings casually debating in their lunch hour whether it's 10% or 50% chance that we're all going to die within the next couple of years. And that, I think, caught a lot of people's attention of like, wait a second, what is going on over there in those labs? Why is this type of rhetoric only coming out of those particular labs? Why does Jensen Wong think that this is ridiculous? Why is China laughing at this narrative? There's something going on here that's not right.

6:23I think people are starting to wake up to this. And I think it really is this family of futurist philosophies that come out of the Bay Area that have mutated and split in various ways that are really defining our current moment and no one really realizes it. But Adam spent years, God help him, immersed in this world, has read everything they've read. He knows the history well. So actually, Adam, I'll throw the baton to you. Maybe you can give us a story of rationalism and effective altruism and where they came from and how they've split it.

6:53To me, it's a fascinating, odd story. Oh, God. I mean, my standard response when people ask me this is, you know, well, I could write a whole book about it and I did. And, you know, I'm not going to get into the same level of detail I do in the book. But, like, basically, this is something that was spawned out of transhumanist message boards and email lists in the 1990s. And what is transhumanist? Yeah, that's a good question.

7:23Transhumanists, I guess the shortest good definition I could give you is transhumanists are people who don't want to be human anymore. They want to use machines and especially computers to basically escape human bodies and solve the problem of death. Okay. Yeah. So those guys. Yeah. Normal shit. Normal shit. Normal way to do things. Yeah, exactly.

7:54Yeah. And so, you know, these are people who are really excited about the prospect of AI, the prospect of, you know, nanotechnology that could, you know, basically serve as a kind of magic to do whatever they wanted. This was the 90s. Nanotech was, like, the hot thing. And it was a bunch of ideas that basically come from science fiction. Right. Um, you know, they, they read too much sci-fi and confused it with reality and thought, you know, oh, well, because all of the sci-fi I read takes place in the future and all of the sci-fi I read has super intelligent machines in it and nanotech and, you know, people uploading their brains into computers.

8:43That must be what's going to happen in the future. If we just, you know, work hard and wish hard enough for it. And so, just so I'm clear, this is, this is a certainty for them. They're 100% sure this will happen. They're 100% sure that they could make it happen. Right. Oh, so it's in their hands. Yeah, yeah, yeah, yeah, yeah. Like, you know, they, they think, oh, yeah, this is going to happen unless something, you know, ends civilization first or, you know, nobody listens to us or something like that.

9:15They think that they're working on the most important stuff in the history of humanity, basically. They think that they are going to, you know, make themselves immortal. And when you say working on, that seems like a fairly broad thing, though, because a lot of these rationalist types don't actually work on the stuff. They work next to it. Yeah. Like philosophy people. Yeah. I mean, it's not even so much that they're working next to it. It's like they talk about it. They like to talk about the prospect of nanotechnology.

9:45They like to talk about, you know, super intelligent AI. They like to talk about these things. But, but yeah, so these people spent a lot of time doing that on message boards in the 90s. And there was this one guy who showed up in the mid to late 90s on one of the most sort of influential of these message boards, the Xtropians, which, you know, don't get me started on that name. It comes out of a misunderstanding of physics. Yeah, that's an annoying name. No, that's how you know this is a serious group of scientists.

10:17That's right. The Xtropians. Yeah, that's, that's correct. Yeah, that's exactly right. And they're like, yeah, we're going to fight the second law of thermodynamics to which my response as a physicist is, you know, yeah, good luck with that. But basically, this guy, this kid shows up named Eliezer Yudkowsky, who is a teenager at that point. Dropped out of middle after middle school, dropped out of schooling. This is like, that is dropped out. That is wild. Yep. Dropped out after eighth grade.

10:48That's correct. That's crazy, man. Yep. Um, and, uh, and yeah, Yudkowsky then goes on to, uh, write an enormous amount of, you know, his own deep thoughts on what he thinks the future of AI is going to be. Uh, and relatively quickly comes to the conclusion that, uh, AI is going to, you know, he, he takes an article of faith among these people, which is that AI is going to become self-improving and lead to a super intelligent AI at some point in the relatively near future.

11:29And lead to, uh, and lead to a singularity where everything changes and, you know, all of the stuff that the transhumanists believe is inevitable is going to happen. Uh, he comes to, uh, Yudkowsky then comes to the conclusion that, uh, such an AI might not be friendly to humanity and probably wouldn't be unless it's done properly. And the singularity is when the computer is smarter than us, just so that I understand. It's when, uh, it's really ill-defined. It's when the computer is smarter than us, or maybe it's when the computer can upload us into more computers.

12:05So we can all be computer and everything is computer. Uh, the singularity is when, uh, is when a lot of tech, but all the time and everything, like it's not a well-defined thing. Um, yeah. And this is the thing they obsess over the thing that's ill-defined. Oh yeah. Yeah. Yeah. Yeah. They, they think that that is pretty much inevitable unless, you know, we have like a, a nuclear apocalypse first or something.

Thought experiments versus engineering

12:33You know, this is worth, let's just pause on this for a second, right? I think it's really important because these ideas that I was talking about, so I'm very familiar to what people are talking about today. They were talking about this. This is about the early two thousands is roughly when Yudkowsky kind of had this change of heart of, um, the super intelligence, of course, is inevitable, but now he was worried about it instead of being excited about it. This was all thought experiment. Okay. I mean, this is, there's no LLMs there, but this was during an AI winter from the computer science perspective. Um, it was entirely inductive thinking directed by ideas from sci-fi.

13:08So they just sat there and thought about this, right? So these are not, this is not an engineer looking at a particular technical trajectory and saying, oh, I'm worried where this is going to go. This is people just thinking through the thought experiments of, well, if AI could recurse yourself improved, it would get super intelligent. What would it happen if you had a super intelligent AI and then just sort of exhaustively working out in their own minds, various scenarios about, well, maybe it would be friendly, but maybe it would trick us. And what if it did this or that? Um, and so I think that's important. So you have all these ideas are established well before there's any particular technology that people are worried about established entirely based off of at this point thought experiments.

13:46Yeah, I think that's, so this is, so this isn't through proximity to actual scientists or so nothing about witnessing the actual technology following other than an arm's reach. Yeah, no, I mean, it is exactly as Cal just said. And another way of phrasing that is it's, it's essentially a kind of extremely dry fan fiction that they're engaging in. They are writing, uh, you know, anxiety fueled scenarios based on these ideas that they got from science fiction, thinking that that's what the future is going to be.

14:25Do they believe it? Oh yeah. So they're sincere. They're just odd. Yes. Yeah. And at the time you can like dismiss them as sincere oddballs. What happens is, uh, Yudkowsky just keeps writing and writing and gets invited to contribute to some group blogs and then spins off his own sort of blog, uh, called, uh, less wrong, which becomes a community hub for people. Yeah, that's him. Um, but the other thing that happens is it's around this time, like the late two thousands, early 2010, somewhere in there that, uh, Yudkowsky meets Peter Thiel.

15:05Oh, and Peter Thiel gives Yudkowsky a bunch of money. How old is he at the time? Oh, we did this math. We did this math, Adam, with the fact checkers. Yeah. This is like 2000, the first decade of the two thousands, he was like a mid upper twenties. Yeah. Something like that. Yeah. Yeah, that's right. Huh? It's like a decade of him, a decade after he was just giving a child money. I mean, that's not out of character for Peter Thiel. But he literally pays people not to go to college, right?

15:36Remember that? Exactly. The thing is, though, there's like the Thiel fellowship where it's like, I'll pay you not to go to college to build something. This is just like, I liked your blog, kiddo. Pretty much. You think the world's going to explode. You better blog about that some more. They met at, I'll tell you if I get this right on, but they were at an event called the Singularity Summit. I believe that's where they met. Yeah. Or maybe that's where, it gets complicated. Go outside. They definitely get complicated. Yeah, it does. Yeah. But a lot of money enters the scene, right?

16:08Like, this is where, like, a lot of money starts entering this world. So, yes, yeah, yeah, yeah, yeah, yeah. And Jadkowski, along with some other, you know, transhumanist, futurist types like Ray Kurzweil, starts this, you know, set of conferences, the Singularity Summit, and, you know, just gets people together every year to talk about this stuff. And the community of rationalists on LessWrong just, you know, start, you know, gets bigger and bigger, and they start meeting up in the Bay Area, they start living together in group houses.

16:46What does rationalist mean? Rationalist basically means you spend a lot of your spare time reading Jadkowski's writing. Jadkowski thinks, at this point, that if you don't believe that the Singularity is coming and that AI is an existential threat to humanity unless you figure out how to do it right, he thinks if you don't believe those things, you're not thinking rationally. And so he decides to post a lot about how to think rationally.

17:17That's it? Yes, that's it. That's it? That's the entire rationality part? Yes, he writes. It's not about living rationally, it's just about their so-called rational belief that the computer might wake up. Are you fucking kidding? Well, they apply it to their whole lives. Yeah, exactly. That's why the two big blogs are called Overcoming Bias and Less Wrong. It's all about this sort of, like, really sort of tech-obsessive way of, I'm going to be super accurate in the way I think, right?

17:47And this is, I'm going to, I am not going to be overcome by cognitive biases, I'm not going to be tricked. And if we can just be, like, hyper Sherlock Holmeses, like, that's how we're going to unlock the potential of the world. But Yukowski thinks, like, the ultimate application of this is preventing the extinction of humanity by preventing superintelligent AI. And that, like, that is why we have to raise an army of, like, hyper-rationalist people. It's our only chance to survive the coming AI apocalypse. So, like, that kind of becomes his obsession, is that, like, this idea of, like, hyper-rationalism that he's the leader of is going to be, that's the John Connor of this Skynet scenario.

18:28Right, but they don't seem to be doing anything other than talking. Yeah, no, at this point, they're not. They're doing some things. I mean, they're in the polyamory. But, like, to be very, very clear, Yudkowsky is anointing himself as an expert on, you know, the art of rationality and cognitive biases and how to think. And, you know, this is an entirely self-trained, self-taught guy whose experience of the world is pretty limited.

19:00And he is posting hundreds of thousands of words about this on LessWrong and then starts writing as, you know, a combination of artistic outlet and recruitment technique. He starts writing literal fan fiction, a Harry Potter fan fiction called Harry Potter and the Methods of Rationality. Oh. Yeah. It's like 600,000 words long or something.

19:30Yeah, it's longer than War and Peace. I'm sorry, even I would never write something that long. Even I would, 200,000 words on anything is terrifying. I mean, this is 600,000 words. But what the fuck was it even about? Was it like? Oh, it was about Harry, who is, you know, I think fairly clearly an author self-insert of Yudkowsky. Harry being the specialist boy and finding ways to, you know, change and save the wizarding world of Harry Potter using his special logic and science powers.

20:11Through logic and reason. Yeah, exactly. Yes. Right. Well, why? I guess it's just creative. Yeah. I mean, I interviewed him for my book and that was what he claimed. And I believe that it started out as a creative thing. But, you know, it was also very clearly him playing with these ideas that he has about, you know, this is the only way to think clearly. And it did end up becoming, you know, quite a recruitment tool for the rationalists.

20:47And just to put a look, just make sure we have like the timeline beats clear and then Adam fact check me here. There's sort of this shift that happens where in the early 2000s, actually, I think his first money was, it was Peter Thiel, but it was also that other, that other internet entrepreneurship couple. They fund something called like the Institute for AI Singularity or the, what is it exactly? Oh yeah, the Singularity Institute for AI. Yeah, originally the funding comes from this couple, the name of which I don't remember, but eventually like a few years after that in like the mid to late 2000s, he meets Teal and Teal gives them like real money.

21:20Yeah. And this is where, so as the 2000s move on, so that Singularity Institute was superintelligence is going to be a beneficial godhead that's going to redeem history. Well, sorry. He shifts pretty, my understanding is like once that's a full-time job and he's thinking about it a lot, he shifts pretty quickly. He does that first big report. He shifts pretty quickly, but like, actually, wait, I don't think we'll control AI. We should not build it at all. It's going to kill us. Like there was this shift that happened like kind of quickly in that area and he shifted Valence and then eventually they renamed it to MIRA, MIRI, the Intelligent Research Institute.

21:55Yeah, that's correct. Yeah. So, so just, just to get the timeline straight here, Yudkowsky makes that shift from, you know, superintelligence is going to solve everything. And we should be aiming for it as fast as possible to, oh my God, it's going to kill us all. He makes that shift around the same time Singularity Institute for AI is founded in the early 2000s. Yeah. Right. Is there a, is there an antecedent or does he just one day wake up? He's just, I mean, he's just been on these message boards for a long time.

22:27And he also saw that like, you know, the, the nanotech guys who wanted a lot of the same things and made a lot of the same promises back in the eighties and nineties, people like Eric Drexler, they had their own sort of Institute. And Yudkowsky, you know, at this point he's, you know, unemployed and, and about 20. And I think he's, he's like, I want to work on this full time. Can I turn this into a job and finds a way of doing that by, you know, co-founding this Institute. But, um, I, I also, I want to, I want to just push back on one thing that you said, Cal.

23:01Okay. It's not quite that, you know, Yudkowsky wants to stop AI at all costs. He, he wants to stop AI at all costs until he and others figure out how to do it safely, at which point he wants to go full speed ahead. He still believes in the, you know, the AI utopia, if you can get a super intelligent AI to not kill everybody. So he just thinks that that problem is harder than he originally imagined. So he's like, this might take us longer. There's a, there's an impedance towards him being the thinking King.

23:36Yeah. Like the thought King that can tell the computer how to be best that loves us. That's basically, yeah, he will, he will be the, the right hand man to the digital God. But, but Adam, is this right? There was no technical antecedent for the shift in his thinking from actually, this might be harder. The best I could tell from your book and looking at other sources is that the shift in thinking just came from, as soon as he got the funding for the Singularity Institute that he was helping to run, he had this big writing project he took on where he's like, okay, I'm going to produce this big report on super intelligence.

24:08And it was in just the, the internal, you know, cognition of thinking through this report that he just convinced himself, oh, wait, I don't think we're going to be able to control this as easily as we think. Because I think it's really important to emphasize, again, this is not someone who has seen something in a technical lab that is scaring him. All of these developments in his thinking at this stage, it's really just, it's Yukowski in his own mind. The modern web runs on tracking.

24:39Every click gets logged, profiled, and sold, and that same data sets your prices and feeds scammers. Surfshark pushes back. The VPN encrypts your connection under a strict, independently audited, no-logs policy, and the suite around it blocks ads and trackers across every device you own. Secure your entire digital life, risk-free, with their 30-day money-back guarantee. Go to surfshark.com slash offline, code offline, for four extra months of Surfshark. All right, everybody, I'm back talking to you about Quince, one of my favorite clothing brands.

25:12And I've been feeling a little dangerous, which is why I'm eyeballing one of their cotton mesh stitch sweater polos when summer ends and the temperature begins oscillating here in New York City. I'm more so likely to pick up one of their cotton knit blazers for my next trip to the stock exchange and really can't recommend their stuff enough. Quince focuses on high-quality wardrobe staples made with premium materials like 100% Mongolian cashmere, organic cotton, and merino wool. Everything at Quince is priced at 50-80% less than similar brands, and they work directly with ethical factories and cut out the middlemen.

25:45So you're paying for high-quality, not brand markup. I love their stuff. I bought it before they advertised, and I'll keep buying it still. Find your next four favorites at Quince. Download the Quince app for app-exclusive offers or go to quince.com slash better. Get free shipping on your order and 365-day returns. Now available in Canada and the UK, too. That's Q-U-I-N-C-E dot com slash better. Financial growth begins long before the first investment.

26:17It comes from understanding what you're building toward, what's at stake, and what success looks like for you. At Oppenheimer, we bring bold thinking guided by the full strength of our expertise to put capital to work building and protecting wealth that lasts generations. Put the power of Oppenheimer thinking to work for you. Wealth management, capital markets, investment banking. Hey, it's Kelly Rowland. You may not know this, but I have eczema.

26:49So I get how it can steal your time. But why let eczema take over when you can talk to your doctor about ebglis? Ebglis Lubrikizumab LBKZ, a 250 milligram per two milliliter injection, is a prescription medicine used to treat adults and children 12 years of age and older who weigh at least 88 pounds or 40 kilograms with moderate to severe eczema, also called atopic dermatitis that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies. Ebglis can be used with or without topical corticosteroids. Don't use if you are allergic to ebglis.

27:20Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with ebglis. Before starting ebglis, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about ebglis and visit ebglis.com or call 1-800-LILLY-RX or 1-800-545-5979. This summer, Prime Video is the place for steamy romances, addictive love stories, and the book-to-screen favorites you've already read twice.

27:53The Love Hypothesis, L, The Devil's Mouth, Sterling Point, The Last Sunrise, and Off Campus. The slow burns, the second chances, chemistry you can actually feel through the screen. It's all here. Obsession is in session. Get ready to obsess on Prime. OnDeck is built to back small businesses like yours. Whether you're buying equipment, expanding your team, or bridging cash flow gaps, OnDeck's loans up to $400,000 make it happen fast.

28:25Rated A-plus by the Better Business Bureau and earning thousands of five-star trust pilot reviews, OnDeck delivers funding you can count on. Apply in minutes at OnDeck.com. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend to North Dakota all loans and amounts subject to lender approval. If you sometimes turn down the podcast just to hear the hum of your engine, Then Shell has the fuel for you.

28:51Shell V-Power Nitro Plus fuels every drive from the Mississippi River to the Smoky Mountains with a fuel like no other. It provides your engine with unbeatable cleaning power by destroying engine deposits to improve performance. Because your car's engine matters. That means more power when you need it and more performance with every drive. Shell V-Power Nitro Plus Premium Gasoline. More power, more performance. Chances are you're not far from a Shell station. Download the Shell app to find it. Formulation unique to Shell. Shell V-Power Nitro Plus Gasoline compared to lower octane minimum detergent gasoline.

29:21Actual effects and benefits may vary according to vehicle type, driving conditions, and driving style. See Shell.us slash more for more information.

29:38I think that that's a really important point. Is that consistent through time, though? Yeah. I mean, from that point on, he's convinced of this. But he's not seen anything, and these people have not seen anything. No, I mean, you know, he later on, he starts pointing to various advances that are happening in machine learning

30:09as evidence that the amount of time that we have to solve this problem that he came up with is, you know, running out. But no, you know, that original idea and his faith in it is based purely on thought experiments and conversations he has with other people in this community. Because the way these fucking people frame themselves when they talk about this, they talk, I'm speaking from, I don't know, a podcast episode, I just did one of them. They don't talk about, and maybe this is how they varnish it, they don't talk about that they would be the authority in some cases.

30:42They just like, we should have one. Which sounds, that's how they always get you, these people. People like this. It's always, they sound reasonable, and then behind closed doors, trousers come off. Okay, that's not true, but, well, maybe. But it's about, it's about how, oh, actually, we are, we are, like, Boris Balkin in the Ninth Gate. I alone am worthy to deal with this. That's terrifying. I don't like that one bit. I mean, oh. Okay, let me, yeah, because we're going to make this worse now.

Ties to modern AI labs

31:14Yeah, right. All right, so this, this then becomes the key, this is one of the key arguments in my New York Times op-ed. Like, why is this relevant to right now? Well, the issue is, so this world is going on, Joukowsky has his followers, he's convinced of this, it's mainly thought experiments, right? There's not a particular AI technology I'm looking at. That world severely overlaps with OpenAI Anthropic and DeepMind. And let me give you, like, the very condensed version, and Adam can add the pieces that I'm missing here, right? But let's go back to the founding of DeepMind, right?

31:45There's three founders of, co-founders of DeepMind. Two of them, Hasabi's and Shane Legg, they're at a Singularity Summit. They are talking with Joukowsky. Joukowsky says, let me introduce you to Peter Thiel. And Peter Thiel gives the initial funding for DeepMind. Shane Legg has been deeply involved in sort of the Joukowsky rationalist world for a long time. He was motivated by a talk that Joukowsky came to give about superintelligence at the place where Legg worked. He went on to write a doctoral dissertation on superintelligence, a frequent poster on LessWrong.

32:18So DeepMind has these deep connections to that community. When they get bought by Google, you know, that's when Sam Maltman and Elon Musk are like, well, that's bad, right? Because this is, in a true rationalist sense, this is the technology that's going to either redeem humanity or kill us all. We can't have a company like Google control it. But that's when Musk puts up the money for OpenAI. You have Sam Maltman, you know, tweeting favorably about Joukowsky at this point, saying he's done more for AGI than anyone else. Joukowsky should get the Nobel Prize. The current president of OpenAI, Greg Brockman, was running a LessWrong reading group at the company where he worked before.

32:52Wait, even Brockman? Brockman was the normal one. No, no, no. All right, so you have these deep connections. The board of OpenAI, half of that board, comes strongly out of the sort of rationalist EA communities. When they tried to oust Sam Maltman in 2023, as Cade Metz reported at the time, it was the faction of the board with rationalist EA connections that felt like he was being insufficiently aligned with rationalist-type ideas. The temporary CEO that the board put in place to try to replace Sam Maltman is someone who is so entrenched in rationalist circles

33:26that there's a character named after him in the Harry Potter fan fiction that Joukowsky wrote. Dario Ambadé and his sister and the other employees who left OpenAI to start Anthropic did so because they felt like, again, they were being insufficiently aligned in part to rationalist ideals about how do we usher in superintelligence safely. You have Amadé a couple years ago parroting common rationalist talking points like, yeah, we have about a 25% chance that AI is going to kill all of humanity.

33:57So these worlds, those three companies, they come out of this world of rationalism, which had a really big foothold in the Bay Area of the time, which means our leading AI labs, the way they think and talk about AI is heavily influenced by everything we just talked about. Yep. No, that's all correct. Yeah. So when they say alignment, because alignment as an idea is not the worst idea. It's making sure the models give good outputs and the outputs aren't inherently evil.

34:29That's not really what they mean when they say alignment. No, but that's exactly what I'm getting at, though. Yeah. Because there is the term alignment, which means the thing I just said to most people I talk to, and then what they mean by alignment, which appears to be something else. Yeah. No, what they mean by alignment, and I mean, this is basically word for word, what Yudkowsky says is an AI that, when it becomes super intelligent, will not kill 100% of humanity. Wow. Yeah. That's fucking annoying.

35:01Yes. That's, I know I meant to sound more eloquent than that, but that's, that makes me very frustrated. Well, to make you more frustrated and to make it even more annoying and infuriating, because these guys were so deeply entrenched and involved with the AI industry, and because they'd been writing about it for so long, it means that when the sort of AI moment arrived for the public

35:31with, you know, the release of ChatGPT in late 2022, the rationalists and effective altruists were able to convincingly portray themselves as experts who had been working on AI for decades. And this is how you end up with things like, you know, Bernie having, you know, a sit down with Eliezer Yudkowsky.

36:00Right. Yeah. This makes more sense now, because the way they frame themselves and they'll say like, oh, AI 2027 has been accurate in some ways, even. And as you'll know, in a podcast coming up, I push them on that and they refuse to actually open the blog in front of me, but it's, it's how they have scary, scary stuff framed in the terms of academia without actually being academic, right? Yeah. Yeah. Cool.

Effective altruism and long-termism

36:26And now there's two important sidebars here. So sidebar one, we've been kind of casually referencing effective altruism. Yeah. This is important, right? Because so effective altruism, this is where it gets complicated. So this is a, another movement that has like similar sensibilities that became intertwined and heavily influenced. They sort of mutually influenced each other. Right. But you have effective altruism, you know, it comes out of a bunch of utilitarian philosophers at Oxford and they have this idea of, oh, we can apply, this should sound familiar, but

36:58it's like, if we're very like rational and empirical about altruism, we can do altruism better, which is, I think is like a, it's like modern utilitarianism. That's a fine idea. This is where they're, they're going around and doing research. Like what are the actual best charities to give money to? How do we measure that? They have these ideas. It's like, actually, if you are, um, if you're at Oxford, like an elite university, the best thing you could do for charity is to go get a high paying banking job and then give up half your salary at charity. So they're, they're trying to think through rationally, how do we be, uh, do altruism as

37:29effectively as possible? Well, they overlap and are influenced by rationalism. And pretty quickly, you have a large segment of effective altruists. You say, actually, the most altruistic thing we can do is prevent an AI apocalypse. Yep. Because why bother, why care about, you know, saving 20 million people now from malaria when we could instead be solving, you know, saving a hundred trillion future lives that will never be born if AI kills us all. So effective altruism, not all of it, it's a complicated field, but big segments of effective

38:02altruism. And this is where Nick Bostrom comes out of, uh, at Oxford talking about existential risk and super intelligence. A big segment of effective altruism really begins to care about super intelligent AI is like one of the core things to care about. And a huge amount of money pours into effective altruism. Uh, we're talking hundreds of millions of dollars starts going to these institutes and groups. Uh, so those, nothing's been done yet though. No, I just want to be clear. Like no actual actions have been taken to stop super intelligence, no safety standards, no

38:34actual science has taken place either. Well, they wrote the Harry Potter fanfare. Okay. So no, I'm not even being sarcastic. Like they, they wrote some posts. They wrote some posts and, and kept talking about how it was really hard because, you know, what does it mean to align something with human values? And like, well, maybe, maybe your question is ill-posed and maybe go work on a fucking large language model. Go and work on AI instead of this, instead of this weird thing. I mean, some of them did, right? You know, some of them went off. And that's how we got anthropic. Yeah. Yeah. Yeah.

39:04And deep mind and open AI. I mean, yeah. But, um, but like just to everything Cal said is right, except, except one thing. And I think it's important. Nick Bostrom is definitely like part of the whole effective altruism community, but Nick Bostrom, he, he wrote, uh, um, he's probably best known for a book, uh, that he wrote called super intelligence that came out, I don't know, a little over 10 years ago, um, that made a big splash, basically, you know, expounding on all of these ideas, uh, and saying, yeah,

39:35you know, super intelligence is really scary and we should, uh, we should, we should figure out, you know, how to control it or stop it before it kills us all. Um, and again, this is before, you know, transformer models and, and the current AI moment. Um, and, and it's, it's not from any particular thing he's looking at, but the thing is Bostrom, who at that time is a philosophy professor at Oxford, uh, came out of the same, you know, place basically as Yudkowsky. He was on those same 1990s listservs and he's the one who sort of is most responsible

40:11for the connection between rationalism and effective altruism because it was when his colleagues at Oxford were sort of working on effective altruism that he went over to them and said, oh, if you're worried about that, you really need to be worried about the AI apocalypse. But these are two different worlds. Yeah, but at this point, they're kind of conjoined. Yeah, they're at, at this point they're, they're like, they're really like, you can't disentangle the two communities. Yeah. There's a talk you can find. So William McCaskill was one of the two big, uh, utilitarians at, um, Oxford that helped

40:45found effective altruism. Yep. Um, and I'd crossed paths with him sort of earlier on in the EA world when they were really caring about, especially like the idea of donate most of your salary to charity. There's a, a talk you can find and I, I, I, it's on YouTube and I, I forget exactly when this happens, but there's a, there's a famous talk you can find in McCaskill where he basically comes out and says, this is the new thing we should care about is existential, um, the existential threat of super intelligent AI. This is the, this sort of long-termerism. So they used this, there's another idea that comes out of Silicon Valley, but this long-term

41:18view of what's going to happen over the next thousand years is actually, if we're just going to do the math, a hundred trillion future lives matters more than, you know, 50 million lives now. So he kind of has this moment where he gives this big speech that where he's sort of announcing, this is the new direction for EA. Um, and so now they're like roughly interchangeable. This is why, like, if you read like Cade Metz is reporting on like the open AI board or reporting about, uh, Amadei and his philosophical connections, they'll sort of use the terms rationalism and EA, uh, interchangeably they'll use one or the other, say both.

41:52And that's why these circles have become very, um, very simple. And they seem to have had different goals that have now kind of conjoined. Yeah, I think that's correct. Yeah. I should also point out, you know, the most famous example of someone who, who took McCaskill's advice and, you know, went off to make a bunch of money in the finance industry and then tried to donate large amounts of it, uh, is Sam Bankman freed. Oh yeah. Yeah. Big EA. Yep. Yep.

42:22And actually, uh, early investor in Anthropic. Oh yeah. Yeah. Yeah. Early investor in Anthropic gave a bunch of money to, uh, rationalist and effective altruist community organizations. Uh, and yeah, no, I mean, McCaskill met SBF when he was a college student at MIT. I get around with a lot of kids. Yeah. Well, they, the Charlie Kirk style thing where they go to college campuses almost. Yeah. Well, I mean, for thinking online dudes, that's the thing.

42:53They, they do do this. There's a, uh, God, there's a piece. I want to say it was in the Washington post a few years ago by Natasha Tiku about, uh, campus recruitment strategies that the effective altruists were into because they have, you know, just enormous piles of tech money that they were using to, you know, get them while they're young. And we're talking like the effective altruists will come in and someone will just sign a check for like $60 million to start an institute. Like we're talking huge, huge money coming from.

43:25It's a tech billionaires. Yeah. Uh, because think about it. Who is the hero? Who are the heroes in the tech industry? You are Neo from the matrix. You're John Connor from the Terminator. You are the people who are going to save the world. And if you're a, a tech, you know, a hundred million, a centimillionaire, a billionaire, you're like, yeah, that tracks. That kind of matches myself. And I, I assume the EA rationalist types don't think, Hey, wait, are we just playing into the hands of the richest people alive?

43:56I mean, they kind of either don't think about it or don't really care. They don't think, they think that if you just think really, really rationally, then things like conflict of interests and corrupting your motives. Well, you know, those are just cognitive biases and you can overcome those by, you know, the, the power of pure logic. That reminds me of this guy, Alex Heath, who is ostensibly a journalist, uh, who has also become a venture capitalist this week. I saw someone comment on his post saying, now you've got skin in the game. You can be truly objective.

44:27And they were serious. Oh my God. That, but that is that, that it sounds like the rationalist people. It's like, now, you know, the stakes, you will be bereft of, of any sorts of human biases you may have and work towards this vague goal. We all have that we've, none of us can describe. I mean, they, they believe that, you know, making bets on things, uh, is, is a way of thinking clearly about stuff. When you say making bets, what do you mean?

44:58Money. I mean, I mean, like, you know, like what are they betting on? Well, uh, I mean, this is, this is all their like, you know, decision theory stuff, but they have, um, there's this, uh, um, forecast gambling website called, uh, uh, manifest, uh, sorry, manifold, um, uh, where they, you know, like bet on, you know, it's, it's a prediction market. Um, and they, they love prediction markets because economic theory, right?

45:31You're going to be, you know, the, the rational consumer when your money is on the line, that's going to help you like overcome biases and get towards, you know, they're, they're all this theory behind it is you're going to get a clear, rational solution. Yep. So this reminds me of just like the most annoying early people online. Oh yeah. They were doing brain, like mind master shit being like, yeah, we're so smart. Oh, well, do you want to make a gentleman's bet about who is the smartest? And then they set very strange, vague goals that they can move the goalposts on.

46:02Yeah. Yeah. I've seen this before, except they control so much money now.

Understanding frontier lab behavior

46:06Well, okay.

Understanding frontier lab behavior

46:07So, but now we've got to connect us to the current moment because it, it gives you, this is the argument I was trying to make my times piece. It gives you a different filter to understand what's going on because it's a confusing moment because we have two things happening simultaneously at the, these small number of frontier labs that are heavily connected to this world. A, you're developing a, a, a complicated, powerful technology spending giant amount of CapEx on this, right? So, uh, transformer based LLMs are a legitimate breakthrough in AI. Um, they can, if you push them, put enough money into them, you can, it's, they're very

46:38jagged, but you can, there's all sorts of interesting things that happen. It's a, it's a live wire technology. And so you have a real technology that's developing, but then it's being run by people who have these, these sort of futurist, almost apocalyptic sort of eschatological views where, where basically I feel like that community split where the hardcore original rationalists like you Kowski or, or, you know, Nate Soares, et cetera, would argue like, you know, we need to bomb the data centers because we don't know how to control this yet. We're all going to die. And then you have like the Altman's and the Amadeus who are like, uh, I think what's more

47:11important is that we control the digital God. We get there first. Right. So it's either going to redeem history or, or destroy us all. So you have people running these labs and they're hiring people who think this way. Um, Andrew Ho online was talking about, there's probably a lot of pressure within these places to think that way. If you're hired there, they're thinking in these almost like apocalyptic terms. Like we need to build the digital God before someone less trustworthy does, right? Like the end of history is coming and we need to be the ones to get there. And you have this actual live technology that's powerful and a lot of money is going into. And so these two things are now mixed together.

47:42And I don't think our sort of public or our media landscape was ready for that. Like this mix of weirdness with a real company and real technology. And it's like these two things together is blowing everyone's mind. They're like, well, I don't understand. Like there's these people saying we're all going to die. Like, why would someone say that? Why are they so calm about it? And I think it's almost like the Rosetta Stone to understand what's really happening in this moment is that we have a mixture of a lot of money and a powerful technology with a weird group of people and that mixture is blowing up.

48:14What does that manifest as? What is the specific thing that people need to know? My concern, I mean, a lot of different things are going on here, but my concern is if you look at what specifically is causing trouble, what are we worried about coming out of the frontier labs? Like in particular, the sort of rogue AI narrative is what seems to have caught people's attention. The experiments that led to something like the hugging face attack were extremely negligent. Right. I mean, it boggles the mind.

48:45You know, what we, I've talked about on your show before, Ed, and people point to that episode, so I won't go through the whole thing, but you know, they're taking an agent is basically a prompt loop. It's a, it's a program that tells a LLM with a prompt, Hey, I'm an agent that has access to these tools. Here's my goal. What should I do next? And then it gets a response from the LLM and it executes that updates its state and then says, okay, here's my new state. Here's my tools. What should I do next? So it just blindly executes what an LLM says, which could be okay. Millions of programmers every day use agents.

49:15There's no problems with them, but they ran these over the summer. OpenAI was running these negligent experiments where they gave them very powerful tools that you could use for computer hacking. They hooked them up to an LLM that had no guardrails and what information they would give them. They made the programs persistent. So they said, we want you to just keep demanding an answer. Don't accept, you know, don't accept an answer of like, I don't know what to do next. And then they let them run for days without supervision. And they read thousands of them at the same time. And why did they run thousands of them at the same time is because what we now know is

49:47what they wanted to do is see if any of these versions happen to like stumble onto a particularly good hacking strategy, they would then use that trace to go back and reinforcement train the model to push towards that type of solution in the future. So it's like putting a thousand self-driving cars onto the interstate and saying, yeah, most are going to crash and maybe hurt people. But those that actually make it to the destination, we can use their data to help update our system. So it's this very negligent experiment they were running. And I think my claim is... And they had no idea that their compute was running as well.

50:18They just was like letting these little bastards run. They, I don't think they cared. And the reason why I don't think they cared is because they have this bigger eschatological belief of we got to get to the digital God before anyone else. And you think that that's what's driving open AI though? Well, this is what I'm worried about is that when that's your mindset, instead of focusing on useful products, things that can give you return on investment, common sense, safety standards, you're going to take big risks because you're racing toward, when you're racing towards super intelligence, you know, there's going to be some casualties along the way that's

50:50acceptable. So that's my concern. I don't know, Adam, if you agree with that, but that's my concern is because they are so convinced this is going to happen. And it's so important and it trumps, this is what their writing says. It trumps any measly concerns we have about like the current moment that they will be reckless. It's also why now Amade and Altman are talking about, well, we need a global slowdown, whatever that means before we can stop doing dangerous things. Because if you believe that's the only way to get digital God, you're like, we can't slow down unless everyone else does. Because if someone else gets the digital God, it's going to enslave us.

51:22Surfshark's newest protocol is called Dozos, and the design is actually interesting. Every user gets a dedicated tunnel, so your traffic never shares resources with anybody else's. So there's less congestion, and downloads run up to 30% faster, especially when the network's slammed. And the security piece covers more than the data itself.

51:53Dozos is quantum secure from end to end, built on ML-DSA and ML-KEM, the current post-quantum standards for signing and verifying a connection. But the best part is that Surfshark isn't just a VPN. It's a complete all-in-one cybersecurity suite, meaning that the connection comes bundled with ad blockers, data leak alerts, and covers unlimited devices. Secure your entire digital life with Surfshark. It's completely risk-free with their 30-day money-back guarantee.

52:23Go to surfshark.com slash offline, or use code OFFLINE at checkout for four extra months. One more time, that's surfshark.com slash offline, code OFFLINE. Alright everybody, I'm back talking to you about Quince, one of my favorite clothing brands, and I've been feeling a little dangerous, which is why I'm eyeballing one of their cotton mesh stitch sweater polos when summer ends and the temperature begins oscillating here in New York City. I'm also likely to pick up one of their cotton knit blazers for my next trip to the stock exchange and really can't recommend their stuff enough.

52:55Quince focuses on high-quality wardrobe staples made with premium materials. Like 100% Mongolian cashmere, organic cotton, and merino wool. Everything at Quince is priced at 50-80% less than similar brands, and they work directly with ethical factories and cut out the middlemen. So you're paying for high-quality, not brand markup. I love their stuff, I bought it before they advertised, and I'll keep buying it still. Find your next four favorites at Quince. Download the Quince app for app-exclusive offers, or go to quince.com slash better.

53:26Get free shipping on your order and 365-day returns. Now available in Canada and the UK too. That's Q-U-I-N-C-E dot com slash better. Financial growth begins long before the first investment. It comes from understanding what you're building toward, what's at stake, and what success looks like for you. At Oppenheimer, we bring bold thinking, guided by the full strength of our expertise, to put capital to work building and protecting wealth that lasts generations.

54:00Put the power of Oppenheimer thinking to work for you. Wealth management, capital markets, investment banking. Hey, it's Kelly Rowland. You may not know this, but I have eczema. So I get how it can steal your time. But why let eczema take over when you can talk to your doctor about ebglis? Ebglis, Lubrikizumab LBKZ, a 250 milligram per two milliliter injection, is a prescription medicine used to treat adults and children 12 years of age and older who weigh at least

54:3188 pounds or 40 kilograms with moderate to severe eczema. Also called atopic dermatitis that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies, ebglis can be used with or without topical corticosteroids. Don't use if you are allergic to ebglis. Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with ebglis. Before starting ebglis, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about ebglis and visit ebglis.com.

55:03Or call 1-800-LILLY-RX or 1-800-545-5979. This summer, Prime Video is the place for steamy romances, addictive love stories, and the book-to-screen favorites you've already read twice. The Love Hypothesis. L. The Devil's Mouth. Sterling Point. The Last Sunrise. And Off Campus. The Slow Burns. The Second Chances. Chemistry you can actually feel through the screen.

55:34It's all here. Obsession is in session. Get ready to obsess on Prime. Cash Flow Crunch? OnDeck's small business line of credit gives your business immediate access to funds up to $200,000 right when you need it. Cover seasonal dips, manage payroll, restock inventory, or tackle unexpected expenses without missing a beat. With flexible draws, transparent pricing, and control over repayment, get funded quickly and confidently. Apply today at OnDeck.com. Funds could be available as soon as tomorrow. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank.

56:07OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval. This is Chelsea Halner from Dear Chelsea. If you travel a lot with your dog like I do with Doug, feeding them well on the road can get surprisingly complicated. Especially when you're touring 30 plus cities, staying in hotels, flying, driving, and basically living out of a suitcase. Frozen dog food sounds great until you realize you don't exactly have a freezer backstage. That's why I started bringing Just Fresh from Just Food for Dogs. It's real fresh dog food made with human-grade ingredients with meat and vegetables you can see in the bowl.

56:38But the difference is it's shelf-stable, which means I don't need a freezer or a cooler and there's no prep. So now I can throw a few packs in my bag and Doug still gets a healthy meal wherever we are. Doug absolutely loves it and I like knowing I am feeding my baby something that's actually good for him. It's also the number one vet-recommended fresh dog food made without preservatives or fillers. Go to JustFoodForDogs.com and get 50% off your first order.

57:09Yeah, no, I agree basically with all of that.

True believers and corporate culture

57:13I mean, this is why they're doing what they're doing and it's just awfully convenient, isn't it? Right? When you have this belief that this is inevitably what's coming, that this is what you are building, and you know that you're right about it, well, it makes it really easy to ignore any other concerns and especially any concerns about, you know, something concrete that's definitely happening as opposed to this. I mean, the nicest way that I can describe their belief in this is that it's highly speculative.

57:50I think that that's too kind and that it's a more accurate way to describe it is it's a religious delusion. So, but could there not be? Because I don't know if I buy that this is everyone working at these companies, but I think what you're seeing right now with this slowdown talk might be a result of the schism between there are a lot of rationalists and then there are other people who are working at a computer company.

58:21And so that's why you've got weirdos like Jacob Cox and going out being like, I'm so scared of the computer without really saying what they're scared of. And also people being like, no, no, I love AI, of course, but this is cloud software because if it was just rationalists, I don't think they'd be so apologetic. But it kind of explains why Amadei is like, I think it's time we finally tell the truth about this while also not telling the truth.

58:51Or saying what they're telling the truth about. I don't think you need that, Ed. I mean, I think if you are, you know, in this sort of rationalist philosophy, the slowdown is like, no, they're afraid of the digital God. But they're like, we can't stop if other people are going. Like, I mean, I think the more I understand about the belief system here, the more I discount, for example, the arguments about marketing or the arguments about this is a good way to make your technology seem powerful. I increasingly believe like, no, this is just what they believe.

59:21Like, they started these companies for this purpose. And I think it's the controlling atmosphere in those companies. Not everyone at the – but so how do we know that, right? I think it's the controlling atmosphere. Yeah, not everyone else feels that way. But what have we noticed with OpenAI in the last four or five months? They've had a steady stream of high-profile hires from other segments of the tech industry that within like a week or a month leave, right? So we've been seeing this revolving door. I have to imagine they're hiring you because they're at a size now where you have to hire executives that aren't from your own world.

59:53They're coming in and perhaps what's happening is they're hearing the rational rhetoric and are like, uh-oh, danger Will Robinson. And they're out the door again. Like, I don't want to be involved. So you think that Jacob Coxon might be genuine? Oh, I think Jacob Coxon is definitely genuine. Yeah. Look, he gave an interview to Wired a few days ago, okay? And in that interview, this is what they used as the headline at Wired. They said, you know, that he was calling this crunch time for humanity. What they didn't say, what Coxon didn't say in the interview, or at least wasn't published in the interview, is that is a direct quote from Eliezer Yudkowsky from like, you know, 15 years ago.

1:00:33What was the exact quote? That this is crunch time for humanity. So the reason the only pushback I'll give you there is he uses the phrase crunch time, and he says, and I'm going to pull the quote now. These are actually just literal quotes from my colleagues at Anthropic. The reason I push back is because all of this started with him doing an exclusive with the Wall Street Journal, which is not traditionally how that works. I used to run a PR firm.

1:01:03You don't get an exclusive with the Wall Street Journal because you left the company and you're scared without the assistance of a PR firm or at least an organization like EA. So I'm like, it feels like this is a co-opted thing, almost. Well, I mean, that might be true. I don't get the sense it's like, he can't even explain what he's worried about. Yeah, well, no, I mean, I think it's true that, right, you have this schism in the EA rationalist world where you have the EA rationalists who are at the actual labs themselves who are like, we have to be the ones to create the digital god.

1:01:38And then you have the giant infrastructure of EA and rationalist institutes that are incredibly well funded that are saying, don't try to build the god at all because it's going to be really bad. So I think that's completely plausible that one of these, you know, he obviously has connections to some of these EA institutes from before he took the job at Anthropics. So sure, that's, but I think that just matches what we're saying, right? So he is more from the camp of don't. There's a big difference between a young man taking an opportunity, which is genuinely what I think.

1:02:10I really think that that's what he's doing. We can disagree. That's fine. And the EA cause taking advantage of that, because like Daniel Kukulachee, whatever his name is, Kukuracha from AI 2027, quote posted it within like seven minutes. Like it was, it was very clearly coordinated on some level. But what's, to what end? I mean, like I've seen that and that might be true. Maybe he's the Rosa Parks of like the EAs or something, you know. No, I don't think he's, I don't think, I think he knew that if he did this, it would get a bunch of, a bunch of attention.

1:02:45Because my big thing with this is, I think like the, I think this is a burn after reading situation. I think that the vast majority of, the vast majority of this, the problems caused by the situation are, there are people who genuinely believe this stuff. There are people who cynically believe it. And then there are the rest of them who don't know who to trust, who are just going like, yeah, right? Like, sure, I agree.

1:03:12Like, this is really bad. Like Sam Altman, I don't think he's a rationalist. I think he's a guy. I think it's actually quite funny watching Sam Altman adjust to this because he's just like, yeah, slow down safety council, I guess. What do you think? I don't really care. Dario Amadei, now that you've said this, I will buy that he actually believes this stuff, which only makes the whole thing weirder. But I think that the mishmash of different cynical people, honest people, everyone's kind of weird, is just, it's causing chaos in the AI industry right now.

1:03:48Like, this is not, people are saying, oh, it's an op, oh, this is all a plan. No, I don't think they have a plan at all. Yeah. No, I mean, I think that's a fair way of putting it. I mean, like, to me, what was more important than Jacob's tweet was the multiple responses from current Anthropic employees. They're like, yeah, he's right. Yeah. He's probably going to kill him. It's so strange. And it was so calm. And I think that blew a lot of people's minds. But I'll tell you who was not confused by that was probably Adam, right? Yeah, no, that's exactly what I expected. Yeah. Yeah. Yeah, exactly. No, because, you know, there are a lot of people who believe in this stuff who work at these AI labs because they think that that's the best way to keep a handle on this.

1:04:26Is to be the ones who control the coming Godhead. And, you know, I understand, Ed, the impulse to think that, you know, that this is something that people are using cynically. And there probably are some people in the industry who are using it cynically. Yeah. But there are an awful lot of true believers. There are so, so many of them. And these beliefs about what's coming have been so, so influential in the industry.

1:04:57It is the conventional wisdom about AI in the tech industry at this point is that AGI is coming, super intelligence is coming, and it is going to radically and permanently transform the world. And it's going to come out of these AI companies working with these kinds of AI systems. And there's just no evidence for any of that and a lot of evidence against it. And that's the thing, just to be clear about what I'm saying. Yeah. Actually, this is a point.

1:05:28I actually need to make this point very clearly. I think that there are a ton of EA rationalist types in this industry. I think Anthropic is full of them. I also think the overall industry led by Sam Altman played into their fear tactics and said, oh, we're scared of what this might do, insincerely. Some of them were sincere. Some of them were not. The outcome is they've created a very real panic about nothing. Like, they won't talk about the real harms of LLMs. They'll talk about the big, scary computer that might wake up one day.

1:06:01Well, and I would push back on Sam Altman being insincere about it. That's just the way people talk in the world he came out of. Yeah, but Sam Altman is the... That's just how they talk. That's how Elon talked. That's how they all... Elon was running all these safety conferences. You know, that's how they all talk about this. I'm serious. Here's the thing, though. Using Elon Musk as an example of someone with a sincere belief, I've read those emails. These are just regular rich guys being rich. Like, I will buy that Yudkowsky believes this or what have you. The rationalist types believe this, what have you.

1:06:33But it's like, the idea that Sam Altman, it's just laughable. I just... I don't... That one, I don't really... I can't get my head around it. Same with Musk. Well, in your defense, right? The defense of your claim there would be the fact that the EA rationalist faction of the OpenAA board tried to get rid of Altman, which was at the time widely reported as, this must just be because he's kind of a slimy guy. But, you know, it's also true that specifically the people who voted to do this are strongly aligned with Yudkowsky's world

1:07:03and thought that he wasn't serious enough about it. So that would actually probably back up your view. Yeah, who was a big EA person, she got kicked out by Altman. Yeah, no, all the EA people on the board got kicked out, including... What's her name? God, I can't remember her name. Joseph Gordon-Levitt's wife got kicked out. Yeah, that's the person whose name I can't remember. Yeah. Jesus. But then, okay, so putting aside that sort of nuance, how else can we empirically test, you know, the effect of this philosophy on these particular companies?

1:07:35Well, I laid out a test in my piece. I said, here's two things we can do. Look at other companies in the space that aren't as strongly connected to rationalism, but are just as cutting edge and really understand technology and see how they talk about it. And I pointed to NVIDIA in particular because, you know, Jensen Wong, he started that company while Yukowski was still a teenager. He doesn't have a lot of strong connections to that, you know, Silicon Valley futurism. And I said, also, you have to keep an eye on China. They have no connection to that sort of Silicon Valley philosophy. And for the most part, they really don't talk about AI with that same sort of fear narratives that we do.

1:08:10Well, what happened this week? Jacob Croxon came out. It became a big discussion. What did Jensen Wong immediately say? He said, you know, that's dangerous and outlandish. So immediately, he's like, that claim makes no sense, right? Today, China came out. Like, what are you talking about a global slowdown? What is this, like, psyops, Cold War nonsense? Why would we slow down our industry? Because they don't see it that way. They don't see their industry. They don't see AI as, like, a nuclear weapon or a godhead they're building. They see it as, like, an economic tool that they're trying to get trade advantage.

1:08:41So I feel like this week validated the claim that this particular philosophy is playing a big role. It's like that Sherlock Holmes story, the dog that didn't bark, right? Like, the thing that matters here is not the things being said by the two labs that we know are deeply connected to these philosophies. It's all the other people who are doing cutting-edge research on this who know it well who are not coming out and saying, yeah, we also think we're about to kill all of humanity. It's NVIDIA, which is, you know, they supply all the chips to all these companies.

1:09:12They're intimately involved in these models and their evaluation. And, I mean, they customize the API interfaces to their chips to specifically match with these – they know what's going on. And he's like, this is outlandish. What the hell are you talking about?

Pushback from Nvidia and China

1:09:26But what's funny is Jensen Huang is barely a week divorced from declaring AGI is here. Yeah, but this was a bridge too far. This was a bridge too far this week. Or last week, I guess. So I'm kind of yes-anding because what I'm saying is he has, as I think some have cynically, engaged in this, oh, AGI superintelligence wank, this conversation about it. And now suddenly it's like, oh, wait, maybe not that much. Maybe we don't superintelligence that hard.

1:09:56Maybe we – actually, it's not superintelligence. It's cloud software. Stop freaking out. And I think that this is the thing. It's regular greedy arseholes locking horns with a cult, a religious cult of posters.

1:10:13Yeah, I think that's definitely part of what's going on here. What am I missing? Well, no, I mean, I think – I mean, I just think that it's totally, totally possible that even some of the rich assholes genuinely believe this stuff. It's just that genuine belief for these billionaires manifests in a different way because they believe that they have the power to warp reality with their money, which to a point is true.

1:10:48Like, you know, Mark Andreessen is someone who I would never ascribe sincere beliefs to. You know, like I don't – I'm sure he has some sincere beliefs and I just don't think that we have any evidence of what they are because I think he just says the stuff that he says because he thinks it'll make him money. But, you know, he put out a manifesto a few years ago where he said that slowing down AI means that you are guilty of murder because –

1:11:20Cool. Yeah, cool. Yeah, because, you know, that means you're slowing down the advent of a super intelligent AI that will save lives. It's just EA in reverse. Exactly. Yes. And, you know, that's a great excuse to say, well, but no, I have to make the investments that I make and you have to let me do it without any regulation or slowdown of any kind because otherwise you're a murderer. I – okay.

1:11:51As we wrap up, though, I was about to rant about these people. They sicken me.

How to pierce the rationalist veil

1:11:56What should we do about this? As in, okay, maybe not the first thought in your head, but what is the way that people should look at this differently? How can they pierce the kind of rationalist veil and at least have a more sensible way of looking at how everything's happening? I mean, I have a few suggestions right off the bat because these are the calls I've been taking all day, every day, it feels like, for the last week. First of all is clarity and terminology, right? Like, just – that's why I wrote that op-ed. Michelle Goldberg has a similar op-ed that came out the next weekend.

1:12:28I think it's really important that people actually understand there is a particular philosophy that's heavily influencing these companies. At the very least, we should be factoring that in to try to understand what's going on. And there's a good quote in Adam's book from Timnet Garibu who basically said – and I'm paraphrasing a little bit, but it was referring to this issue. If it turned out that the major AI companies were run by Scientologists, we'd want to know what's the influence of Scientology on the decisions they're making or how they run their companies. That's what's happening.

1:12:58I mean, it's just rationalism EA instead of Scientology, but it's a very eccentric sort of almost cultish internally looking self-police type of belief system that's playing a big role. We should at least know that's the case so we can factor that in. Two, we cannot allow the people in these circles to get away with co-opting the general term AI to refer to specific things they're doing that are causing problems. I see this terminology everywhere in the media this week where it's all about AI is getting out of our control.

1:13:30We need to slow down AI. AI is getting smarter and what's going to happen. We have to be very specific, right? And I made this point in my newsletter as well as it's – Good. The terminology of pacing the frontier is in their terms. It's taking their philosophy about this. It's we need to slow down. Everything we're doing is fine. We're just doing it too fast. When in fact, LLM is on the right thing. This is a Max Tegmark mental model from his book, Intelligence 3.0. They have this idea of AI –

1:14:01Is that the future of life guy? The future of life guy. Yeah, which is also deeply, deeply connected. I've met him. He's nothing special. Yeah. But he had this idea of thinking about AI capability like a water level that rises, right? So that if anything is getting better, it leads you to think all of AI capabilities are getting better, which is partially why open AI is spending so much money on obscure math problems, right? There's not a financial – why aren't you building tools that like businesses will buy?

1:14:31It's because of the mindset of, well, if we can solve a math problem this hard, we can do everything comparably as hard. So they want to connect it all together. But if we get more specific – no, the issue is a very specific thing. First of all, it's in the world of LLMs. Most AI is not LLMs. Within LLMs, it's specifically in the world of agents. But not all agents. We use – millions of people use coding agents every day. They never go rogue. There's no issue with them. We're not worried about that technology. So it's a particular type of agent, which I call over-equipped, under-constrained, where they give them very powerful tools.

1:15:04They run them with no constraints. It's a very specific type of system that caused the hacking attacks and all of their concerns about recursive self-improvement is because they want to next take these over-equipped, under-constrained, LL-empowered agents and allow them to update their own code. So now they're completely unconstrained and unmonitorable. It's that that we're worried about. So I think it's very important that we talk about specifically the thing we're worried about because it's very narrow. And then third, if I'm Congress, don't pass a bill about we need to, in some vague sense, slow down AI.

1:15:36Bring these people in to testify and say, tell me about these over-equipped, under-constrained, LL-empowered agents. Why are you running those experiments? Was there no way to run those safer? Is there other things you could – why are you focused on that? Is this what you mean when you're saying you're worried? Like, okay, what do you believe? Do you believe – and going through like Yakowski, like we really need transparency, I think, on this situation. That's what I would do because I think it would really change the dynamic. If we could pick apart what's true about the technology, what is being hyped, and what is being driven by these beliefs and is causing more danger than – because you could shut down those over-equipped, under-constrained, LL-empowered agent experiments tomorrow.

1:16:20It would have almost no impact on your ability, all the products they're working on that are actually money-making. It should have no effect on their ARR and their revenue or anything, but they're not because they think that's the way to get the superintelligence. So that's what I would do is like I would be super clear about what's really happening here. I'd be super clear about what rationalism EA philosophy is, and I would start pouring transparency on specifically what they're doing and why, what they believe, what experiments are dangerous, why they're running them so dangerously. That's the hearings we need right now.

1:16:52Adam, any additions? Yeah, yeah, I think that those are all good ideas, but I think the thing that I would add is we also really need to try to shift the public conversation on this stuff, right? The response to, you know, Coxon, the whistleblower at – I don't even want to use the word whistleblower – the guy who left Anthropic and made a big splash recently, the response to that should have been, yeah, of course you believe that because, you know, you're part of this group that has these weird beliefs.

1:17:24Like, it's a nothing burger. Like Cal said, I wasn't surprised by this. You know, I wasn't surprised by what he said and what his colleagues and former colleagues said in response. I think that we need to get the message out there that this narrative about existential risk caused by super intelligent AI going out of control and that that is the most important and most likely thing that we need to worry about with AI.

1:17:55We have to get the word out that that that is just a belief that comes out of a almost religious cult-like object, the effect of altruists and rationalists. Like, this is not something that is based in reality. And the real harms imposed by this technology are very concrete and very well known. Awesome. That is great. I am so glad you guys were here for this because I feel like at this particular time in history, as both of you have said, there is no clarity here.

1:18:31There are a lot of assumptions being made. And I'm really – thank you so much for joining me because I think this will really help people clear this up. Well, thanks, Ed. I think it's – the more we get this message out, the better. Oh, by the way, I would add one little extra caveat. Don't look at X. Ah, yes! It's absolutely – It's completely caught up in this engagement farming off of this rationalist EA belief system. If you're a reporter out there who's listening to this, do not look at the X conversations.

1:19:01Call real sources. Talk to them in person because X is off the rails right now. Indeed. All right. I'm going to end it there. This has been Better Offline. I'm Ed Zitron, joined by Adam Becker and, of course, Cal Newport. I'll be back with the monologue next – well, I think it's going to be – It's going to be Friday. Anyway, this Friday. Cheers, everyone. Thank you for listening to Better Offline. The editor and composer of the Better Offline theme song is Matt Ossowski.

1:19:33You can check out more of his music and audio projects at mattosowski.com. M-A-T-T-O-S-O-W-S-K-I dot com. You can email me at ez at betteroffline.com or visit betteroffline.com to find more podcast links and, of course, my newsletter. I also really recommend you go to chat.wheresyoured.at to visit the Discord and go to r slash betteroffline to check out our Reddit. Thank you so much for listening. Better Offline is a production of Cool Zone Media. For more from Cool Zone Media, visit our website, coolzonemedia.com, or check us out on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

1:20:11Your favorite friend group is back. The modern web runs on tracking.

1:21:03Every click gets logged, profiled, and sold, and that same data sets your prices and feeds scammers. Surfshark pushes back. The VPN encrypts your connection under a strict, independently audited no-logs policy, and the suite around it blocks ads and trackers across every device you own. Secure your entire digital life, risk-free, with their 30-day money-back guarantee. Go to surfshark.com slash offline. Code OFFLINE. For four extra months of Surfshark. The path to retirement is different for everyone.

1:21:36And as life changes, so do priorities. Fidelity can help you get where you want to go. With a free personalized plan, goal tracking, and timely insights, you'll be set to take on retirement. Whether you're saving for it or already living in it. Get started at fidelity.com slash takeon. Expenses charged by your investments and other costs and fees associated with trading or transacting in your account apply. Fidelity Brokerage Services member NYSE SIPC. This summer, Prime Video is the place for steamy romances, addictive love stories, and the book-to-screen favorites you've already read twice.

1:22:11The love hypothesis, L, the devil's mouth, Sterling Point, the last sunrise, and off-campus. The slow burns, the second chances, chemistry you can actually feel through the screen. It's all here. Obsession is in session. Get ready to obsess on Prime. Most dog food brands don't really want you seeing how their food is made. Just Food for Dogs is the opposite. They actually invite you in. You can walk into any of their kitchens and see real human-grade ingredients like chicken, beef, carrots, and peas being prepared right in front of you.

1:22:49It's real food made in real kitchens. Nothing is hidden behind labels. And that kind of transparency says a lot. Nothing to hide. Everything to love. Go to JustFoodForDogs.com and get 50% off your first order. Hey, it's Kelly Rowland. You may not know this, but I have eczema. So I get how it can steal your time. But why let eczema take over when you can talk to your doctor about EBCLIS? EBCLIS, Lubrikizumab LBKZ, a 250 milligram per two milliliter injection,

1:23:20is a prescription medicine used to treat adults and children 12 years of age and older who weigh at least 88 pounds or 40 kilograms with moderate to severe eczema, also called atopic dermatitis, that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies. EBCLIS can be used with or without topical corticosteroids. Don't use if you are allergic to EBCLIS. Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with EBCLIS. Before starting EBCLIS, tell your doctor if you have a parasitic infection.

1:23:52Paid partnership with Lilly. Respect your time. Ask your doctor about EBCLIS and visit EBCLIS.com or call 1-800-LILLY-RX or 1-800-545-5979. Some days, it's the little things you look forward to most. And with Pepperidge Farms' thoughtfully crafted cookies, there's always something worth savoring. Maybe it's a Milano made with premium chocolate between two delicate biscuits, a Sausalito baked with rich milk chocolate chunks and macadamia nuts, or the simple buttery goodness of a chessman.

1:24:23So pour the coffee, find a comfy chair, and take a moment to treat yourself to a cookie. Or two. Pepperidge Farm. Have a little taste. This is an iHeart Podcast. Guaranteed human.