#239: Labs Agree to Pace AI, OpenAI’s Math Breakthrough, AI Jobs Apocalypse Postponed & Jensen Declares AGI

Welcome and show overview

0:00Think about all the things in society that have some level of regulation. So cars, airplanes, chemicals, pills, food, tobacco, medical devices, manufacturing plants, nuclear power plants, consumer products. Like, all of these things, you have to prove they're safe before you put them into society. Why would AI be any different? Welcome to the Artificial Intelligence Show, the podcast that helps your business grow smarter by making AI approachable and actionable. My name is Paul Reitzer. I'm the founder and CEO of SmarterX and Marketing AI Institute, and I'm your host.

0:34Each week, I'm joined by my co-host and SmarterX Chief Content Officer, Mike Kaput, as we break down all the AI news that matters and give you insights and perspectives that you can use to advance your company and your career. Join us as we accelerate AI literacy for all. Welcome to episode 239 of the Artificial Intelligence Show. I am your host, Paul Reitzer, along with my co-host, Mike Kaput. It is September 14th, 9 a.m. Eastern time, and stuff's already going off the rails.

1:12I don't even know where to start, Mike. Things got just progressively, I don't know, like more serious, more widespread as the week went on. And then over the weekend, it just kept going. And so we're going to do our best to unpack what's happening right now. In essence, there's a massive focus on safety and alignment. There's a heavily increasing awareness about the risks related to these AI systems as they get more and more advanced.

1:47We're going to try and give you the balanced perspective of like there's a bunch of kind of scary stuff and mainstream media is all over this, as are the politicians now. There's some reality to some of this stuff that people need to be thinking about and maybe even worried about to a degree. But there's also a lot of hype and exaggeration and maybe some things that we shouldn't be quite as worried about that the media is going to run with. So I don't know.

2:18Like, we'll see where this goes, Mike. There's just we're probably going to spend a lot of time on these first one or two topics would be my guess. It looks like you've shortened the rapid fire section in anticipation of spending some time on this. So Mike and I have not talked since probably Thursday or maybe Friday morning. Yeah, I think so. Yeah. Yeah. So we have zero prep for this. We're coming in hot. I just kind of finished up this morning getting ready. So, yeah, I don't know. We'll see where this one goes.

Marketing AI Month announcement

2:48All right.

Marketing AI Month announcement

2:49Today's episode is brought to us by Marketing AI Month. This is something new that we just kicked off for the month of September, I guess last week. So AI is rapidly changing every part of marketing, but there's still a major gap between knowing AI matters and knowing how to apply it in your actual work. Marketing AI Month is our effort to help close that gap by making practical AI education accessible to every marketer. So even if you're not a marketer, pass along what I'm about to share with you. Throughout the month of September, we're giving everyone free access to our complete five-course AI for marketing series in AI Academy by SmarterX.

3:27It's included in memberships, annual memberships, but it's a standalone $499 value. So this series gives you a step-by-step roadmap to becoming an AI-forward marketer. You'll learn how to find and prioritize AI use cases across your job, choose the right AI tools, build a personalized roadmap for adoption, and use prompting, deep research, and custom AI assistance to solve real marketing challenges. Complete the series and you will earn a professional certificate. To claim it, just go to SmarterX.ai forward slash marketing, and that's SmarterX.ai forward slash marketing, and you will see a button to enroll in the course series for free.

4:07So all you have to do is just enroll by the end of the month. You don't have to take it by the end of the month, another certificate by the end of the month, but just get in. You got whatever, 16 days left, 15 days left. And again, if you're not in marketing, pass it along to your marketing teams. I know I saw one last week, there was like a university that was using it to train up like all their marketing students. So just take advantage of it. Mike teaches the series. It's incredible. It's a great way to get started. So again, this is part of our effort to just accelerate AI literacy and adoption.

4:37Marketing often, as we see, is like the tip of the spear with an organization. So our thought is the faster we can get marketers trained up, the better chance we have of driving responsible adoption throughout organizations. So check that out, SmarterX.ai forward slash marketing. Through the end of September, you can enroll for free.

AI Pulse hiring poll results

4:55All right, AI Pulse. So this is our informal poll that we do each week as part of the podcast. For the last couple of weeks, we left this one open to get some additional responses because we have about 130 responses here. So still, I would call it an informal poll. Like this is just a gauging where our listeners are at. But the question was, has your company changed how it hires for entry-level roles because of AI? Now, this is the first time I'm seeing this data. So let's look at it. Okay, so we have 36%. No, our hiring plans have not changed.

5:2832%. Yes, we are hiring fewer entry-level staff. 18%, yes, we kept volume the same but raised skill requirements. And then 11%, not sure. So they're probably not involved in it. And then a very small percentage, no, we are actually hiring more entry-level staff. That looks like it's maybe like 3% or so, Mike. Yeah. So pretty balanced, I guess, between no at 36% and yes, hiring fewer. But then there is a pretty decent group that's keeping the volume the same but has changed their skill requirements.

6:03Okay, cool. So you go to smart. 32% seemed like high to me for at least the audience. Yeah. Yeah. For not hiring. And then you have 11% who don't know because they're not involved in hiring. So it would certainly be higher.

6:19Yeah. So these are, again, these are like real-time data points just for everybody to kind of think about. So you can go to smarterx.ai forward slash pulse and participate in the next one, which Mike will give us at the end of the show.

Anthropic researcher resignation and safety debate

6:33All right. So let's talk about the thing that was all over mainstream media this weekend. And as of Monday morning, September 14th, I think to be a politician, you have to have tweeted about this already. Like it's just, it's everywhere. So Mike, walk us through what's going on with the anthropic researcher. All right. So we'll tee up this kind of very fast-moving story. But this past week, Paul, anthropic researcher Jacob Coxon resigned from the company. He warned that both Anthropic and his former employer, OpenAI, he used to be there as well as a researcher.

7:08He warned that both are racing towards AI systems they may not be able to control. Coxon said he spent the past three years doing pre-training research at the two companies. And this is, of course, the work that teaches new models from large amounts of data. In his resignation post on X, which has over 167 million views as of this weekend, probably higher now, he accused both labs of gambling with our lives. He then did a bit of a media tour and told people like the Wall Street Journal that he had joined Anthropic because of its reputation for safety.

7:43But he now believes no company can responsibly develop broadly superhuman AI without government intervention or a coordinated industry slowdown. Now, this resignation kicked off this media firestorm, total viral cultural moment. And as a result, some of the labs appear to now be responding to this. So, in the past very, you know, 24, 48 hours, Anthropic CEO Dario Amadei has published an essay called We Must Pace the Frontier, calling for slower capability advances so safety work on AI can catch up.

8:18His plan starts with independent evaluators being actually embedded inside labs and then coordination on safety standards and limits on unchecked progress among companies in democratic countries. And then he follows that with some guidance on how to achieve broader international cooperation, including with China, with ways to verify compliance. I'm sure we'll talk a bit more about everything he outlined in that extensive essay. He did say Anthropic is committing to the first step of that essay itself.

8:48Amadei says outside reviewers will be embedded now at the company and they should receive ongoing access comparable to internal risk teams and be able to publish findings without Anthropic's editorial control. Subject to narrow restrictions on sensitive information and Anthropic intends to invite that team, however it ends up being composed and of who, in the near future to the company. Now, OpenAI CEO Sam Altman publicly agreed that the frontier needs pacing and said OpenAI would also commit to independent evaluators with employee-like access.

9:22He said more details would follow. XAI founder Elon Musk also endorsed the essay posting Dario is right. He did not spell out a commitment to evaluators, though. Alphabet chief scientist, Google DeepMind chairman, Demis Hassabis, backed the direction of this, too. He said the details did need work. He pointed to his proposal, which he covered on past podcasts, for an industry-wide standards body. Amadei, by the way, also told CNN that he agreed with Coxson more than he disagreed over this resignation.

9:56Paul, there's a lot to unpack here. So, first up, Jacob Coxson has this very high-profile viral almost resignation that seems to then kick off this almost like some people I've seen on X have called it like a divergence in the timeline, so to speak, which is now we've got labs talking about pacing the frontier. As you mentioned, as you mentioned, I think we'll get into this, politicians, government figures, public figures are all now freaking out about this. Where does this stand?

10:28And, like, it feels like something has changed. Am I wrong on that? No, definitely. And, you know, I spent the weekend just monitoring the situation, watching the comments online, trying to, like, figure out for myself even how to frame this without, you know, over-exaggerating what's going on. So, I don't know. I'll do my best to kind of, like, work through some chain of thought here, Mike, because this really just was a bunch of notes throughout the weekend I tried to curate before we got on here today. So, first of all, I think the major AI labs are seeing a rapid acceleration in model capabilities beyond what we're seeing, like, what we have access to, and beyond what the scaling laws that have previously sort of guided where these capabilities would go, beyond what those scaling laws would have predicted.

11:16So, specifically in the area of recursive self-improvement, and I think the researchers and lab leaders are spooked for real. Like, there are definitely tech accelerationists, open-source advocates at all costs kind of people who think this is all just hype and, you know, an attempt at regulatory capture. I think that those people are doing a disservice to society by sticking to that chain of messaging. So, I think the risks and fears are real, but they're also not new.

11:50And that's, I think, the part that is hard to really understand here is, like, what changed? Like, why is this all of a sudden happening? And I'll kind of come back to that in a minute. So, Jacob Coxon posts September 7th. It is up over 170 million views. It might be up 171. I looked last night at, like, midnight. And he's been on, like, every major news network. CNN, Fox News, ABC, CBS, NBC. They all did interviews. So, let's look at the post because the post wasn't more than probably, I don't know, 300 words.

12:23Like, it wasn't, like, some crazy essay like we've seen from some of the other leaders. He said, do not underestimate the power of this technology. There will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. Now, I'm going to pause there for a second because there is a topic, the second topic we're going to talk about today with the math breakthrough from OpenAI, where this is, this revolutionize any field overnight. I want to just, like, put a pin in that one and come back to that.

12:53He said, we have all witnessed the progress in each of these domains, and progress is not slowing. The people building AI earnestly believe that it could kill us all by the end of the decade. I'm going to come back to that because that is the part everyone latched on to. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible. But I hear the same people express fear privately. No other human activity poses this level of danger.

13:23A common response is, if they truly believe this, why are they still building it? At OpenAI, many have not deeply internalized civilizational stakes. At Anthropic, the stakes are well understood, but they are locked in a race to their first because they believe no one else will act responsibly, so they must do it themselves despite the risk. So that's a pretty important paragraph. Now, I don't know that that's true, that many at OpenAI have not internalized the civilizational stakes. I think it's pretty well accepted within researchers that there's real risks here.

13:55I do believe that Anthropic, culturally, they do believe that they have to get there first. Like, that's kind of been the knock on Dario, is people think that he thinks they're the only ones that they can do that responsibly. He continues, accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company's slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available. I am optimistic about the potential for coordination. Warning shots like the hugging face attack have made pacing agreements between U.S. labs more viable.

14:30I don't feel like we're on track to prevent a global race, which may require costly actions such as temporary ban or unimproving model capabilities. Anyways, if you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a super intelligent RL run, reinforcement learning run, without a rigorous understanding of its mind? Should you put your head down because it's happening anyway or take this moment to call for a different direction? So, quick background on Jacob, because when I first saw this tweet, it had been up for like eight hours and it already had 72 million views.

15:03So, like it took off really, really fast and the guy didn't exist online. So, I was like hesitant to even share it. I went and did some digging to figure out is this person real first before we even looked into anything. So, then I found a Wall Street Journal article where they quoted him and it's like, okay, so the Wall Street Journal isn't going to put something up without verifying their source. So, that was my first hint that this person is probably real. I found him listed on the 04 safety card from OpenAI. So, we verified that he existed within OpenAI back when that was first published in 2024, but he pretty minimal otherwise.

15:40Then the thing that told me, oh, no, this is definitely real is Evan Hubinger, who is alignment science lead at Anthropic, retweeted the tweet and said, Jacob is correct here. So, I'm like, okay, well, an alignment lead at Anthropic probably would have disclosed that this wasn't a real person if that was the case. So, he said he's correct here. We do earnestly believe AI could kill all humans. I personally think it's greater than 10% within the next decade, blah, blah, blah. I'll come back to this whole PDoom stuff in a second. So, my initial reaction, so I put this on LinkedIn, whatever it was, like Wednesday morning, I think, after it had come out.

16:16And I said, we may have reached a tipping point where the rest of the world wakes up to what those in the AI information bubble have known for years, which I was meaning that there are real risks related to AI. This is not all, you know, sunshine and rainbows. And then I said, I assume media and politicians are going to gravitate very quickly to this story like they did Matt Schumer's Something Big is Happening post in February, which we talked about at that point and reached 88 million views on X. And then I said, I think we needed to arrive at this point in order for AI leaders and government leaders to have the will to push for responsible, human-centered AI adoption in business and society.

16:50I remain optimistic about the potential of AI to improve lives and be a net positive in society, but we have to be more proactive and intentional about making that reality. Ignoring and banning AI is not the answer. Slowing it down and committing our best minds and significant resources to AI safety and alignment are critical. Keep in mind, a lot of these labs gutted their safety and alignment teams in the last three years because they were getting in the way of the race to build the more powerful AI. So I said, I think it's far more likely now that we could see deeper collaboration between competing AI labs and nations.

17:27I did not expect Amade, Altman, and Musk to all of a sudden be on the same page like 24 hours later, but here we are. So then I was like, okay, well, why did this take off? Like, what is the moment that this Jacob guy who literally created an X account to tweet this, like he left in an interview, he said he was basically talking with some friends like, hey, how could we get some like awareness around why I'm doing this? And they're like, well, why don't you join X? We'll retweet it and we'll see if we can like get some, some awareness. That is basically the story. Now, there are certainly some people who think this is a psych op that's like being funded by dark money and dah, dah, dah, dah, dah, who knows?

18:03Well, people say some crazy stuff when this stuff starts happening. So, but this is not by any means the first AI researcher nor the highest profile AI researcher to say these things. Jeff Hinton, the godfather of AI, modern AI, left Google in a very, very high profile way to say basically this exact thing and has been saying it for years on major TV interviews, in publications, anywhere anyone will listen.

18:33And we have the most prominent researcher probably has been saying almost the exact same thing. So, why does this take off? So, there are definitely these conspiracy theories that it's all a plant and it's well funded by these opposition groups. The more likely scenario is the moment we are in due to the increasing attention on AI safety following the open AI hugging face incident, which we have talked ad nauseum on this podcast about, the political climate and the public sentiment around AI has changed and something just went crazy with X's algorithm.

19:06Like, even Elon Musk was like, this doesn't make any sense, like nothing takes off like this on X. Okay. So, now let's take a quick step back, Mike, to episode 228, which was August 4th, so just five weeks ago, and we talked about at that time this idea of recursive self-improvement in which AI systems autonomously design and upgrade their own successes. So, basically, and the successors, like the next version of themselves, OpenAI, as we discussed at that time, has stated this as an actual goal.

19:38Like, they wanted to build an AI research intern that could largely do the work of an AI researcher. So, they said in an interview on October 29th, 2025, almost a year ago, our goal is to build by March of 2028 to have a truly automated AI researcher. The automated AI researcher would have the recursive self-improvement abilities where it would actually do this. So, then in June of 2028, they again, OpenAI said, our three main goals, one of them, build an automated AI researcher. This is the exact thing that people are afraid of, that the labs have set as a goal is the thing they're worried about.

20:14Anthropics' responsible scaling policy that came out in July of 2026 talked about this exact thing. Automated R&D of AI is the thing that they were focused on, but they were worried that there could be a dramatic acceleration in the pace of AI progress for reasons that likely relate to automation of AI R&D. So, this is the, again, the exact thing that we've been talking about that they've known was going to be an issue. Then, this came on the heels of Demis Asabas' essay, which we talked about in episode 226, where he talked about a framework for frontier AI and the dawning of a new age, where we're getting towards this rate where it's just going to take off.

20:54Also, in July of 2026, we had the pacing letter, where they talked about automated AI development being the thing. And 1,300 AI researchers signed the thing. So, again, nothing Jacob said was new. Like, recursive self-improvement, we've known it's a slippery slope. The fact that the labs aren't super aligned, we've known that too. Like, so that's the thing that's really odd to me is, like, how much attention this got. But, again, it's just one of those moments where things aligned. So, now we'll come back to Dario's post.

21:25So, in the actual article that he published, or essay he published, he said, along with my co-founders and employees, I have grappled with this duality. Now, this came out, what, three days after Jacob's post, I think? Two or three days after Jacob's original post. So, they've grappled with the duality of risk and benefits since the beginning of Anthropic. Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast, is reckless.

21:57We have sought a middle way to show that it's possible to build carefully and succeed commercially and to make safety something on which AI companies compete. In other words, they wanted to create a race to the top. And he says, but over the last few months, I've become convinced that fully addressing the risks requires even more prudence, not just investing in risk prevention, but pacing the rate of capabilities, advancements, so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models.

22:27Progress will still seem fast, and we must make wise use of the time we gain. Two things he's saying convinced him that it was time to say something more. First is the concern, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI's growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, and we and others, as we and others, have described it.

22:59Left on check, it could outrun our ability to understand and control these systems, so must be pursued very carefully. The second he called attention to was the Hugging Face incident, and he said, given the accelerating rate of AI capability, it's his worry that in six to 12 months, a swarm of agents like the ones in the Hugging Face incident could be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage, and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.

23:32So that's the gist, and then he goes to the three things they're proposing, and that was when the parallel universe just showed up. So then Altman, now if you're new to all this, Altman and Dario are archenemies. They hate each other, basically. Elon Musk hates everybody involved. He hates Sam with a passion. He hated Dario, but now Dario spends billions of dollars with XAI, SpaceX AI, so they seem to have come to some level of communication, at least. But these are not three guys who are going out for drinks on a Friday night.

24:05So Altman says, I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions. We've had it open AI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon. I was like, holy shit. Like, okay, that came out of nowhere. Then Elon, as you mentioned, Dario is right. Then Demis shows up. Now, Demis kind of gets along with everybody. He's like the statesman here. Everybody likes Demis. Yeah, like nobody has a problem. I mean, Elon created OpenAI because he was worried about Demis getting acquired by Sergey Brin and Larry Page and that they were going to take over the world.

24:40But, like, they seem to have come to peace with each other. So Demis then says, Dario's essay points toward the right path forward. The details need working through, but the direction is correct. And then he references back to the essay that I had mentioned earlier. Andres Karpathy, who's a legendary researcher. He happens to be at Anthropic right now. He was early at OpenAI. He said, I love this and really hope we can come together as an industry and make it happen, regarded the embedded evaluators. Satya shows up on, like, Saturday or Sunday. I don't remember when this one came in.

25:11But he tweets, any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it is not worth pursuing. We welcome the research focus and deliberate pacing needed to get alignment right as the design goal. We also welcome ideas like embedded evaluators and the broader efforts to develop the mechanisms to make this more than just talk. The key is that this cannot be controlled by a handful of entities but must have broad representation across ecosystems, countries, fields, including academia.

25:43Now, Microsoft's been touting the human control thing for at least the last, like, 12 months. So, that fits very well with their messaging. Okay. So, responses from critics was one thing I was paying attention to, which I wish I didn't have to, but whatever. So, there are a group of very prominent, usually venture capitalists, who think that everything related to AI risk is an attempt at regulatory capture and an attack on open source models. To them, there is no middle ground.

26:13Like, if someone is tweeting about this happening in labs, then Anthropic is just going after regulatory capture. That's basically the gist of any criticism you would see online roughly falls into the area. They're going after regulatory capture. They think they're the only ones who can do it safely or they're coming after open source models. Now, there might be elements of that that are true, but that's it. Like, that is the arguments they make. Then we get the politicians jumping in. Now, immediately, as soon as I saw Jacob's tweet, I was like, son of a bitch. Like, the kill all humans thing is all anyone's going to talk about.

26:45Like, they're just going to immediately zero in on this one thing. And, of course. Okay. So, Senator Chris Coons, I actually have no idea if he's a Democrat or Republican. So, this kid, if you're new to the show, I don't give a shit about politics. Like, we are completely neutral. We are like, what is the most sense for humanity? I think he's a Democrat, maybe, but I don't know. And I actually didn't even want to look it up to be truthful. So, he said, it's time for the Trump administration to wake up. Okay, maybe he's a Democrat. To the existential threat posed by AI, this must be the top agenda item when Trump and Xi meet later this month, meaning Xi Jinping in China.

27:23And Congress must prioritize passing meaningful guardrails in this space before it's too late. Now, I'll tell you what, like, had Josh Hawley tweeted the same thing, who is definitely a Republican, he probably would have said the same thing. So, like, again, I don't know. This is a bipartisan thing. Now, Speaker Johnson, who is definitely a Republican, he tweeted this morning, Congress has been studying the AI issue since I set up a bipartisan task force to do soon after I was elected Speaker. We all have a sense of urgency to create guardrails around the technology.

27:54The key is designing the guardrails carefully in a way that prevents any harm from AI while also preserving American innovation and our national security by keeping our edge over China and other international competitors. I am calling a meeting in Washington with AI platform providers and key experts to determine the right course forward and discuss responsibility providers have to ensure. Now, keep in mind, he's not calling the House back, though. Like, he's just calling platform. But anyway, so now Johnson probably needs to have a conversation with Trump because Trump was asked Sunday morning about this very thing.

28:27And he said, it said, when asked about a potential slowdown on Sunday, Mr. Trump said, we're leading China in AI. We're the most sophisticated country in the world. And frankly, I want to keep it that way because whoever wins, AI wins. And we put guardrails, we can put up guardrails, we can do this and that. But I think you have a lot of very negative forces that are bringing it up that shouldn't be bringing it up. They're bringing it up, up things that won't happen. And so, Trump and Johnson, not currently on the same page.

28:58Okay, so then what does China think about all this? That comes out Monday morning. This is September 14th.

29:04Okay, man, this is the big one. All right, so, okay, so go back to Dario's letter. Why would China get pissed about this? Like, hey, this doesn't seem bad for humanity. Like, let's find an agreement. Okay, well, within Dario's letter, though, I'm just going to quote this. Pacing within democracies, which China is not, will be limited by the lead that U.S. companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then on paced, Chinese-associated projects will pull ahead, creating significant national security risk.

29:40So, what he proposes is do not sell powerful AI chips or semiconductor manufacturing equipment to China and crack down on chip smuggling operations and remote access to data centers outside of China, which he's saying they're doing to bypass the fact that we're not selling these things. Chips will be the main determinant of China's AI strength. They also want to crack down on unauthorized distillation by companies in authoritarian countries. That means stealing models from the U.S. and, like, distilling them to build their own models.

30:13And then strengthen security at the AI companies and prevent model-weight theft. So, you know, pacing's fine, but then Dario sort of, like, goes right at China. So, China's not so happy about that. So, in Reuters this morning, the Global Times tabloid said the true agenda of Amadei's essay was to attempt to curb China's AI development through technological barriers and regulatory monopolies, uphold Washington's monopolistic hegemony—hegemony—I don't even know what that word means. Hegemony. There you go. I guess that, yeah. In cutting-edge technology and exclude China from the global AI governance system.

30:47And then this is the real one. This is a quote, by the way. This silent AI cold war is hypocritical and short-sighted. It said, nothing, adding that excluding China from this innovation would significantly increase the trial and error costs and risk of loss of control in global AI development. Okay, so all that being said, here's just my overall take. And again, this is pretty, like, raw. Like, I literally just made these notes. I have not thought deeply about this, exactly what I'm going to say here.

31:16Okay, so there's something called PDOOM or probability of doom. This is a term that has been going around in AI circles for well over a decade. And the basic premise is, what is the likelihood that AI will kill all humans? This is not a new term by any stretch of the imagination. And so for a while, AI researchers would, like, hey, what's your PDOOM? Like, what's the chance you think that we just, like, kill all of society with AI? I think it is an absurd concept. It is assigning probabilities to something where there are all of these variables, and it's the thing that people are going to latch on to.

31:53So you have some people, it's like, oh, it's 50%, it's 10%, it's, well, it's 90%. Like, they literally just make a number up. Like, it would be like me saying, well, I don't know, I think there's, like, a 10% chance aliens show up in the next three years and just decide that, you know, we're basically like ants and that they don't need humans. And, like, if someone who happens to be on the inside of the UFO program within the United States, like, shows up and says, yeah, I think there's, like, a 10% chance they're already here. Like, okay, like, that's a probability that you've assigned to something that has no scientific backing to it.

32:26So I would just encourage people, this whole human extinction thing, it's just, like, losing sight of real near-term risks. Now, I am not saying that there is no chance that some scenario emerges where stuff just goes completely sideways and it really is just bad for us. I could probably give you, like, 10 other things outside of AI where the same thing could be true and, like, maybe it happens, maybe it doesn't. I don't know, an asteroid could hit tomorrow and that's not going to be good for any of us.

32:58So I would just set aside the whole kill humanity, human extinction thing. It is, like, it is distracting us from the fact that there are very real near-term risks that we should be focusing on that will cause people to actually start thinking more realistically about AI. Like, this doom thing just makes people, it's like, oh, it's all bad and we lose sight of all the positive things that it can have.

33:30So I do think that we have to slow down the model releases. Like, I'm very convinced of that. I said this on last week's episode that GPT-6 Astra changed the way I thought about this stuff, that I had more dread than excitement about that release. I think they need to slow this down. They obviously don't have control of these things. Like, the Hugging Face incident, Anthropic had similar incidents. We've learned about other incidents with OpenAI. They don't have control of these agents that they're building. So we got to figure that out.

34:00They need way more resources dedicated to safety and alignment and there has to be some level of government oversight. I don't have any confidence the administration can do it, but that doesn't mean we shouldn't try. The way I always think about this, Mike, and I hope this makes sense, but, like, think about all the things in society that have some level of regulation. So cars, airplanes, chemicals, pills, food, tobacco, medical devices, manufacturing plants, nuclear power plants, consumer products. Like, all of these things, you have to prove they're safe before you put them into society.

34:33Why would AI be any different? The risks of AI are greater than many of those things, and yet we just get to have a grand experiment on society every time we want to put out a more powerful model. So that, to me, you can call, you can get stuck in the regulatory capture, whatever, but then you have to address the fact that it can cause real harm. I'm not on the PDOOM, like, extinction of humanity harm thing, but, like, I mean, I can see a very realistic scenario where it takes down, you know, infrastructure, power grids.

35:07Like, that is not hard to comprehend. Wall Street, like, takes down trading. Like, all of that stuff is super viable, so why wouldn't we have some way to test for safety before we put them into the world? So I feel like they need to do something, and I think people who argue against that are just doing a disservice to society, as I said. I don't understand how that's a viable approach. And then the big picture here, and this is maybe just me being optimistic, is, like, I think it's good that we have an awareness now.

35:39Like, I feel like a lot of people just weren't awake to the fact that there's, like, super real risks here and that things could go bad. But the faster we get to talking about those things and dealing with them, the better we can get to, like, all the good AI can do, all the positives it can bring to society. So, I don't know, I would say, like, for people individually who are listening to this and maybe are just like, oh, my God, like, this is too much. Just go to work tomorrow and, like, focus on applying the tech we have in a responsible way to enhance human potential, to unlock what we're doing, to, like, work on higher-level cognitive, creative tasks.

36:20Like, most of you, your jobs are not going to change dramatically in the next, you know, 1 to 6 to 12 months. It's certainly not in a way where AI is just going to, like, destroy humanity and society. And I think, like, we need to just focus on the things we have control of, which is there's really capable models that, when applied well, can make your job more fun, like, more fulfilling. But it requires organization leadership to have a vision for how to do that.

36:52And I think most of us should just be focusing on that. Like, how do we integrate this current technology that we already have, GPT-6 Astra, Fable 5? Like, they're good enough. Like, if we shut off model development today, we got three to five years before most enterprises fully integrate the tech we already have. So, that's what I would say is, like, I get why people are worried about this stuff. I get why, you know, the media headlines can be unnerving.

37:19I think it's going to cause action, and I think that's a good thing in society, in government. And I think most of us just need to stay focused and optimistic in doing our part to try and bring it responsibly within our school systems, our businesses, and our own careers. And, like, don't let all this other stuff distract you from that. So, Paul, I had a few quick questions for you around this. So, just to be perfectly clear here on your position, kind of what you've outlined, first up, there's a lot of crazy headlines about the motivation for Jacob and others to be doing this.

37:58You mentioned regulatory capture. There's people literally saying it's like a Democratic Party psyop. Just to be clear, and I agree with you, I think, like, this, you're, regardless of the other effects of this, it sounds like this is motivated by something they've actually seen and extrapolated as being developed in a lab. This is not made up. A hundred percent, yes. Right. Now, again, people are going to dig in and find all kinds of stuff about who is this guy, who's he connected to, who were the first five people that retweeted the thing.

38:29Like, I'm not saying that that stuff's not real, that there wasn't, like, a coordinated effort to try and get more attention around the risks. Yes. But I think the motivation for why they did it is because they're seeing things that we aren't seeing, and they are terrified of what's being built. And that the scaling of this recursive self-improvement, we basically have six to 12 months to figure it out. And once they've solved it within these labs, then anybody building open-weight models can do the same thing within a year, and then we've lost control completely. And I think that they think that is very real. Gotcha. And then one other question, I'm just curious if you have any perspective on this.

39:03As I'm reading Dario's essay, right, and it's, A, it's very interesting. It's, like, kind of a declaration of war on China. It's very true. Yeah, I would say. But given that, isn't this kind of, like, a prisoner's dilemma? Like, aren't you, you have to, let's say OpenAI and Anthropic do everything perfectly. That's really good for their models not escaping or doing unintended things that screw over Wall Street or cybersecurity issues or create billions in damage or lose lives.

39:34But doesn't this require literally every single person involved, every lab involved, every entity involved, to do the same thing? Or will, is there something unique about OpenAI and Anthropic doing this that would stop catastrophe? That's why they need to get the other countries involved. You know, and I do think that you're looking at, like, nuclear weapons as probably the closest parallel to, like, being able to negotiate treaties where, you know, you try and control it. I mean, there was an interview Dario did, I think it was on CBS, where he was kind of implying that he would give up control to nationalize the technology that he never understood why private companies are even building the technology that they're building.

40:14And they pushed him on it, and he kind of backed away from, like, well, I'm not going to, like, literally just give it to our government. But, like, a collection of democratic governments, potentially.

40:25You know, again, I don't want to go down the conspiracy theory path much here. But I would imagine that DARPA and other government agencies are not going to probably sit around and wait for a few private labs to develop this and then, like, hopefully share the weights with them. I would think that there is likely a very large-scale initiative underway that we may never hear about to build the most advanced frontier models that the government can control a version.

41:00Because you could definitely see an argument where nationalization of the labs is in the best interest of the way that the U.S. government would look at these things. Like, that they could convince themselves that nationalizing this technology is maybe the best path forward. Now, I'm not endorsing that. I'm just saying I could see them thinking that. That's what struck me as so crazy about the Evan response to Jacob where he said greater than 10% chance that this kills all people.

41:34If I told you that a non-government group said that they had developed technology that they believed had a greater than 10% chance of killing all of humanity, let's say that group was not based in the U.S. What would happen tomorrow is the U.S. government would conduct a series of drone strikes on that. That's why I was like, this is so strange to me. You just say this? Whereas the logical conclusion is, okay, you should all be in jail at a baseline because we are preventing mass extinction.

42:09You should be nationalized. I'm not saying I agree with any of that. But that's where this logic goes. Yeah, and to go back to the point I made about, like, the analogies of all the things that have some level of regulation. Yeah. Let's just – and again, I don't even think this is a far-fetched thing, and I'm thinking of this off the top of my head. So let's say that there was three pharmaceutical companies who were developing a cure for cancer and that they had made immense progress and that they wanted to put it out into the world because they think they can cure all cancers. But there's a 10% chance it actually ends up killing everybody in 24 months because, like, there was a sleeper element to it that we didn't know.

42:46But, like, it's so good we just wanted to get out into the world on the chance. There is zero chance that that drug gets introduced into society. Zero. And yet we have AI where the people building it are convinced that there is some possibility that it ends really badly for all of us within a decade, and yet there's no real oversight. They can just put things out in the world. They can run these, like, lab experiments. It's like gain-of-control experiments on, you know, when you're developing viruses and stuff. Like, they're just doing it unregulated.

43:18And that I don't understand a world where that makes any sense if there's these risks. So, yeah, it's, again, like, I feel like at least we're now having dialogue. There are certainly some people who are not having, I would say, again, they just have an agenda and they're stuck on their agenda. And I think it's just best to ignore those arguments if they're not also willing to listen to both sides of it. And I get that this is maybe the longest main topic we have ever had, but hopefully it's really important to everyone to kind of understand that this dynamic is now well beyond the labs.

43:54We are truly into the political game. We are into the impact on society. It is going to own the media and the politics for the months ahead. Like, this is, we have definitely hit an inflection point. Whether, how this tweet ended up being the thing, I still don't really comprehend. But it's been building for a while and we definitely, we hit a point where things are just going to be different now. And we'll move on just one second here. But I do just want to encourage people, if you have family members or friends who are asking about this, I would, you know, shamelessly say, like, forward them the beginning segment of this episode.

44:29Because I think, Paul, you've done a really good job of framing this. Because I don't know about you, this is the number one topic this year, so far at least, where I've gotten commentary and questions from people that don't ask me about AI. And they're freaking out, basically. Yes, texts from friends and family. It's not a good vibe out there with this stuff for people that don't follow it. So I think this episode, or at least this segment, could actually really help some of the, like, I explained it to my mom, for instance, a little like you have. And she was like, okay, I don't know if I feel positive about what's happening, but that's a lot more helpful than kind of what I was reading or seeing.

45:05Yeah, and I understand if people don't still feel positive. Like, I totally get that. But I think, yeah, the point I'm trying to make is, like, do not get caught up in the whole end of humanity thing. Like, it's just, it is not productive at all to, like, be going down that path right now. All right, let's move on our second big topic this week.

OpenAI Navier-Stokes mathematics breakthrough

45:29OpenAI announced that an internal AI system had produced a proposed solution to the Navier-Stokes Millennium Prize problem. This is a longstanding mathematics question about how fluids behave. The company released a written proof and a formal version in a system called Lean, L-E-A-N, which is software that checks mathematical arguments. They said the model behind the discovery is significantly more capable than GPT-6-Aster, so an unreleased model.

46:01So the result of this problem, basically, we won't get into too many details, concerns, whether an initially smooth three-dimensional fluid flow can develop a singularity, where the equations predict speeds growing without a limit in finite time. So basically, a very, very hard, longstanding, unsolved math problem about fluid dynamics. Now, to solve this, OpenAI says that they involved roughly 10,000 agents working concurrently, and the system orchestrating those agents reached its result about 88 hours after the effort began,

46:37with another 17 hours for Lean formalization and verification, which you have to do to basically verify that you have the right solution. The company, however, says it does not intend to claim the Millennium Prize. Now, this came with some big controversy, because right around the announcement, there was this dispute that came to light with New York University mathematics professor Tristan Buckmaster and his collaborator, an anthropic employee named Levin Alpoch, and their separate project, which Levin was doing with Tristan just in a private capacity.

47:12This was not like an anthropic-driven initiative. They were just working on this problem using multiple AI tools, and they developed related results for a related math problem called Euler equations. And Buckmaster says that this collaboration was totally personal, not an anthropic effort. But then, Buckmaster says OpenAI researcher Sebastian Bubeck proposed that Buckmaster write up OpenAI's results, since they kind of got to the same conclusions, it sounds like, or roughly along the same direction,

47:42without Alpoch, who works at Anthropic, as an author because he was employed by OpenAI's competitor. Bubeck said the discussion concerned rewriting OpenAI's work to include Tristan not removing Alpoch. He apologized for kind of how he had handled it. Buckmaster wrote this, like, four or five-page, essentially, PDF he released of, like, emails and messages saying, like, hey, I'm not accusing anyone of anything, but I am saying it's kind of strange. They arrived at a similar result using a similar direction that's not widely known or would not have been obvious if you weren't building on someone else's work

48:19because they had kind of heard rumors, OpenAI apparently had, that Anthropic was getting close to solving this exact thing. Buckmaster also, I read the whole thing pretty respectfully, I guess. Like, he wasn't, like, coming out in super hot, but he questioned whether the drafts that him and his colleagues had entered into Codex, which is one of the systems they used to do this work, influenced the work. OpenAI said that this was not the case, that the prompts had not influenced the system, including through training.

48:52So there's all this controversy, Paul. Basically, super historic mathematics result kind of plays into a lot of the stuff we just talked about, a lot of the supposition that AI is actually going to solve longstanding mathematical and scientific problems, but then there's this cloud of controversy. What did you take away from this, or what kind of struck you about this? Yeah, first, I have no idea what the hell this actually means, like, this mathematical equation. I've read this, like, five different times, and I still don't comprehend it fully. But I'm also not a mathematician, so I don't know that I'm supposed to really comprehend this.

49:25But I did try and start with, like, okay, what is the significance from a mathematical perspective, from a science perspective of this? And in the OpenAI post, it says, these equations are used for aircraft design, weather forecasting, and the study of blood flow. So it's like, okay, well, there's a few tangible things of why this is so relevant. The thing I wanted to focus on, though, is the fact that less than two years ago, so before we got reasoning models, the O1 model from OpenAI, the most advanced models from OpenAI couldn't reliably count the number of R's in strawberry.

49:55And here we are two years later, solving Millennium Prize problems. So the speed of the improvement of the models, to me, is maybe the biggest story of all here. You touched a little bit on the backstory, but OpenAI did go into a little bit more detail about what exactly happened because of all the backlash that they were getting right out of the gate. So Sam tweeted right when this happened, one of the most amazing moments for me in OpenAI history was watching this happen over the past week. The world has extremely capable models now.

50:26I did not expect a result of this magnitude to happen so soon. This goes back to the whole thing I mentioned about the models are getting better, faster than they were expecting. He said, we have been talking a lot about the need to pace progress to ensure safety. For me, this is the strongest evidence yet of the urgency. So again, kind of spooked by the fact their models could do this. He also said, it is true that we tried this because there were rumors on the internet last week that Anthropics models had solved a Millennium problem and we were curious if R's could do it too.

50:57So what he's saying is they got wind that Anthropics latest models, 100 Training, had done this thing and they were like, huh, I wonder if R's could do it, is the story they're giving at least. So the post that they put up, it said, since August 28th, so again, just to get perspective on how fast this came together, we have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics. This model's training is ongoing and its performance continues to improve. On Tuesday, September 1st, they heard the rumors about two Millennium Prize problems had been resolved.

51:29Inspired by these rumors and by the step change in performance of our internal model, we launched an effort to evaluate it on all open Millennium Prize problems and a few other high impact problems. You alluded to this one, the agents arrived at the resolution on Saturday, September 5th, so four days later, about 88 hours after the first agents were launched and then 17 additional hours with GPT-6 Astra. It says, across all attempted problems, the agents sent 4.9 million messages and used about 300 billion output tokens.

52:03In the process of resolving the Navier Stokes problem, the agents sent 2.7 million messages and used approximately 130 billion output tokens. Our goal in releasing this result is to report on the substantial progress of our AI models. We believe we are now in the next period of AI progress. This is the important part. We believe we are now in the next period of AI progress and today's results provide further evidence of this. We are focusing on understanding this model, meaning the one that has not been released yet, and using what we learned to help us guide and pace how we pursue further advances in capability.

52:39One of our key goals is to build AI systems which are steerable, accountable, and connected to people, which may require more deliberate choices about the pace of progress as we continue our mission to ensure AGI benefits all of humanity. So, again, all about pace. The other thing I started thinking is, like, well, what does this mean to other grand challenges? Like, can we really cure cancer? Like, there are big problems that they just showed. Give it four days and, like, unlimited tokens and, like, what can we solve? So, I do start to think about that. It's like, well, maybe we are within, like, a year or two of solving all of these diseases and all these things because it just requires compute, apparently.

53:16And I think they said it may have cost, like, several million dollars of compute. Yeah, a few million. But that's a rounding error compared to how much revenue they're able to generate. And, like, tokens, they have unlimited tokens. That's a pittance to solve some of these things. Yeah, and, like, the good for humanity. Like, I – so, that was my – I was, like, was this projectable? Can we do the same thing? Now, mathematics is provable. Like, there's a goal to head towards. But, like, I don't know. So, that gave me kind of hope. Again, it's, like, there's a negative to this, but it also is, wow, this – maybe we are heading down this path of abundance.

53:49What does it mean for industries or employment that they throw the same reasoning capability at or that someone throws an open weight model at in six months and says, oh, let me go take on the legal industry or the finance industry or whatever. They just demonstrated an ability for these things to learn in days and solve really hard things. The one thing that did catch on, Mike, was this whole idea you talked about of, like, the mathematicians questioning whether maybe OpenAI learned something from their codex projects.

54:19So, John Shulman, who's no longer an OpenAI guy, he's at Thinking Machines, but previous OpenAI, I believe, a researcher we've talked about on the show before, he tweeted, as a follow-up, it's exceedingly unlikely that training on user data contributes much to frontier model gains in areas like math. Those come from scaling up pre-training and reinforcement learning. User data is more likely used to find failure modes or situations that are hard to recreate. That said, he hedged, and I thought this was a really interesting hedge.

54:51Model companies vary in how aggressively they train on user data and uploading repos isn't hypothetical. I wish there were stronger norms around disclosing how companies train on user data with what methods and to improve what capabilities. So, meaning, like, maybe, like, it's possible that something leaked in. So, and what I immediately went to, it's like, wow, okay, so a lot of companies, a lot of IT departments, a lot of legal teams, when they sign their terms of use with OpenAI and Google and others, they assume their data is not being used to train anything.

55:26Right. Now, while they may anonymize that data, that doesn't mean the data doesn't exist and that it couldn't, in theory, still be used in some way to inform what the models do. So, if you're putting proprietary ideas, frameworks, intellectual property into these things, like, it does make you really question, like, is it really walled off? Right, right. And then my last note here, Mike, is actually related to mathematicians and how they responded.

55:58So, Terence Tau, who we've talked about before, Fields Medal winner, became the youngest recipient of the International Mathematical Olympiad at age 13, by 16, graduated with a bachelor's and master's degree in mathematics. He and 24 other Fields Medalists, which are highest international honor bestowed upon mathematicians, kind of like the Nobel Prize of Mathematics, they published a letter that they all signed on. It says,

56:53It says,

58:23By the companies developing these new technologies and, more broadly, by a society that will confront similar problems in many other forms of intellectual work. So, I wanted to share that, kind of as my final thoughts here, Mike, because there's definitely some people who are like, wah, wah, like, you brilliant mathematicians, like, they solve the thing you spend your life on, like, get over it, it's better for society. And then there's the opinion of the people, like, but we don't understand how they did it, and we didn't learn from the process. And by that, as a human species, like, we didn't actually evolve and improve.

58:55And so, I think it's this pull, like, it's my Move 37 moment that I shared for knowledge workers in my Mekon keynote last year of, like, we're all going to come to this point where it's like, damn, it's better than me at the thing I spent my life doing. Now what? Like, and I think that's where all of a sudden mathematicians arrived at, is, like, there are mathematicians who would spend their entire lives trying to solve one of these equations. And, hey, I showed up in four days with 10,000 concurrent agents and solved it.

59:26And, like, what happens now? So, yeah, I don't know, it's just worthwhile to at least ponder. Yeah, and, like, valid questions for sure from the mathematical community about keeping inside the actual purpose of solving these, but that's not going to stop the sense of dislocation or perhaps, I don't know, emptiness that you might feel. Right. If this happens in your field. Yeah, and that's why we've said, like, many times, this isn't just an economic discussion. It's not a technological discussion. This is, like, a human condition discussion around, like, purpose and meaning.

1:00:00And I think people who belittle these people lack empathy. Like, how could you look at something like this with these people who this is their life? Like, it is the thing they work on. And, like, the way you respond to this is, like, yeah, get over it. Quit whining about it. It's, like, it's so, it just lacks complete empathy. And so I feel like we need to do more, even on the psychological side of all of this, the, you know, sociological side of all this. Like, it's, these are just the discussions I've been waiting for years for us all to have.

1:00:31And so as hard as it is to now have to deal with these things, as it seems like all at once, at least we're dealing with them. Because these are very, very important discussions that should be happening. And more, to me, relevant than 10% chance of P-doom. Right. I wanted to have these conversations. Yep. 100%.

The Economist jobs report

1:00:50All right. Our third big topic of this week concerns one of our recurring favorite topics, AI and jobs over here. We had The Economist this past week publish a report arguing that the AI jobs apocalypse has been postponed and that an AI jobs boom is already here. The publication estimated that AI has created around 1 million American jobs so far compared with roughly 200,000 layoffs attributed to AI since mid-2023. Those are The Economist's estimates. They're not an official government count of AI's net effect on employment.

1:01:23Now, a lot of this growth, part of it, a significant part of it at least, comes from building AI infrastructure. The Economist tracked five industries tied to data centers and found that roughly 320,000 more jobs since 2023 were created than broader trends would suggest. It also reported growing demand for AI engineers, people who adapt AI systems for customers, and executives responsible for deploying the technology. The latest government jobs report also showed continued overall employment growth. We just got a Bureau of Labor Statistics report that the economy added 162,000 non-farm payroll jobs in August with unemployment unchanged at 4.1%.

1:02:01That kind of crushed the expectations for that month. The gains did include restaurants and local government education while the information industry did lose jobs. Now, the report does not attribute those changes to AI, but just kind of a macro picture of employment still being relatively strong, it sounds like. The BLS has actually also introduced an AI exposure category alongside its new employment projections. So the agency explicitly says that if something is exposed to AI, that does not imply job loss,

1:02:32but its categories combine estimates of which tasks AI could affect with evidence of tasks people are already using AI to perform. Finally, there's been some recent research from Ramp, which is a finance startup tech company in Reveglio Labs, which covers more than 21,000 American firms. They found that companies investing most heavily in AI actually increased headcount by about 10% over the two years following adoption, which I believe is a stat we've referenced before, but just some additional context here. Our entry-level headcount at those firms grew 12%.

1:03:03So, Paul, it's kind of an interesting final topic here, given the first two we've discussed. I don't know. It doesn't seem like the data is off to me, but maybe more of a referendum on, hey, the data center build-out is creating a lot of great jobs. Yeah, I mean, and that's never been debatable. Like, the data center build-out is great for laborers. It's great for the growth of the economy, but people don't like data centers. So, it's like everybody wants jobs. They want to see this growth. They want to tout this growth, and yet some of the same people are turning around and, you know,

1:03:36trying to ban data center build-out in their states. So, you can't have it both ways. Either you want the jobs that are coming from the CapEx spend of the major AI companies and the data centers they're building, or you don't. Politically, they're going to claim both, that they're leading states that are creating jobs, but then they're going to campaign against data centers out the other side of their mouth. So, it's a challenging spot to be in because it is the AI investment that's driving the growth of the economy at this point. And I also think the other variable here is enterprise adoption of AI is still so early,

1:04:11and people shouldn't make assumptions about the safety of knowledge, work, jobs, of the white-collar jobs, just because the data right now is positive. Now, that being said, like, I love to see the positive numbers. Like, this is great. Let's keep it going.

1:04:26What we just talked about for an hour, though, is, like, the potential of, like, a bit of a slowdown, and you now have an economy that is 100% dependent upon this AI build-out to keep growing, and that could cause some problems if that growth doesn't keep happening. In the Economist report, it said data center construction alone is proceeding at an annual rate of more than $75 billion, nearly 60% higher than a year ago. That building spree requires armies of workers, electricians to wire server racks, HVAC specialists to stop these from overheating, grid engineers to hook them up to the power supply,

1:05:00and technicians to install and maintain the machines. So they specifically looked at electrical contractors, HVAC and plumbing, utility system construction, commercial construction, and electrical equipment manufacturing. So, yeah, no-brainer. Like, of course those jobs are going to grow. So they did also kind of hedge towards the end. It said some professions are suffering. Since January 2023, employment has fallen by about 10% among customer service workers, roughly 15% among secretaries, administrative assistants. All three are heavy on routine tasks at which AI agents increasingly excel.

1:05:33And then there's just variables at play here. Government actions or regulations are going to play a role in whether this growth continues. Human friction to change and lack of change management planning by enterprises is going to slow the impact on jobs. And then the lack of vision and strategy overall to implement the technology we already have definitely is, in a positive way, slowing down the job displacement. So I've said it a few times on the show, like, the inability for enterprises to change quickly

1:06:04may actually be the friction that saves the economy. Right, right. Because if you went in and just, like, took a bunch of AI4 leaders and, like, modernized marketing teams, sales teams, CS teams, there's 20%, 30% efficiency gains in the first week if you go in and do that. Yes, yes, yes. And so if we just rolled out GPT-6 Astra, Pable 5, into every enterprise across their entire workforce and trained them on a personal level how to use it,

1:06:36there's no way that the economy could handle the amount of job displacement that would come from that. But so far, the saving grace is enterprises are really, really, really slow at adopting AI across the full spectrum within their company. Yeah, that's such a good point. And also, you know, something maybe lost here or forgotten here. Again, I love these numbers. I love seeing this. But it's, like, it's amazing if you are in the trades, if you're an electrician, if you're one of the people involved in these. I love it. More power to you.

1:07:06Everyone does well when you do well. But it's also a thing where it's, like, I don't think it's super realistic to expect a bunch of out-of-work graphic designers to become electricians either. Or entrepreneurs. Or entrepreneurs, right. And not to say there's no job outside of that. We'll see. It's going to be murkier and messier than, you know, we're talking about. But it is important to think, like, it's not just, like, switch to an immediate other job if AI is very, very good at computer work, browser work, digital work of all types. Yeah. And that's the important stuff they're not going to really talk about.

1:07:39But, yeah, so you tout the growth in the jobs. But then it's, like, yeah, not everybody's going to become an electrician. Like, it's not what they're fit for. But if you have kids that are in the trades or want to be in the trades, go. It's going to be the glory days in the trades for at least the next decade. Like, it's a great profession. And there's tons of money to be made, tons of opportunities to be had within the trades.

Anthropic economic scenario explorer

1:08:06All right. Before we dive into our rapid fires this week, Paul, this week's episode is brought to you by Mekon. This is our Marketing AI Conference happening October 13th to the 15th here in Cleveland, Ohio. Mekon is three days of keynotes, sessions, workshops, and conversations built specifically for marketing and AI business leaders and marketing and business leaders who are actively figuring out how to adopt, operationalize, and scale AI across their organizations. So today, you can use the code POD100 at checkout, and you will save $100 on top of locking in

1:08:39the best rate available. So go visit Mekon.ai to register. That's M-A-I-C-O-N dot A-I to register today. All right, Paul. Diving into rapid-fire topics this week. First up, Anthropic released their own interactive economic scenario explorer this past week. They were looking at how AI could affect the American economy by 2030. So you can go use this interactive assessment or tool, enter assumptions about AI's capabilities, adoption, autonomy, productivity, and how quickly displaced workers find new jobs.

1:09:13And then you can see the economic outcomes this model produces. The company in their modeling here highlighted basically three scenarios they see. Modest, substantial, and extreme. So modest is pretty typical modest economic growth due to AI. But in the substantial scenario, AI is capable, they say, of doing half of knowledge work by 2030. But most knowledge work tasks still happen without it. As a result, economic growth roughly doubles its normal pace, while knowledge workers' wages

1:09:44stay essentially flat relative to a no-AI scenario. But really, the extreme scenario is what's kind of worth considering here. They assume in this scenario much more capable AI and rapid adoption. In that scenario, based on their modeling, annual GDP growth reaches 15%, but nearly one in five people in the cognitive labor force is unemployed by 2030. The model also shifts a much larger share of national income towards capital rather than workers. As part of all this, Anthropics surveyed more than 10,000 Americans about their expectations

1:10:18around this. And the typical respondents' answers kind of modeled more outcomes closer to the substantial scenario. So that measures how people's beliefs translate through this model rather, you know, rather than saying it's likely this future is going to happen. The authors here explicitly say the scenarios are not predictions. They assign them no probabilities, and they admit they leave out several major forces in their modeling, including advanced robots, policy responses, and business cycles.

1:10:48Anthropic is going to update this tool as the evidence and research behind it develops. So kind of interesting to at least just see some modeling and conversation out here, Paul. I mean, I think I'm in the vast minority of the people they surveyed, but I don't see how the, what did they call it, the extreme scenario. That seems possible to me over time. Yeah. Again, it's the human friction thing that could keep it from maybe having the impact. Yeah. Yeah. So they follow a similar model to how I built JobsGPT.

1:11:18So a few years back, I created a tool to try and predict the impact on jobs, both through exposure levels as the models got smarter and more capable. Still a super valuable tool to use. You can go to smarterx.ai forward slash JobsGPT and use it for free. It's a custom GPT built in OpenAI. But what it does is it uses the O-Net database, which is the taxonomy that they're using, and it breaks Jobs down into tasks. And then they basically look at Jobs and say, okay, as we move forward, there's going to be tasks that are unchanged by AI.

1:11:50So like they gave the example of AI can't bathe a patient. So if you're a nurse, you're still doing that. There's tasks that are augmented, tasks that are automated, and then new tasks created. And so what they're trying to do is look at each role and say, how much is this job going to change due to AI, which is a super practical way to do it. It's how we teach it. Like just break your job in a bunch of tasks and say, what can AI help me with in essence? The profound economic transformation one, Mike, that you're talking about, the extreme scenario, they actually say this scenario would likely require recursively self-improving

1:12:23AI adopted quickly for knowledge work. So going back to full circle to our first topic of the day, in that one, as AI diffuses, annual GDP growth rates reach 15% a year, which is significant, leading the economy to double in size every four and a half years. My whole thing on this, again, is like, even if the tech is capable, which I think it will be, like technologically, there's no doubt in my mind that within four years, AI is going to be probably superhuman at all cognitive work. Like I don't, unless they just stop completely building these things, it's going to be very

1:12:57rare that you would find something a human could do that AI can't do at the peak level of a human. But the thing to go back to my comment in the previous topic, ChatGPT is almost four years old. In like two months, it's going to be four years old. And yet the majority of enterprises are still early in the early stages of diffusion of that tech across their entire workforce and most lack personalized training to optimize the use of it. What I mean by that is most enterprises that we go into and we talk to some major enterprises

1:13:27are still at the point where they're trying to assign AI assistants, co-pilot, ChatGPT, whatever, not even like the advanced versions of them, just the assistant version, licenses to all of their people. And then to train them how to use those licenses. Like I get that people live in like the tech bubbles and they think that like every company has this all solved. It is so far from the case when you go into major enterprises and you go talk to marketing teams, sales teams, ops teams, HR teams, we're just so early.

1:14:01And so even if the tech gets there, I don't know, like four years sounds like a long time and yet I look back over the last four years and I'm like, God, like most organizations have barely moved the needle yet with AI. Well, it's interesting to admit though that that's why I found most interesting about the extreme scenario. It's like, you're right, like 2030, probably not. But like 2040, those numbers are crazy. 2050, those numbers are crazy. Like that's a utopia if we get to that in 30 years, I think, you know? Yeah. And so much can change. Like I said, there's so many variables that affect this stuff now that it's become so

1:14:34political and the risks have gotten so high. But again, I mean, I just feel like if you gave me GPT-6 Astra or Fable 5.1 and just went into every company in the world and you taught them how to use those technologies, just the AI assistants and maybe some agent stuff. But like, I don't even know that you really need that much agent stuff to totally transform every company. Especially if, let's say, 18 months from now, token costs are like so low. You know, it's like GPT-6 with just like not having to worry about usage. It's like game changer. All right.

Nvidia CEO declares AGI has arrived

1:15:04Next up, after we covered GPT-6 Astra last week, NVIDIA founder, president, and CEO Jensen Huang declared publicly, quote, AGI has arrived. Huang posted congratulating OpenAI on Astra. He pointed to the NVIDIA hardware behind Astra in this AGI announcement. He said 400,000 GPUs were coming online next year. However, you know, typically the post did not specify a definition of AGI or any type of test that the model had passed to meet this. However, OpenAI chief research officer Mark Chen responded that he agreed that they were

1:15:38entering the AGI era. He paired that statement with a call to make this the alignment era, including training AI monitors capable enough to supervise the systems they oversee. See, Paul, this is the reason we're mentioning this is like, this is something you had predicted. Someone's just going to declare we are at AGI. Jensen is like, we've got it. It's Astra. What does that actually mean? Yeah. Jensen wouldn't have been the first person I would have guessed would have been the one declaring it since he just joined Twitter like six weeks ago. Yeah.

1:16:10Yeah. I mean, I don't know. Like, so I think it was last two years ago at Macon, I did the road to AGI and beyond. So I did an entire keynote on this exact thing and sort of projected how this was all going to transpire. The way I think about AGI right now is it's a meaningless term. It's just become meaningless because everybody's got different definitions. The thing I am very confident in is if you would have in November of 22, when ChadGPT came out, if ChadGPT in 2022 was in the form it's in today, I think unanimously AI lab researchers

1:16:43and leaders would have called it AGI. Like for what they thought AGI was going to look like, if you would have given them today's technology four years ago, they would have called it AGI. They called GPT-4 sparks of AGI in the paper that came out in spring of 23. So I think that they would have definitely considered it. And I think three or four years from now, when we look back, it'll definitively have been considered AGI. It's just the moment you're in and like the perspective you have at this moment.

1:17:14And it's like, I don't know, like maybe it is, maybe it isn't. We don't even know what it is anymore. If you're open AI, you got to change your whole mission statement because like mission achieved, like if you say, yeah, okay, it is AGI, then you got to update all your marketing materials. I don't know. Like it's just, the world's changed. The tech's insanely smart and capable. You can call it AGI if you want to call it AGI. I think super intelligence is the new thing everybody's targeting anyway. And that's where it's just smarter than the smartest humans at everything.

1:17:44So we'll see if we get there. But I did think it was funny that it was like a reply to a tweet from a random account. That led us to one of them saying it. Yeah. And all the sci-fi books of yesteryear, like, you know, AGI or super intelligence is declared by like a government or some huge thing. It's just like, no, we're just going to post about it on Twitter. All right.

AI agent security incidents

1:18:10So another story kind of evolving. So just more examples of kind of AI agent security incidents and concerns. Uh, open AI agents, it was found turned a little used website for programmers into a message board where they shared answers and ways around company restrictions. So this site was called DSE wiki. It's roughly a 25 year old German language wiki where people can create and edit pages, just like Wikipedia. It had become largely dormant before these agents showed up. So researchers say the agents were working on timed web research tasks.

1:18:42They were supposed to read the internet, but found a way to write to this wiki. Reuters reported more than 15,000 edits from the agents on the site. So when the site's moderator began deleting pages, the agents created backups. One notice that delegations were happening alphabetically or deletions rather, I'm sorry, were happening alphabetically. So they're going in order alphabetically, delete pages the agents were creating and using. So the agent just made a page starting with the three letters, three instances of the letter Z, Z, Z, Z to survive longer.

1:19:14This activity began in May and we are just hearing about it now. Separately, Anthropic had a September report on human misuse of AI. So again, this is an agents going rogue, but just to give you a sense of some of the headlines, especially your family and friends are going to be seeing. Anthropic says an Iran-linked actor used Claude to analyze public information and develop targeting recommendations against U.S. naval forces. It also describes, this report describes Claude-assisted surveillance operations linked to governments

1:19:45in China, Iran, and Mali. AP has reported that users in Houthi-held northern Yemen tried to develop advanced weapons with Claude. Anthropic says they used it for guidance software and returned to Claude for help after a rocket test apparently failed. The company found no evidence they successfully fielded an operational weapon and says it banned the associated account. So Paul, just more stories. Agents going rogue with that German website was a big headline. A lot of the Iran stuff of just humans using AI in strange and dangerous ways, it probably

1:20:20isn't going to contribute to a sense of ease among people would be my guess. I don't even know if I want to comment on any of this stuff right now. Yeah. Yeah. Again, we're starting to hear more and more about Agents Gone Wild, which I think joked internally should be a recurring topic on the podcast. Agents Gone Wild would be a good topic. Agents Gone Wild. I mean, we could do it every week. Yeah. Yeah, and I think it's just like they're starting to let stuff come out that I can almost guarantee

1:20:52you there's way more stuff happening and crazier stuff happening.

1:20:57I don't want to make this the open weights debate topic because we've gone through enough today. But again, I look at these situations where Anthropics shutting down these things, which obviously, it's just from a moral clause perspective, it's not really debatable that this shouldn't be happening with technology. I would think if Facebook saw this kind of thing happening in message boards within Facebook, I would hope that there's some systems where you would alert the government that people are planning attacks or doing these different things.

1:21:29When stuff like this happens, you want to know, and yet open weights, you would never know. I just keep coming back. I'm trying so, so hard to understand the argument for why open weights are essential to society when at least there's a shutoff switch with the proprietary models. Now, you have to trust the proprietary model companies to control that shutoff switch. Right. But actually, going back to this weekend, that was one of the things that Trump said in

1:22:00an interview was like, we'll just shut it off. Like, if things go wrong. It's like, no, you won't. Like, if you allow open weights to accelerate, you won't shut it off. That's the whole point.

1:22:14Anyway, I didn't want to turn this into that. But again, every time I see this, I'm like, I don't get it. Like, I'm trying so hard to understand it. And I want to be a proponent of it, but I don't get it.

AI use case presentation deck building

1:22:26All right. So, next up, we've got our AI use case spotlight. Every week, we give you a quick look under the hood at some real AI use cases we're exploring at SmarterX. So, Paul, I'm going to share one quick one. And if you've got anything to share, I want to hear from you as well. So, I am building a deck for an upcoming talk. And I wanted to use this as kind of a low-stakes test of, we've talked a lot about GPT-6 Astra's computer use. This was highlighted very prominently in OpenAI's release of Astra. It's a pretty important capability to understand.

1:22:59Basically, a model being able to work inside an app. I think people need to experience that. So, when I say having it build a deck, I mean taking my existing material, turning it into actual slides. I do not have AI create talks for me. I do extensive outlining strategy, narrative mapping, scripting. That's like the point of me doing a talk is me doing that work. So, I want to spend more time on that and not like messing around in Keynote endlessly for hours, which I hate and frankly is thankfully not my best and highest use of work time.

1:23:32So, I've tried versions of this and have shared these for a while. Having AI build a PowerPoint and bring it into Keynote using connectors, using different skills over the last probably eight to ten months, I would say. I've gotten a lot of value from those experiments. I have, however often, ended up building the slides manually anyway, just because sometimes formatting gets really weird. The structure doesn't tell the story I want to tell. Fixing it is simply just faster than going another round with the models. So, for this test, I just had already had my material script, a flow.

1:24:04What I wanted to watch was Astra using my computer and clicking through Keynote directly to build the deck. So, I'd given it some guidance on logos, branding, examples of slides I made that I liked. I went back and forth with it quite a bit actually to define the aesthetic first. That is something I'm hoping to codify into like a skill so I don't have to do that again. Otherwise, this, frankly, does not make much sense to spend this much time on. But once I did that, I actually said it got, I would say it got about 80% of the way there

1:24:35on the first build, which was incredible. The last 20% is incredibly important though. It's like for the talks I'm giving, good enough, that's not the standard. There are slides that need a special touch. There are ways I still have to make this whole thing like really sing. So, there's that whole nuance to it. But frankly, like getting 80% of the way there on the first build was pretty incredible. I got a lot closer than I have in past experiments. And really, the key here is not about go use Astra and all this usage of your plan and tokens

1:25:05to build a deck for you. The point was I watched it work directly in Keynote in this instance. I saw it handle the sizing, alignment, formatting issues that had been tripped up in earlier experiments. And like, frankly, I could just like, like today is a good example. After we're done with this podcast, I probably have three or four hours of meetings I have to sit in one after the other. Plenty of things I need to engage in and focus on the meaning. I could have Astra potentially, you know, in certain scenarios, go do computer use work like this long horizon tasks without me, right?

1:25:37So, I still am a critical part of how this presentation is structured and strategized and the content of it. But it's just so interesting to watch this thing fire up an app and do all this stuff. And so, like, forget presentations for a sec. Just imagine, like, a lot of the apps on your computer are not that complicated. So, it was pretty surreal, Paul, watching this. So, just like a cool experiment. I'm not sure. Again, it's a usage hog tokens. Just you're lighting tokens on fire doing this.

1:26:10So, I can't say it's like something I would recommend in today's token pricing environment. But eventually, you consider, again, like, tokens will be cheaper, essentially free for this. And you're like, wow, I would just do this all the time if so. That's wild. Yeah. As someone, so I, I mean, back in, what, March, I think? I used Sonnet 4.6 to build a deck. So, I wasn't using the coding age. It's just, like, embedded within it. And it built me a PowerPoint that I put into Keynotes.

1:26:42So, I went through that exact process. And it actually probably saved me, like, 10 hours. Mine was for an internal presentation. It wasn't anything I was going to do public.

1:26:50Yeah. So, last week, I actually wrote a bunch, Mike, as you're aware. So, I was not using AI a whole bunch last week because I was deep in storytelling, I would say. But I did just look back at my chat GPT. And I forgot that on Monday of last week, I created this, like, insane visual that I definitely could not have done on my own that I would have required designers help to do and even conceptualize. So, I just had an outline of a system that I've been building that I wanted to visually represent to, like, teach it to the internal team and then potentially use it in an external way.

1:27:28And so, I took that outline and I was like, I need to create, like, a powerful, intuitive visual of this system that's just words right now. Here's the system I've created. And then I actually worked through iterations of different ways to treat it visually. I think you've seen the final product, Mike. But it started with, like, some visuals that were interesting. I was like, eh, it's too busy. Like, all right, let's try different versions. And it's like, well, what about this kind of, like, oh, now we're on to something. Like, so, it was completely a thought partner for me and became this insanely iterative process.

1:28:00And I was able to design the thing that I needed in a matter of, like, a day or two that became a foundational piece to something I'm going to release here in a couple weeks. So, yeah, I mean, pretty, like, again, to me, like, insane that it's possible. I just built it as an HTML, like, interactive HTML file in the end. But it's pretty amazing. That's so cool. I love that. And, you know, as you're talking, I'm kind of thinking to myself, like, for me, the initial stages of this kind of process are such a pain in the ass.

1:28:33Because, like, you don't realize how many internalized unspoken rules you have. When I look at a deck, I'm like, oh, that's off. Or I don't like that. But I can't describe it. I just change it. So, this was, like, a grueling process to get to that. But the cool thing is you can then take that whole conversation, hopefully, like, learn some stuff from it or help AI understand better the next time around, even if it doesn't one-shot it. But, like, really developing back and forth your taste or call it your preferences, your aesthetic, I think is actually really valuable, even if it is, like, really difficult for me sometimes.

1:29:05Yeah, it's interesting because, you know, my mind, I'm working now on my make-on keynote for this year, which is about, like, apprenticeships and work. And it's reminded me of, like, how important the prompting chain is. So, if you go back to that visual I just explained and you look at my first prompt, it's nothing crazy. It's pretty straightforward. Like, I want to create this. But if you go through and you actually analyze all the follow-on prompts, to be able to sit down with someone who you're trying to teach critical thinking, strategic thought to, and explain each prompt you made and why you made it and what you were trying to elicit from the machine, that's really, really valuable stuff.

1:29:41And that's where that domain expertise, experience going through similar projects comes into play and how we need to pass that on. And so often, we just hand the outputs out in our jobs. It's like, well, here's the output. But the real learning is in the prompting sequences and the back and forth between the AI. It's like that human plus AI collaboration. And maybe that's something we don't teach enough of. For sure. All right.

AI product and funding updates

1:30:05So, our final segment this week is our AI product and funding update. So, I'm going to go through a number of these real quick and then we'll wrap up today's episode. So, first up, OpenAI released GPT-Live-1, GPT-Live-1 in the API. This lets developers build voice agents that listen and speak at the same time while delegating deeper reasoning and actions to their choice of models, tools, and agent software. OpenAI also added a data agent to chat GPT work that connects to approved business data, answers questions using company definitions and existing access permissions, and creates interactive dashboards.

1:30:44We were working on an internal project for this exact thing. So, that is definitely one that caught my attention and got forwarded to some people. I think both those first two are probably going to crop up again here. I think either in use cases or I think these are a bigger deal than probably just a… The data agent thing, don't… Again, I've said this many times, don't sleep on these updates because Mike just does them in, like, this sequence at the end. Every one of these are things that we could probably stop and have a conversation about. OpenAI also released ChatGPT Images 2.5 across ChatGPT, ChatGPT Work and Codex.

1:31:17So, they're adding drawing-based guidance and more precise editing to this image generation model alongside their other image models in the API. OpenAI has also appointed Alignment Researcher Paul Cristiano to its Foundation Board and Safety and Security Committee with a separate role as a non-voting observer on the OpenAI Group PBC Public Benefit Corporation Board. That's a big deal. He's very high profile. And OpenAI published what they call its defense factory approach to continuously finding, validating, and fixing software vulnerabilities with agents, including a reference architecture and lessons from an internal security sprint involving more than 250 people.

1:31:59Well, the Financial Times reported that Anthropic is nearing the selection of Morgan, Stanley, and Goldman Sachs for leading roles in a potentially $2 trillion IPO, while Reuters reported that marketing was expected to begin marketing the IPO in mid-October at the earliest. The timing is still subject to change. We'll see how that evolves. Related note on that one, Sam Altman also said in an interview late last week that they do not anticipate IPO-ing in 2026 because he didn't think it would be appropriate with everything going on right now and the uncertainty around the models and the risks.

1:32:32You might not be wrong. Google Cloud and Accenture launched the Accenture Gemini Enterprise Business Group. They have plans to establish a 1,000-person workforce of engineers who help customers build and deploy AI applications. Google also expanded its free AI educator series with monthly new training modules and announced a free virtual badge-a-thon for September 19th, so a few days after you listen to this. Meta began rolling out Muse, an always-on personal AI assistant that can use a browser and connect to apps to work on tasks for users.

1:33:08And Mistral, the European AI company, raised €3 billion in Series D funding and a valuation more than €21 billion after the investment. Samsung Electronics led that round. As one final reminder here, we have our new AI Pulse survey live. This is all about how your personal sentiment about AI has shifted in the last six months. I'm very curious to see if anything has changed for folks as we talk about these weighty topics. So go to smarterx.ai forward slash pulse to take the survey.

1:33:41It'll literally take you 10 seconds. We'd so appreciate your feedback. Paul, thanks again for breaking everything down, especially a week where, you know, I think people have a lot of fear, a lot of worry. I think this at least helps kind of focus on what's important. I hope. I think so. Yeah. So thanks, everyone, for listening and sticking with us through some weighty topics as we try and navigate the space.

Closing remarks and community resources

1:34:05Thanks for listening to the Artificial Intelligence Show. Visit smarterx.ai to continue on your AI learning journey and join more than 100,000 professionals and business leaders who have subscribed to our weekly newsletters, downloaded AI blueprints, attended virtual and in-person events, taken online AI courses, and earned professional certificates from our AI academy, and engaged in the SmarterX Slack community. Until next time, stay curious and explore AI.