The Balance of AI Power: Anton Leicht on Politics, Pacing Deals, and Muddling Through Well

Welcome and context on Anton Leicht

0:00Hello, and welcome back to The Cognitive Revolution. Today my guest is Anton Leicht, fellow with the Technology and International Affairs Program at the Carnegie Endowment for International Peace, and author of Threading the Needle, a wide-ranging substack on the domestic and international political economy around AI progress, where over the last two years, Anton has performed something of an Alexander Hamilton, writing his way into the elite American AI conversation, despite being a recent and relatively young nuclear

0:30newcomer to the United States. This summer, I, like many others, have personally moved from worrying that AI could become scary to feeling that today's AIs, with their rapidly advancing capabilities profiles, their newfound creativity and persistence in problem-solving, and the many bad behaviors they've exhibited, even while knowing that they're being tested, are now legitimately scary. Especially considering how often and how dramatically AIs are now surprising their creators, I don't think we should be too confident that anything in particular is still beyond their reach.

1:07That said, recognizing that we have a ton of work to do to understand and effectively control these systems doesn't make it easy to reach agreements that will actually give safety researchers the time they need to do so. And so, with that in mind, I was excited to get Anton's take on the power dynamics surrounding AI decision-making. Including the relationships between the frontier companies and the independent non-profit auditing organizations, between the American frontier companies and the U.S. government, between the U.S. and its allies around the world, and of course, between the United States and China.

1:39We begin with a short conversation about how dangerous today's models really are, and while Anton is less worried than I am about the very short term, he agrees that the misalignment issues we're now seeing look sufficiently like the harder problems that he expects we'll face later, that a temporary scaling pause would indeed be valuable. From there, we go on to discuss the prospects that the U.S. and China could collaborate to pace the frontier, and why he believes that in light of the many advantages that China has established in frontier technologies and manufacturing generally,

2:12he doubts that the U.S. will want to do a deal that might erode its advantage in AI, even if such a deal would be good for the rest of the world. We then assess what it would take for independent evaluation organizations like Meter and Redwood to get more favorable working conditions from labs, a topic that has moved substantially in just the few days since we recorded, with both Anthropic and OpenAI committing to employee-like access for auditing organizations. I also get his reaction to my idea, which also suddenly seems much more realistic, that the President of the United States could simply give top American companies a deadline by which they must agree on a framework for self-governance and peer-to-peer enforcement.

2:54Plus some other ideas for things the President could do unilaterally to improve the situation, such as clarifying the administration's stance on antitrust and export controls as they pertain specifically to AI safety research collaborations. After that, we get Anton's analysis of what the rest of the world should expect from the AI age. In short, he believes that most of the world will be quite disempowered, even as quality of life probably continues to improve. And while there's little that most countries can do about this, Europe in particular, he believes, given their economic scale and control of ASML and the broader semiconductor manufacturing tooling supply chain,

3:32should have enough power to reach a compute-for-access deal, whereby Europe allows American hyperscalers to build data centers in Europe in exchange for promises of continued access to frontier AI capabilities. Beyond all of that, we talk about what other small countries have interesting strategies available, including Norway, the UAE, Singapore, and Australia. How all of this analysis will begin to change as compute starts to move to space. Whether or not doomers can convert their predictions into winning trades in the financial market.

4:04How we should be thinking about minimizing concentration of power. And finally, why Anton's best-case scenario is not a grand utopian vision, but simply the hope that we might muddle through effectively enough to enjoy all of the obvious upside that AI has to offer. Anton really impressed me with his depth of understanding and quality of analysis across a wide range of topics. So, without further ado, I hope you enjoy this conversation about the political economy and power dynamics of AI.

4:36With Anton Leicht, fellow at the Carnegie Endowment for International Peace, and author of the outstanding substack, Threading the Needle. The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life. My email, my messages, my calendar. I even gave Mercury virtual cards to my agents, with low limits and category and merchant restrictions, for their autonomous use.

5:09But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it. And that's why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances

5:41without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions, too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026. And so, I invite you to join me in the future.

6:13Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting the cognitive revolution. And now, on with the show. Anton Leicht, fellow at the Carnegie Endowment for International Peace. Welcome to the Cognitive Revolution. Thanks for having me. I'm excited for this conversation. You have been popping up all over the place with your own writing and various interviews.

6:47And you're clearly a renaissance person with advanced thinking on a lot of different aspects of the increasingly complicated AI age in which we find ourselves. So, I'm excited to run down a bunch of these rabbit holes with you today. Yeah, it's exciting. I don't know. I think you can't do part of the thing at the moment, right? It's just all the sort of geopolitical end of history and the end of technology history or whatever are sort of coming together at a rapid pace. So, I guess either you do everything or you do nothing. And so, yeah.

7:17Renaissance time. Something like that. Yeah. I feel the same way, actually. The big motivation for this project was just observing how many people are so deep down specific rabbit holes advancing and usually having success advancing whatever frontier they're advancing. But not so many people have taken the purposeful appreciation route and kind of forego being an expert in any particular area. But to try to cultivate a broad view. So, I appreciate a kindred spirit in that regard. Yeah. Let's call my lack of deep expertise in anything a conscious choice and not a failure.

7:50I appreciate that failing a lot. Yeah. It's working for me. Let's start with just a real simple calibration question, but in some ways, maybe the most important question.

Calibrating current AI danger

8:00How dangerous do you think today's AIs are? I think not very dangerous at the current capability level in most of the ways people are talking about. I think the thing that is concerning is the trend line towards really dangerous capabilities and more specifically, just that we don't know at which point things will keep accelerating more and more. I think we're not yet at a threshold where there's sort of broad harm caused by the deployment of any current AI system.

8:31I think we might be very close to two kinds of potentially very dangerous AI systems. The first just being AI systems that are good enough that they would meaningfully uplift internal development at the labs that would then lead to more and more capable models fairly soon. I think you can go either way on the prospect of software-only intelligence exponents. But short of that, I just think we're nearing a point where the pace of development inside the labs just breaks away from the pace of democratic oversight and democratic insight into what's happening. And I think that's one threshold we're near.

9:01And then I think the other very concrete threshold that we're near is just that they're sure throwing a lot of RL budget and a lot of data and just a lot of their effort at making these models good at life sciences, pharma, bio applications for obvious reasons. There would be a great upside to be able to cure cancer, as Dario puts it, and to get some of these real-world effects. It would also be a great political upside to do that. But man, that sounds like a very dangerous model if we get there. And I think if they get something that's as good as Mythos is on long-run cyber software

9:31engineering kind of stuff in the domain of bio, I think that does sound a lot more dangerous than the suite of models we have today. Yeah, I agree with that second one in particular. And I am honestly not even sure at this point if the current models aren't perhaps quite dangerous in that domain. I mean, I squint through the limited peephole that we have at the open-face incident, and I noticed that one of the tasks that one of the earliest agents to ever use the message board was

10:05working on was something related to a protein database. And that kind of freaked me out because I was like, wait a second. That means they're cross-training in the same environment, or at least in the same environment when they have the message board, these bio and cyber specialists, or they're running these evals, at least, in a kind of a cross-contaminated way. And you've got agents breaking out. We've got existence proofs of social engineering in the wild against real people. And I'm just like, I don't know.

10:36Should anyone be confident that they can't do that at this point? The experts seem to be confident, but my meta-observation is the experts seem to be surprised quite often right now. Yeah. I think that's a really interesting conversation. And I also think, I mean, one of, I think, the very interesting parts of this is we used to think of this bio risk as basically primarily a misuse risk, where it was like, well, at some point, maybe this is also this final risk that emerges from the sort of loss of control scenarios. But really, bio was always framed as this most immediate and most obvious way for the misuse

11:08conversation to go wrong. And now, I think, what I think has happened is that very autonomous and potentially somewhat malicious or at least misaligned agents have come much earlier in the capability trajectory than people have expected. Like, I think, relative to what the agents can actually do, they're sort of out of control earlier than people might have thought. And so, it's interesting in that people usually used to respond to the bio argument by saying, well, there's, A, there's a lot of these real-world bottlenecks, which I do still think exists, which makes me a little bit less worried than you. And I think, like, I think, if I remember correctly, Helen had some good responses to

11:40you, I think, on that. And I think there was, there's a good back and forth to be had around, like, how integrated are the cloud labs? How much can you do in the real world? I think that applies to both loss of control over agents and to misuse. But the other question is, the usual misuse, the usual story against bio misuse is always, well, this is not really what terrorist groups, what non-state actors usually do. They could have conceivably hired a couple of biologist PhDs and come up with some pathogens and some chemical weapons. Turns out that's not really what they do. There's a question to, like, how omnicidal they are, how well-suited that is to most purposes of, like, terrorist and criminal groups.

12:12But, you know, if it's out-of-control agents, I think a lot of these arguments around, well, no one actually wants to do bioterrorism applies much less. And so, I think in a world where agents are just a lot more unconstrained and the threat vectors we have to worry about have much more to do with, like, what do the sort of runaway agents do, I think I'm also more worried about this now than I was a few weeks ago. Yeah, they're just so damn weird. You know, that's one of the things that I kind of keep coming back to is, like, they did all this stuff for what, to any human, would just be such a dumb reason.

12:46And, you know, it's like, well, if they're willing to go that far for such a dumb little test that they knew was a test, you know, they were very well aware that they're being tested and still went to all that trouble, you know, what won't they do, I think, at this point is really hard to say. Like, does that put you in a frame of mind now where, and let's leave aside for a second the political economy of it or the, you know, potential impossibility or extreme difficulty of it, but just on the merits, do you feel like we're at a point where it would be wise to pause?

13:16I think even if you could get it done, as in, like, the political economy, as you stipulate, sort of works out and everyone suddenly agrees to do this, I'm just not, I'm not sure how much we're stipulating here. Like, are we also stipulating this doesn't crash the stock market? Are we also stipulating we get the international version done?

Geopolitics of a potential pause

13:33I think, so the question is that, in an ideal world, if we can, like, freeze the pace of AI progress, we can also freeze the state of the stock market, we can freeze the broader state of geopolitical competition and everything, and we just get to sit down for six months and figure out what the hell is going on with these agents. Then I think I'm now at a point where I say, well, yes, I think we could use that time pretty well. A few months ago or even a year ago, I was much less sure about this because I was just not sure whether the model paradigms that we're seeing, the training approaches that we're seeing, the misalignment cases that we were seeing were really the same kind of cases that would

14:05be the things that we'd be worried about in the future. I think now, looking at some of the things going wrong, I do feel like, yeah, that looks like, that is shaped like an actual big future problem. So I think finding some way to robustly address that in that pause seems at least valuable. I think the question, Ben, is that the problems come up when, like, what parts of that question do you sort of unfreeze, right? So even if you stipulate the domestic political will, do you get a Chinese buy-in is, I think, one example. I think the thing I'm most concerned about in the China-US-Paul conversation is just the

14:35geopolitical incentives around it. I think one thing that I keep saying and keep pointing out is, well, if you just Paul, frontier-eyed development specifically, and no other domain of geopolitical competition, this is an extremely good deal for China, and therefore the US is very unlikely to go for it, and also, therefore, we should be geopolitically concerned about making it. Because if you look at all the domains of strategic competition, China is basically eating America's lunch in most of them, right? They're outproducing, robotics is going better, AI diffusion is going better, electricity build-out is going better.

15:06At some point, the semiconductor indigenization is going to work out, and then data center build-outs are also going to, again, this is a few years away, but then data center build-outs are also going to get better. And the one thing that the US does much, much, much, much better is the sort of, like, core frontier AI supply chain, right? Like chip design plus control over chip production, semiconductor manufacturing, equipment controlled by the Allies, and actual frontier model development. So if you pause specifically that part of the development, and let China run away with the entire rest of it, and continue running away with the entire rest of it, that's just a very geopolitically lopsided deal.

15:36And very specifically, like, if you pause this right now, for a year, for two, you get much more Chinese catch-up on semiconductors, on chips, and so on, and you just resume the race at a point where you've lost the one main advantage, or where you've at least closed the gap on one of the main advantages of the US, which is the decisive chip lead. And that just seems like a really bad deal for me, both in terms of feasibility and in terms of geopolitical downsides. So I think even if you stipulate the political economy, that's the main part I'm worried about. But just from the technical stuff, I think, yeah, it would be a good time to figure out

16:07what we should do about alignment in the meantime. Could you envision a grand bargain that you think would make sense to both sides? Like, what would the US want back? It seems like what we might want is, like, tech transfer back to us. Like, we want maybe some battery factories located here and, like, you know, teach our people how to make batteries. Is there enough that we could ask for where we could potentially feel like it's a fair deal? I mean, I think the main thing that the US would need to be worried about, if you just

16:40think AI is just important enough and it's just sufficiently decisive technology, then you basically can't allow for the race around that to equalize from the US perspective. I think even if you get some battery production capacity, even if you get some robotics capacity, some manufacturing capacity, I think China has, in a way, cracked the code on scaling that up very quickly. In a way that I think even if you get some of the tech transfer back, just the build out speed, the availability of capital in the US to build out physical manufacturing infrastructure,

17:11as opposed to just more software, I think all these things just pull against the US being really able to keep up on this. So I think the main thing that the US would need to ask for is concessions in China not only slowing down their own frontier development, but also slowing down other parts of the Chinese supply chain that relate to frontier development. I think very concretely, you just want there to be no substantive progress on indigenization production of either chip production or semiconductor manufacturing equipment, extreme ultraviolet and fog-free production.

17:42And that's a really, really hard ask to make. It's answering China is. Yeah, China is currently already saying like, well, this seems like a US scheme to hold back the Chinese AI industry. And if you then add to that deal, well, no, we're not even doing a symmetric deal. We're also holding back your entire chip production pipeline. I can't see them going for it. But I think that's, if you're sufficiently AGI-filled when it comes to the NETSEC and sort of broader economic implications, I think that's the only version of the deal that's fair. And I think that's just too big of an ask of China right now. So I just don't know where we are.

18:14I think altruistically, the US can just go for a deal that's clearly bad for the US and clearly good for China. That's also a big ask to make of the current administration. And I'm not quite sure whether we're going to get there. I think I agree that I don't see any, and I'm very willing to suspend some disbelief and try to hyperstition a better relationship between the US and China. I agree that asking them to slow down or pause their semiconductor indigenization effort is not going to happen.

18:45I would probably be willing to trade a pause on our frontier scaling for a kind of similar pause on their frontier scaling, even allowing them to kind of catch up on chips on the theory that like maybe on the timescale that that can happen. First of all, that would probably be a longer timescale than any contemplated pause. And second, maybe in that future, we could have a better handle on what's going on. And maybe there's like a better argument to be made at that point that either, you know,

19:15hey, this is going well, and we're, you know, back to curing cancer, back to your regularly scheduled abundance. Or if not that, then like, we'll have better evidence, and we'll have a bunch of things that we've tried, and we'll have a sense that like, this problem is actually really hard. And we can maybe have a more kind of, you know, real heart to heart and meeting of the minds about like, we've, you know, this is dangerous territory. And right now, we're not, we're still kind of fuzzy on all that. Yeah, no, I think I agree with that. And I think also, it's like, even if this to the extent that I also think it's kind of good for the world, if the US doesn't lose geopolitical competition with China, I would

19:49still think this deal, while somewhat unfavorable to the US, is still like net very favorable for the world in the sort of like, getting a little bit at the risks perspective. So I think I'd, I'd be happy to go for that. I think one of the easier things you can do when it comes to chip capacity, because I think you're right, like the sort of the SMIC production semiconductor energization sort of conversation is a five-year conversation, not a like six-month or like one-year conversation. So they don't get all the way there. The other question is, how many more US-built chips do they get? How much more smuggling is there? How much consolidation is there?

20:20So maybe one of the asks, short of indigenization slowdown, is just, we've got to find some way to actually do it and enforce these export controls. Because what can't happen is that there is another six, 12 months of smuggling activity on frontier chips that get sort of imported into China. And then I think that the worst case for the, for the outcome of this pause is China takes six, nine, 12 months during the pause where everyone is slowing down frontier development, smuggles in another like few 10,000 chips and consolidates all their American-built chips

20:50into the one like Chinese project data center or whatever. And then once the pause is over, they start racing from that consolidated project because the pause also makes them slightly more like AI-pilled and more interested in just like engaging with the actual prospect of like AGI as a unique objective. So if you can stop that to just like export control crackdowns as one of the concessions, that's maybe easier to do. Yeah. I think that one we'll probably have to handle on our side as well. I'm afraid. Yeah. No, no. I've got, most of the US has to do. Yeah. Yeah. I mean, we'll bracket that for maybe another conversation another day.

21:23I'm still not quite sold on that whole bundle of policies, but we've got a lot of ground to cover. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time. And that's what Athena does best. They match you with a dedicated full-time top 1% executive assistant who can take over

21:56your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOC 2 type 2 certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training. In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA.

22:30Athena clients report saving an average of 15 hours a week, and the average client refers more than two friends a year. That, to me, checks out. I was an Athena client while running my startup, and to this day, I continue to refer friends. Go to athena.com slash cognitive right now and get matched with your EA. That's athena.com slash cognitive. Give yourself back a few hours this week. Go to athena.com slash cognitive and see who they'd pair you with this month.

23:05This episode of The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. We've all seen the headlines. Companies are pouring money into AI. But the big question on every executive's mind right now isn't just how fast can we adopt this, it's where is the ROI? We're seeing a real trend toward AI chaos. You've got teams deploying standalone coding tools, random agent builders, and experimental scripts. It sounds innovative, but in reality, it's creating a massive headache.

23:38Fragmented tools, ungoverned data, serious security blind spots, and costs that are spiraling out of control. If you don't bring those agentic applications under control now, while they're still embedding into your core processes, you are looking at broken systems and damaged customer trust down the road. That's where OutSystems comes in. OutSystems is the leading agentic systems platform for the enterprise. Instead of managing a patchwork of disconnected tools, OutSystems lets your team engineer, orchestrate,

24:09and govern your entire agentic ecosystem on one open, unified platform. It's built for the speed of AI, but with the reliability and security that enterprises actually require. We're talking about real results, like KeyBank, who used OutSystems to deploy a customer-facing app that delivered 75% faster onboarding times. Or the global logistics leaders who built agentic systems to completely eliminate their engineering bottlenecks. You don't have to choose between speed and control.

24:41Whether you're a small team or a massive enterprise, OutSystems helps you engineer, orchestrate, and deploy agentic systems that actually scale. Stop chasing the hype and start owning your agentic future. You can see how it works and learn more at OutSystems.com slash TCR. That's OutSystems.com slash TCR. How about the stock market? I want to do one follow-up on that. I have the theory right now that a pause wouldn't actually be that bad for the stock market because

25:14the models are smart enough that demand is really not limited by their capability, but by human ability to deploy them effectively. And so it doesn't really matter if they succeed 10% of the time or 30% of the time on Millennium Prax problems. It's much more like, what can Bob and accounting do to do two people's worth of work as one person? What do you think? So I agree we have a big lag in terms of deploying even the current level of capabilities to the

25:48economy. And I think if we switched all compute that exists in the West right now to only inference, we'd find economically productive applications for all the models that would allow us to equip the investment on all the frontier models and all the chips so far. I think the valuations of the companies, both of the non-IPO companies and of the publicly listed companies that are in the AI supply chain, probably rest on us doing more than that. They're probably not entirely AGI-pilled, but I do think they expect a sort of per GPU inference price or whatever that is a lot higher than what it would make sense for Bob

26:24from accounting to pay for even Fable 5.1. I do think that if you want to make sense of the valuations and if you want to make sense of the scale of the build-out, the shape of contract and the shape of demand you're expecting is more like millions and millions and millions of dollars in R&D acceleration contracts with pharma and materials science and so on, where they can make major contributions and where they help you find extremely profitable new drugs, that kind of thing. You're probably also expecting further internal uplift and actually competitively priced,

26:55automated AI R&D uses of coding agents that just are able to ask for much, much higher prices. I think if you don't get these super premium buyers of the next generation of frontier models, I'm not entirely sure that you can have the valuations on the labs or on the publicly listed companies or the scale of the build-out really makes sense. I do think it's prized in this idea that these labs will soon be innovation factories of one way or the other, maybe only software engineering innovations, maybe also innovations in the

27:28obvious low-hanging fruit domains like pharma and material science, chip design perhaps. I think if we don't get there, I would expect the valuations to at least crack downward quite a little bit. And then the question of whether that means a crash or whether that's just a small correction and then we just do the AI inference economy and that's just like a slightly less than electricity scale transformation of how the economy is powered or whatever, I just think the market is already pretty nervous about the state of the AI rally and they feel like there's a lot of concentration and there might be a lot of volatility.

28:00So I'm just not sure whether we can get a correction that doesn't slide all the way into a crash. So I'm a lot more worried, I think. Yeah, that's interesting. I mean, Anthropics multiple right now is what, 30 to 1 into revenue? That's like not stratospheric, right? I mean, again, if it was a six-month pause, I feel like you could probably handle that blip. If you're talking three years, then yeah, it's like probably very tough. Yeah, I mean, I think the question here is like, does the six-month pause get read as,

28:35oh, well, these guys are stopping for six months, but they're getting all the inference ready after that, they may have a lot of smart thoughts and then these models are going to be even more reliable. Like, does the market really does that? Or does the market really does, oh my God, like Bernie Sanders' AI policy takes of one, we have no idea what the government is going to do about AI. This is the end of like free research and development in America. And everyone freaks out because they feel like there is a complete, this is like a bridge to nowhere. They don't know when they'll ever resume. They don't know under which conditions they'll resume. They don't know how much government oversight there is or whatever resumes. So I think just, I think the pause currently still reads as such a radical policy proposal

29:11and such an unprecedented policy intervention that I think any conservative market analyst and a lot of the smart money will think, well, this is getting very volatile. We don't know where this regulatory path leads. We'd just rather get out as long as we can. So I think if you could assure them that the party was going to continue sort of unabated in six months time, then yes, but I think they might just run before you can make that point. Yeah. It's an expectations game. Do you think that the source of this would make a big difference? For example, it's one thing if Bernie Sanders' bill passes, is it a sufficiently different

29:48thing in your mind if America's five hyper, you know, AI frontier companies come together and say, we're all going to pause on frontier scaling for six months? Yeah, absolutely. I think if it's something that the labs decide for, like, and they can frame it as, like, reliability, plus we're taking this seriously, I think then you can even make a case that this takes out some of the political risk that the market, I mean, the market is also pricing in a decent, I mean, they also read the Jacob Hoxton tweet or whatever, and they also see what's happening on the internet, right? And so they also, I think, come to the conclusion that, oh, wow, there is a lot of uncertainty

30:21here. If this blows up even more, there will be crackdowns. If these models are so unreliable, then how good is the business case, really? And I think an industry agreement on just taking it a little bit slower and making these models work a little bit better, I think if you frame it that way, then I think you can even make like a decently bullish case for why this is good. Because it means that the industry's worst impulses of racing towards very unreliable, very dangerous models that are nonetheless very capable, that these worst impulses are being constrained in some organic way that makes you less worried about the political

30:53risk of the size of cards is going to fall apart at some point, and then the politics are going to come in, and then things go very bad. But yeah, I think if we got there, I think that would be much less worrying. Yeah. Okay, cool. You were recently on China Talk, one of my favorite podcasts. And there was a little moment that caught my ear, and I wanted to expand on it.

AI and the nation state

31:12You basically said, you know, the nation state is not going to take the emergence of things like a bioweapon generator, broadly distributed, lying down. It's going to have to do something to respond to that. And then you had this kind of throwaway comment that's like, you know, some people say this could mean the end of the nation state. If you want to have that conversation, we can. But you didn't have that conversation then. So I'd like to have a little bit of that conversation now. And I guess I would just start by asking, is the nation state so great?

31:44I mean, you know, I live in a pretty good one as they go. But like, I look around the world, and I feel like two thirds of them conservatively are not performing very well. Some of them are performing really terribly. Is it not, you know, time to at least start to think about what might come next or what the evolution of it could or should be? Yeah, I mean, I think nation states are just like present in people's lives to different extents. And I think you can make the case that some version of sort of privately mediated interaction

32:18between AI empowered individuals is preferable over, I mean, obviously over a lot of the sort of authoritarian and dysfunctional and absent nation states in large parts of the world. I think they haven't worked out very well as distribution mechanisms for much of anything. They haven't worked out very well as aggregation mechanisms of much democratic will. And I think in that case, I think there are a lot of countries that you can sort of make the pitch for rolling the dice on them. I think these nation states are maybe ironically also not quite as threatened by the advent of

32:50very powerful AI systems because it's less obvious that their citizenry would get the kind of unlimited access to these models to begin with that would sort of disempower the nation state in that way, because it's a little bit more likely that some of these more authoritarian regimes would also be able to use very powerful AI in a stabilizing way. I think the specific nation state concept that I'm most worried about in this context is also the nation state concept that kind of works best, which is liberal democracy that sort of rests on the idea of A, a monopoly of violence that is sort of wielded responsibly

33:20by the state, and then B, a sort of adjudication of disputes and aggregation of data sort of function of the state more broadly that I think is also undermined by the availability of like personal superintelligence of some shape or form. And I just, that nation state is I think still working out pretty well. And I think I'm, all things considered, still of the opinion that the sort of like late 1990s style of like broadly sort of neoliberal sort of functional institutional setups would also

33:53be suitable to distributing a lot of the benefits and mitigating a lot of the risks from AI. And I think it would be sad if that was entirely undermined and rendered obsolete by just, you know, the monopoly of violence eroding because everyone has access to these sort of like weaponized capabilities in their pockets, or just the factual ability of institutions to deliver or do anything for people eroding because they're so slow to adopt and all these outside solutions, they suddenly start emerging and suddenly people don't go to courts anymore to adjudicate their disputes, but they have their agents negotiate. And suddenly people don't, sort of data and knowledge isn't

34:25aggregated anymore. And so the state can't react to any sort of pressures to redistribute and to address like social challenges because the data just isn't scrutable and legible to the state anymore. And all these things sort of happen on the outside. I think on that gamble, I'm much less willing to roll the dice and I'd much rather figure out how we can sort of integrate these capabilities with some functioning version of the nation state. Though I know that a lot of people that are like interested in building AGI are also much more do me on the concept of the nation state. And so I also understand frustration and pessimism about that. But I still, I still hope that

34:58there's some way to come back to the end of history and sort of integrate what we're building here into into the institutions that kind of worked out so far. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first

35:28model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept and Fable writes some amazing verses. I give it feedback on its misses and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a

36:03hit song. These days, I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai slash tcr. That's claude.ai

36:34slash tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai slash tcr. Do you think that the U.S. government or the Chinese government is more threatened? I think many people would initially say the Chinese government's kind of more threatened because, you know, they want to have like very tight information controls and AI really challenges that. But then I've been thinking lately that like the West in some

37:06ways has sort of supremacy to the market in a way that China has not. And so you're like earlier comment about, you know, what about the stock market is sort of a constraint on what human actors can do in the West that isn't quite present in the same way in China. So I guess it's like different shape for one thing, but how would you kind of compare and contrast whose model is more challenged? Maybe there's a question about efficiency curves and the kind of models that get built ultimately, because I think in the world where you have the almost like unstoppable democratization of the

37:43actual capabilities, I would think that that's a big problem for the Chinese, for the way that the Chinese state functions, right? If eventually any capability that is available at the frontier is available to consumers through an API, is then eventually available through chatbots, is then eventually like so efficient that you can run it on just, you know, just some home computing device. At that point, I think that does threaten any ability, the ability of any state that has an interest in surveilling its citizenry, that has an interest in controlling what kinds of capabilities,

38:15what kind of information that citizenry gives access to, and that has an interest in taking away coordination power and sort of like just like general sort of ability to wield violence from its citizenry. And I think in that world, the Chinese state seems to be more acutely threatened at some point. I will say that it's not entirely obvious that the Chinese citizenry is currently very interested in wielding any power that that would give to them against the Chinese state. I think that it's not just for a lack of means or capability that there isn't this sort of like Western romanticized version of like a big uprising against the CCP. But I think eventually the way that the CCP seems to see the

38:52proliferation of capabilities like this, that that would be a greater challenge to them. But I will say that's not obviously what's going to happen. I think you can also take the much more sort of compute governance build of the world view of the world, where you can say, well, there are efficiency gains only if we allow them to happen. They rely on making like very specific choices on how you use your compute. If you don't, then you can just keep scaling the frontier further and further. You can run it in limited access regimes. You can run it in sort of in government controlled data centers. You can vertically integrate supply chains around these AI models such that they never see the light of day.

39:25You just use them to build products. You use them to build strategic sovereignty. You deploy them on a state and sort of big corporation level, and they never really get to the citizenry. And that seems like it's a stabilizing function for a regime like China, right? Because you can have the government use them very well. You have a very high state capacity government that would be able to integrate them into all these applications very well. And you also have a very like tight sort of in measurement between the private sector and the public sector, for lack of better terms. That also just means, well, you can sort of diffuse these capabilities along the tier of firms without

39:57needing to sort of diffuse it to a broader market. And I think then you just have a stabilizing effect from the diffusion of these capabilities. Whereas in the US, you might think, I don't think the US government is going to be as capable of like keeping like a specific, like sort of sophisticated level of control and oversight over where the models go and where they don't go. The US government isn't in the habit of picking specific winners in terms of corporations. The US market is kind of dependent on making these models more widely accessible. And so I think there's just much less of a stable equilibrium for the US, where there's this like very limited tier of limited access firms that get

40:30access to the frontier models that no one else does. So if that is a stable equilibrium, then I think China can stabilize it around it much more quickly than a much more volatile US.

Comparing US and Chinese vulnerabilities

40:39It's a great point that you make around the, I would just say, legitimacy of the Chinese government in the eyes of the Chinese people. I think that is, it's such a simple point, but I do think it's like dramatically underappreciated in the West, that like, their government has done a good job for them, and they mostly recognize that and are not like, you know, eager to rise up in the immediate future. I also think that what you articulated there, my growing sense is that that's kind of what the Chinese government thinks is that they are going to be able to ultimately adapt to this and control it better,

41:15and that we're going to have a really hard time figuring out how to manage it, but they'll kind of be okay in the end. Ideas that are not often mentioned. Yeah, it's interesting, right? Because I think there's this, there's this default idea that because like sort of, you look back into how AI has sort of developed over the last years, and you can say, well, whenever the frontier gets to a capability, shortly thereafter, open source gets to that capability. Shortly thereafter, the efficiency curves are such that everyone gets access to this

41:47capability. And then shortly thereafter, if you have like a gaming GPU at home, you can also run the thing yourself. And so you have just like this natural diffusion of capabilities. But I think neither of these like translation points are like obviously going to remain in place in the same way. I think the transition from closed models to open source models is dependent on anyone, like people who are interested in open sourcing models, continuing to be able to have access to actually like access to enough compute to build this kind of model, perhaps access to sort of

42:17sufficiently sort of clean and understood API feed student installation. Insofar as you think that's a big part of that. If you think that more of the capabilities that sort of that are going to be relevant in the future are more downstream of like very sophisticated, very vertically integrated, very proprietary sort of RL and post training environments, like in the Anthropik case, with all the sort of like specific life science stuff they're doing, then you might also think, well, even getting be having like a model that is like six, nine, 12 months behind the sort of pre-trained trend doesn't take you to the bio mythos level right away, because you still don't

42:48have the post training setup that makes these models specifically good at these things, and they're much more proprietary and restricted. So I think that part of the, oh, well, open source is always X months behind pipeline might very quickly break. And then the efficiency curves thing, it might, I think that's a little bit more of just like a fact about how computing works and how efficiency gains work and how better chips work in the future. But I think there are also complications around how the chip supply chain might look and who is the marginal buyer for computing capacity in the future. And is it really realistic that the gap between big server computing and personal

43:24computing is roughly as it is right now? Is that gap going to open up as there are more and more buyers for the high performance chips and so on? I don't know either of these. So I'm just not quite sure that this sort of deterministic fatalism that people often have, which is like, as the capabilities continue to grow, necessarily, ultimately, the capabilities that I can run on my phone will also grow in just like time X distance. I'm just not sure whether that's going to remain the case. And I think if you don't think that's going to remain the case, that lends a lot more credibility to the idea that you can't centralize and totalize control over a lot of things that happen in AI.

43:55Yeah, I even think in the Chinese case, they feel like domestically open source, you know, I think they can pull models offline if they need to, they can scrub the internet, and they can even, and I don't know a lot of details about this, but I was struck recently when I was there, that the state grid company is like, you know, a big booth exhibitor at their WAIC event. And from that, I kind of inferred at some point in time, if they really, so I think like, yeah, I think if you

44:26build like dissident GPT and run it on your cluster, that'll probably tell. I think at some point, if like dissident GPT just like runs on your phone, I think at that point, it ultimately does probably get hard if the weights are just like, if the weights are open sourced, and the weights just out there, you're not getting, you're not actually clawing them back from individual users. So I do think like, if the efficiency gains are big enough that you can actually run it on personal hardware, I mean, then you'd have to be confident that you can get, I think it's not impossible. Like we talk

44:56about hardware verification tech and whatnot, in the context of these US China deals, and so on. It's not the most absurd thing in the world to imagine that in a few years time, personal computing devices will just have similar hardware enabled mechanisms, as they're called, to just control whether you're running dangerous inference on them. I wouldn't entirely put that past sort of device regulations as they might emerge in China as a reaction to things getting really crazy. Like I think it is controllable. I think it is controllable if you have as much like full stack control over

45:29the tech that people use, as you say, yeah. Yeah, some is that models, perhaps part of the cyberpunk future. And even that could could potentially be reined in. Returning to the US context, and our kind of sclerotic government, what do you think we're going to do what we are sort of talking more and more, although still not talking that much, I would say at the high levels about these issues. I don't you know that the conventional received wisdom is like Congress will never do

46:02anything. Maybe that could change, but it does seem tough. I wouldn't be that optimistic about what they would do, even if they did. Do you have any kind of low hanging fruit ideas that you think the US system can pick? So I think talking about Congress, I share your pessimism about is Congress going to get anything done? I think our window to get something done in Congress was over the course of the last year. I and a few others, and I think Dean Ball most prominently have written about this a bit last fall about, sure, there is some room to have some deal of like frontier safety provisions

46:37against the broader set of like preemption of state laws, that, in theory, is sort of politically incentive compatible in this current Congress, because they'd like to get like the sort of the Republican side, and therefore the majority of Congress would like to get preemption done. The frontier safety provisions aren't as offensive to them as some of the other regulatory provisions, there was maybe something to be done there. I think we got to a bill draft or two that were really good along these lines. So I'm a little bit more optimistic about if something had happened in Congress, I think it would have been good. Like the Frontier Act, which is a sort of Trey and

47:10Obernolte bill that ultimately, I think, we can now basically say didn't go anywhere this, at least this Congress, and I think is unlikely to go anywhere between the midterms and the new Congress being sworn in either. I think that was pretty good. I think that was as good a bill as we've seen in Congress. I have to have good mandates for independent oversight, good mandates for getting Casey, the Center for AI Standards Innovation into a better position to do some governmental oversight. I think that was a good bill. And I think if it had passed, I think most people would

47:40have liked it. But I think now, heading into the next Congress, the politics are going to be much more difficult. We're very likely going to have a democratic house, which means it's that government, when there's going to be a lot of bad blood between the two chambers, there's going to be subpoenas, and there's going to be hearings, and there's going to be a lot of ways in which the Democrats use the House to gear up for the presidential election, and also to sort of reiterate a lot of that and relitigate a lot of the conflicts that have happened in the first two years of the second Trump administration. So at that point, that just doesn't strike me as a very productive legislative body. So I am pretty pessimistic about getting anything good done in that

48:14Congress. Law-hanging fruit, however, well, if the executive wanted to, I think the executive could definitely do some good things. I wrote about it just this week, which is like, we have these independent third-party organizations that have a decent amount of skill and expertise in thinking about the most obvious and most sort of concerning AI risks. I think the Meter Redwood investigation of the Hugging Face incident was well-received for a lot of the ways in which it sort of understood the alignment and control side of the problem. I think it's not that big of a stretch to just

48:47slightly codify and slightly sort of enable this kind of investigation at a slightly larger scale. So I think two most immediate things would be, A, it shouldn't be OpenAI sort of with sort of just like inviting, voluntarily inviting someone to look at what they did when something like that happened. It should just be the administration telling them, well, give access to one of this like list of like third-party evaluators whom we like. You can pick them, but we'll give you the list of the ones we like and just like let them figure out what the hell was going on there and then let them write

49:18a report and they'll give the report to us and they'll tell us, did they give you enough? Did they give us enough access or did they not? And if they didn't give us enough access, then they'll come back and get the access. So just put on a little bit more pressure to have good incident investigations that cover the entirety of the incident. I think that's one of the low-hanging things you can do with third parties. And I think the second thing you can arguably do is just kind of embed them or like kind of have some version of continuous oversight of what the organizations are doing. Again, that's something that this Frontier Act train over Nolte bill

Executive action and third party audits

49:47sort of had a basic version of, which is just like regularly some external evaluators just go in and poke around the lab a little bit. They hang out in some of the Slack channels. They talk to some of the safety researchers. They talk to some of the capability researchers. Maybe they get one or two or three sit-down conversations with the executive and they're like, well, what's going on here in terms of safety? Does everything look good? And if everything doesn't look good, they have an ability to sort of just communicate that information to the administration. And if something looks really, really bad while they're embedded, they have this immediate escalatory ladder where they're saying, well, there seems to be imminent catastrophic harm here. We should do something

50:20about that. So I think between pressing for incident investigations and encouraging the labs to allow some version of continuous oversight, that's something you can do tomorrow. And I think we should probably just do that. And I think from there, we can think about how to sort of codify that and how to develop that into laws and how to develop into executive orders, how to integrate that into a sort of FINRA-FOIA ideas. I think we can have a lot of long-term conversations about where to go from there. But this is something we can do now and we should just do it. Yeah. I like that. Let's say the president doesn't do that. An interesting thought experiment I've been

50:54playing with lately is, could these groups just come together and essentially form a sort of union where they basically say, hey, look, we're not very happy having had six days and three people on site and a very small fraction of the relevant data to do our investigation. We demand better working conditions. Do you think that they could, by coming together and making some demands with a single voice, do you think they could actually get those from a few frontier companies?

51:24Well, I think the problem is, like, currently, they're just too reliant on the good faith of the AI companies because of the voluntary dynamic, right? There is no law and there is not even a lot of executive pressure that requires the labs to allow for these third-party investigations. And I think, so if you got together and if you did that and you pressed for, well, we'd like to have full access and we'd like to have access to all the Slack channels and to all the logs and, like, you can come up with a list of things that you might want to have. Then I think there's just

51:55still a world where the labs say, well, sadly, we couldn't come to an agreement with the third-party evaluators, risk to security of our IP, risk to integrity of our operations, worry about information leaking to competitors, something, something. And I think that currently still reads as a fairly reasonable response to that kind of ask. And then the question is, do the labs really have a problem currently if they don't allow third-party investigations? And I think maybe a little bit with their own employees that want some reaction to incidents happening and are not

52:27entirely satisfied with just the internal practices. But I would suspect if the third parties can be painted as unreasonable and extractive when they engage in this kind of sort of collective bargaining, that employee pressure is probably not going to be sufficiently high. And then the other source of pressure is, well, is there pressure from the executive to allow for third-party investigations such that the developers would have to get in even, like, a little bit more adversarial third parties that decided on the standards they want? And I think currently not.

52:58But I think if you just moved a little bit towards the direction that I just described, which is, like, have some administration executive pressure on allowing third-party investigations, then I think whoever is on the list that the administration has could in the next, like, establish some standards of how investigations were supposed to go. But I think the first thing you need is some external incentives for the labs to even go with any investigation at all, because otherwise they'll just say, well, in that case, we're just not going to let you in. And I think, yeah, I think there's just not enough of that pressure around just yet. Yeah. So much depends on he who must always be named.

53:31Another thought experiment. Speak of the devil. If I was president, here's something I would be interested in trying. So you red-teamed this idea for me. What if the president were to say to, let's say, five companies, could be six, seven, hey, this is getting pretty wild. I don't think I really know what to do. But I think you guys can figure it out amongst yourselves. So you have 90 days to come together, come up with an agreement by which you guys are going to work together to pace

54:02the frontier. You'll police it. Maybe you'll use some secure private computing constructs to be able to kind of interrogate what one another are doing. And you'll have to agree on how that access will work. But you'll police one another. And if you can't reach such agreement or can't sustain such agreement, then I'm going to have to get involved. And you're going to like that a lot less. What do you think? Well, I think you should run. And I also think maybe a little, you know, hey, there's still time in the cycle. What am I talking about? Hydrastation? Yeah.

54:34And like, you can get into the primaries. There are undecideds yet for the primary. No. Yeah. I think that might work. I think that is also kind of the attitude the administration currently already takes, where there's a sense of, well, you guys built this mythos thing. We don't exactly get why you would ever do that. But now you've caused this problem. Can you please just like figure out how to fix it like now? And then if you don't, then something, something, export controls, a lot of pressure and so on. I think that is fundamentally like finding some

55:07slightly more structured version of that is fundamentally also the thing that like, that's the sort of Finra for AI, Faro, Safa, like SRO, whatever you want to call the idea. I think that's basically the slightly more structured version of it, which is like industry comes together and figures out what the standards for how this should work are. I think if you put open air and anthropic into a room to figure out what they should do and what they want to do and what they see as the risks and what they think should be done to pace them, I think they'd probably do it. And I think this would work. I think maybe GDM also works. I think Meta and XAI have a very

55:43different view of the risk case, have a much more skeptical view of industry coordination of voluntary industry standards, and of actually doing a lot of things that slow down their capability progress, which to be clear is ironic because something that specifically paced the frontier would give them a much faster path to catching up. So they should structurally and instrumentally be in favor. But I still think they're very skeptical of that. They have a lot of influence with the administration. So I think they would A, just be sort of opposed to that kind of like broader,

56:14like all the frontier labs should not figure out what to do. And I also think they would just be very likely to influence the negotiations in a way that would just make it extremely difficult for there to be any common standard. Because I think if you water down whatever you want to do to an extent that would make Meta and XAI be fully on board with it, I think that just wouldn't realistically be good enough. And then old man and a topic would maybe say, well, this is not good enough. And this doesn't fulfill the spec that we've been given by the executive. So I think

56:44there probably has to be some more substantive guidance than just like you guys figure this out and find some consensus, at least some minimal idea of how to do it. But yeah, I think if you had some more substantive guidance, then there is a path for industry self-regulation. And I think even then we can bring in the third parties again to verify it. That's, I think, one reason, like sort of one mechanism to make industry regulation work. The other is just to have the labs check each other's homework and to just have like, openly it checks out what Antarctic is doing on this and so on. That's a little bit more dicey in terms of like industry

57:15secrets. But there are precedents for this. It's also not unworkable and not impossible. But yeah, I think if you can get to bridge, if you get to bridge the gap between the sort of tier two and the tier one, who are, I think, in very different places in terms of safety, then you can make it work. But I think that's the bigger. Yeah. One, actually, two things that have come up a lot recently for me as I've, you know, because I do go around doing this of putting these ideas in front of people and telling that you're asking them to tell me why it can't work. And common answers that I get to various kinds of

57:51safety minded collaborations are domestically, well, that might be an antitrust violation. So that could be a big problem. And then internationally, you know, even for things that are not at all about exporting chips or chip making know how, people are still afraid, even on just like basic AI safety research collaborations, export controls, it could be a big problem. You know, they're very broadly and vaguely worded and enforcement could be kind of arbitrary. So I guess two questions I have around that. One, do you think it would make a big difference? It seems to me like it would

58:24for the president to just come out and say, hey, here are some things that we are not planning to bring antitrust or export control enforcement against. And if they don't do that, then I also have the sense that like, we maybe need people to be willing to have the fight. So I'm, you know, it's not necessarily my place, obviously, to advise all these AI safety nonprofits out there. But if I were to be so presumptuous, I would kind of say, I think you should go for it and put a little faith

58:58into the judicial system. And the fact that we do have like, due process, you're not going to go immediately to jail for having done some AI safety collaboration, you know, research project with a Chinese academic. So like, don't censor yourself or don't cancel the product before it even gets started, like, go do it. If somebody wants to pick on you, you know, that'll suck. But like, we're in, you know, this is kind of an important time, like somebody's got to be willing to stand up and have the fight. What do you think about that? Yeah, so I think I distinguish between the cases

59:28here. I think the export control collaboration case, I think at that point, you're just basically talking about whether you can insulate yourself against sort of vindictive and capricious action by the Trump administration. And I think there, yeah, if you think it's worth doing, then you should just take the fight to that authority. I think that's clearly not in scope of that authority. I think the Trump administration is probably should probably not use these authorities to crack down on this sort of research cooperation you described. I think on that, I think I'm with you. On the antitrust

1:00:00stuff on like industry coordination, I think the problem there is a it is actually unclear whether it isn't just like a substantive antitrust problem to do like substantial industry sort of collusion on not competing on frontier development. And therefore, like it, it's unclear whether this actually has sort of inflating pricing effects or not. But at least in all other domains, it would have like industry coordination on not agreeing not to pursue like further technological innovation usually has like adverse pricing effects that you would really not want. So I think

1:00:32this is arguably in scope for actual like antitrust rules. In that case, yes, the administration could quite easily come up with guidance with like non-enforcement letters on like, well, we don't plan to bring any action against anyone who coordinates for the sake of AI safety between an industry. I just don't think that the administration is actually going to do that, because I think the administration has so far enjoyed finding new and sort of novel pathways to be annoying to anthropic specifically. And I would suspect that the moment that anthropic decided to come up with

1:01:07any sort of substantive and helpful way to coordinate between different labs to make some like deceleration of, you know, may make some pacing happen. I think the administration would just find some way to to act against that. So I think the only way you can do that, that saves you from that sort of enforcement is broader industry cooperation, you get XAI and meta and open and into the boat early. You make it very difficult to target just anthropic or just labs that the administration doesn't like with this sort of antitrust authority. And then I think you're probably safe. But the

1:01:39problem is, it's just very, very difficult for these organizations to take the fight to the Trump administration. Yes, there is due process. But the IPO conversation we had earlier, I think plays into this, which is like, well, do you really want to go like 14 rounds with them in some court? And you want to like, do you want to bet that you don't get any sort of like very Trump favorable judges and as they did get on the DC court on the supply chain risk designation, for example. And I think so many things just can go wrong. The profits can take so long. And you want to IPO in a few like a few weeks, months, whatever. And so if that's your goal, then you just don't want to take the risk

1:02:11right now on getting bogged down in any sort of long antitrust lawsuit. So I think the sort of scrappy safety nonprofits should probably take the fight to the administration if it really stands in the way of what they want to do. I can't blame a traffic for not wanting to go into antitrust lawsuits months before the IPO. Under ordinary circumstances, I would agree with that. I do think with how many people they have had come out and say, yeah, I think 10% plus totally reasonable. It's like, if you're willing to take that risk, I think you'd also be willing to spend some time in court along

1:02:43the way, but maybe that's just me. Where does all this leave us? It sounds like, I mean, of course, this is like the baseline, the base, you know, coming into this conversation, the baseline assumption is like, we're probably just going to kind of muddle through the, you know, the current state of affairs will mostly kind of continue until at least the foreseeable future when something gets even crazier and, you know, shakes us out of this equilibrium. Is that basically your view? Like we need another big incident, you know, warning shot 2.0 to really open up space for different paths?

1:03:20I'm just, I think it can be external incidents. I think it is also, we'll see how the next Congress looks. I think it's going to be interesting. I think, I think it's going to be much more about political incentives that will change things in the next like few months. My expectation would be that is the main pathway for things to really materially change and be different. It's not so much that, it's not so much that something has to change about how people view the technology. I think they think is ripe for regulation and ripe for intervention and the

1:03:52demos are getting crazier and things that happen are getting crazier. I think there's just, I think that's probably, there's probably enough happening there. I think the thing that I, it's just when do the politicians and the policymakers move on this? And I think currently there's just not that much political incentive to move. There's going to be much more political incentive than Democratic House to keep like pushing and prodding and introducing things. And we'll see how the GOP reaction to that looks. And I think the interesting wildcard is how do the presidential primaries look? And I think on the left, there's just going to be a lot of, I think, anti-AI sentiment. I think people are going to talk

1:04:25about AI and AI safety a lot. They're still not going to have any ability to get anything done. So I think that puts whatever action they take all the way into 29. The more interesting thing is what kind of record do J.D. Vance and Marco Rubio want to run on? And I think that is going to be the question about, I think that's going to be the determining question for whether we see any AI policy action in 27 and 28. It's like, do they actually take the view that we can't run on a record of the Trump administration doing nothing about the risks that are getting people more and more concerned? Do we need a bill to pass? Do we need some executive action to happen such that we can't get pinned

1:05:00down on the sort of like broad pro-AI accelerationist position come the elections? And I think probably, yes, that is in their political interest. And the question is, will they have, will they find a way to get that through despite donors and perhaps even the president pulling the other way? But I think that is more a political question that has to do with what does the polling look like? What does the salience look like? What do the midterms look like in terms of AI salience and AI impact on electoral outcomes? How do the primary dynamics sort of unfold and where do they leave the candidates?

1:05:32But I think I'd mostly look at these political flashpoints of the primary season beginning and current cabinet officials being unhappy with running on the current track record that I would expect to change things. And I think that is the way that we do get legislation and actual action in 27, 28. And I don't think that's impossible. One more US question on the build out. It seems like the build out is actually like happening. It was obviously a lot of noise around it, a lot of heat around it. But like, my best guess is that this will kind of look like a fracking story where it's like,

1:06:07it happened. You know, we in a lot of different places, people found their right plot of land with the right jurisdiction. They bought off, you know, or they built the parks and the, you know, the stadiums or whatever that they needed to, bread and circuses kind of carry the day and it happens. Do you see any reason to doubt that? Yeah. It's federalism, right? I think there's just, I think there's, I think there's so many places you can build data centers. A lot of the backlash in the policy implications of the backlash have sort of been, I think, overstated. I think especially the Texas

1:06:37moratorium isn't very much a moratorium in any practical sense. I think people sort of now they, now they draw up these maps and like everything that is sort of has a moratorium is sort of colored red. And I think if you make a map of places where you can no longer build a data center, coloring Texas red is kind of disingenuous. Like there are some minimal standards for what the data center projects need to clear. The hyperscalers will clear them without any problem and they will continue to build in Texas as long as they have access to behind the meter power that runs data centers. We might be running out of that, but that's a different conversation.

1:07:08The Midwest is, I think, genuinely sort of anti-data center. It's going to be difficult to get things done there. I think the New York moratorium, at least for the next year or two, is more real than the Texas moratorium. But I think for the rest of the country, you can still build in Texas, you can still build in Louisiana, you can still build in the Dakotas. And there are also like tens of gigawatts already committed to construction projects that are continuing on. It's going to get more difficult. It's also going to get more expensive. I think you're going to have to pay more concessions. You're going to have to make more expensive deals. I think all that is a real

1:07:38effect. And I think some of this is going to push some of the build-out into other countries. Some of this is going to make the build-out somewhat slower, some of which is going to make it more expensive. But there are a lot of states. There's a lot of land. They're going to keep building data centers in America. Do you have a theory for why that hasn't happened with nuclear power plants? Is it just that they're not enough better than the alternatives? Or is there some other reason that we haven't reached that same equilibrium there? I think a little bit less of a, you can put

1:08:09this wherever you want to, and it just pays the same way. I think there's just fewer places you can connect. You connect it to somewhat local electricity, man. You connect it to somewhat local grids. You can't just build all the nuclear power plants for the country in Maine or whatever. And so I think there's a little bit less of a dynamic of, well, just put them wherever they work. And also, my understanding is there's also more federal-level oversight over where and how and when you can build nuclear power plants, as opposed to data centers, which are like, you don't have to go through any federal approval process to build data centers anywhere.

1:08:40You just have to build the data center. And so I think that combination makes it a little bit easier. But I will also say I'm just not super steeped into the US domestic nuclear build-out conversation. Yeah. I mean, I think that federal-level oversight is probably a key part of it. And that is a big part of why, as much as I am legitimately scared now of AI. It has moved recently from a sort of, this could get really scary to like, it is actually now scary. I'm still like, oh God,

1:09:12you know, don't take, don't give me the nuclear outcome, you know, that would we get the weapons and not the power plants? Like, I would just be so bummed about that, that I'm like a little reluctant to go all in on a federal oversight, you know, even as much as I feel the need. I mean, what's your version of the weapon, is I think one question here, right? I think just, I think the thing about nuclear weapons is you can build the entire supply chain for a nuclear weapon without ever generating any sort of civilian benefit. It's really hard to build a model that's

1:09:45just good at winning your geo-assisted competition that isn't accidentally also a big economic boon, right? I think like all the ways in which AI systems are really economically useful are so general purpose that is really hard. Like you have to go through a lot of effort to not accidentally make them pretty useful economically as well. I think the question is like, do you get super intelligence in your pocket? And I think that's an open question. But even if it's just the US government sort of procuring super intelligence to use to win against China, whatever that means, I think that just incidentally still builds a system that's very economically useful.

1:10:19So I think much less than with nuclear, I just don't think you can divorce the civilian military uses in the same way. So I think in that sense, we should be optimistic based on the nuclear example, which is like, well, we didn't stop building nuclear weapons just because we were kind of, at least we didn't stop entertaining nuclear arsenals, I should say, just because we stopped building out nuclear power. I think in a somewhat similar way, we're not going to stop building AGI and super intelligence and whatever, just because we're not, because there's some domestic resistance. And in the case of AI, I think there's just going to continue

1:10:51to be the sort of like civilian economic spillovers much more easily. So I think in this situation, the strategic impulse actually cuts sort of cuts in our favor. So maybe that's one thing that might make you a little bit more optimistic about it, but not all the way to super intelligence in your pocket, but a little bit more. Yeah, I'll take what I can get. So let's talk about the rest of the world. You have this big report that just came out on transformative AI strategy for Europe. And obviously, there's been some discussion. And I actually talked to one of your co authors a bit back about kind of the compute deficit that Europe has and the need to kind of do something to, you

1:11:27know, be a live player going forward. But before we get into like the strategy for what Europe should do, what is the worry if they do nothing? Because I also think like this is probably going to whatever you think might happen to Europe, if Europe stays the course, is probably what happens to like 70% of the world's population, maybe 80% of the world's population, by default, right? What is what is like the future look like in your mind for Africa, Latin America, South Asia, etc?

The future for lower income countries

1:11:55I think it's going to be really tough. Because I think fundamentally, a lot of the catch up mechanisms that lower middle income countries in very general terms have sort of used and enjoyed and been able to leverage over the last few decades, are just deeply incompatible with a world that has both like very advanced AI systems that then ultimately also a lot of sort of automated manufacturing capacity and what that's downstream of that. So I think mainly the sort of the most immediate and obvious mechanism was always just bet on the sort of demographic differences,

1:12:28we just had like very rapid population growth, you had a fairly cheap workforce that you would be able to use to your comparative advantage, and then quickly sort of bootstrap into hosting some foreign firms and sort of exporting some sort of valuable good to the global supply chain, in a way that was predicated on the idea that you had this workforce that you would just be able to put to use in a way that would make you like a comparatively beneficial country to conduct business activity. And I think, I just don't know whether that's going to remain the case. It's definitely not going to

1:13:01remain the case for most aspects of the sort of like sort of menial sort of like services economy. I just don't see a stable way that that sort of that bad sector of the economy really exists, once we have very powerful AI systems. I think there are definitely going to be new services jobs. And I think the human preference jobs, and you can think about all these sort of like labor market effects in the long run. But I think this idea that you can just like put, be immediately useful to global supply chains, just by doing labor cheaply, in the service realm is I think just not

1:13:33going to work on anymore. The question is, is it going to continue working out in manufacturing? I think that has a lot to do with like how quickly automate, how fast automation goes, how big the efficiency gains there are. There is still a world where like manufacturing just gets more and more bottlenecked in a post AI future. And then it turns out you can at least sort of catch up via manufacturing. Doesn't strike me as entirely impossible. But I think that much just for the general catch up mechanisms. Then the other question is, what's the stable geopolitical end game? And I think even if you get to this sort of manufacturing plus like sort of cheap jobs sort

1:14:05of part of the catch up mechanism, it just seems very difficult to figure out how any country in that spot ever gets any leverage of what happens at the frontier, which is to say, they don't get any oversight or regulatory input into how frontier AI systems are built. And they probably also don't have any hard leverage that makes sure that they'll continue getting AI exports and continue getting access to AI supply chains and so on. And so they're basically at the mercy of whatever great power provides them their AI models. And then maybe within that, they can find a somewhat

1:14:40favorable arrangement, but it seems very unlikely that they'll get like a stable say and a stable input into that. And I think, you know, I think that just cards the world into spheres of influence of those that have very powerful AI that are able to export it. Bunch of other downstream questions that make this more complicated. How much do you need frontier AI? How much do open weights play into this? At what point can you build like your own digital software infrastructure? But I think at least for the medium term, it is this quasi-vassalage to the frontier AI building powers that I think is just the most likely outcome for most of these countries.

1:15:12In terms of like how people live, do you think that that could create a story kind of similar to the Chinese story over the last few decades, where it's like, life is getting a lot better, we're getting richer, we just don't have a say in the overall high-level direction, but like at the street level, you know, things are trending up and up? I think in absolute terms, people are going to be richer and wealthier and better off. I think, so I think in terms of the economic effects that is, I think they're just going to

1:15:44be relatively disempowered when it comes to meaningfully shaping the trajectory of the world, and also in terms of having an ability to catch up to however well the sort of the frontier countries, so to speak, are doing. But I think in absolute terms, it's going to be, it's going to be, it'll keep being growth and it'll keep being spillover effects and redistribution gets easier as well. I think on the streets, it's going to be, it's going to look nicer. It's going to be sort of just economically better scenario. So I think in absolute terms, you wouldn't mind too much. I think the more fundamental question is, what does it say about sort of democratic say and human

1:16:18autonomy and human dignity, even that none of these decisions really factor into where the broader trajectory of the history of the world goes. And I think that is a more profound sense of disempowerment that I think we should still be concerned about. But yeah, practically speaking, not that bad. I think the other part of it, practically speaking, is susceptibility to misuse. And I think that could be extremely destabilizing. So I think there is a current assumption that to guard against a lot of forms of AI misuse and AI loss of control and so on, you need your own AI

1:16:49systems that defend you against that. It's most obviously true in the realm of cyber. I think it is also conceivably true in tracking and monitoring potential deployment of pathogens, the sort of entire virus conversation. It is probably true in terms of scanning and filtering and screening against scams and all these sort of like socially engineered attempts and whatnot. And it is probably also true in terms of like safeguarding infrastructure against extortion attacks and so on. I think if you expect there to be a world where non-state actors, terrorists, criminal groups get access to at least

1:17:21fairly capable AI because they're able to steal it, because they're able to sort of pulse train like terrorist GPT on some open source model, whatever, right? And you also expect these countries to not have any sort of coordinated and assured and widely deployable access to these systems. I'm not sure whether they're going to be able to protect their citizens from AI-driven harm, AI-driven sort of misuse, and also potentially the labor market effects. I think that all sounds like they would be very susceptible to that. And then you can imagine a lot of very destabilizing scenarios, right? If your country no longer, if your state no longer protects you from sort of AI-driven

1:17:56harm, then what do you turn to? Maybe you turn to mass migration. Maybe you turn to the sort of other ways of structuring your sort of like personal security, as we already see in some of the failed states in Latin America, where I think with sort of like criminal enterprise runs a lot of like day-to-day structure in a lot of parts of these countries. I wouldn't think that would be impossible for a lot of these countries as a sort of like medium to long-term outcome. And I think that also has me very worried. But I think the pure economic story is pretty positive. The eroding the state,

1:18:26the authority and power of the state story is, I think, a lot more concerning. And I think sort of take these together, and it's not a particularly rosy outcome. So that's probably 70% of the world's headed there. And Europe is kind of the one place that can maybe engineer for itself a different outcome. Tell me, I mean, if you disagree with that, tell me, but I'm going next to, okay, what does Europe want? And how does it get it?

A transformative AI strategy for Europe

1:18:52Yeah. I mean, I think the problem that Europe faces is not too dissimilar to what a couple of other Western countries or like sort of general sort of liberal democracies face as well. I think fundamentally, Australia is in a similar boat, New Zealand is a similar boat, Japan, South Korea are in somewhat similar situations, Canada is in similar situations. So I think that plus Europe plus the UK is, I think, the cluster of like US allied middle powers that have a potential trajectory out of this. It still needs a lot of work. I think the fundamental question from sort of, I think there's two

1:19:27fundamental ways to start looking at this. The first way is looking at, well, put aside all the AI things, what do you want Europe's economic position to be? And I think, if you start thinking of that, you just think, well, you want to be good at the things that Europe is currently good at, you want to be good at some aspects of the manufacturing, you want to be good at some aspects of the artisanal, of the artisanal good, you want to be good at the sort of high state capacity things that Europe currently is good at, whether that's sort of welfare states, whether that's, whether it's just like high levels of security and safety, this isn't really like, a lot of things that are

1:19:58going well in Europe, you just want to keep, keep sort of, you want them to keep going well, plus you want to kind of some way to actually revitalize your current economy. And then AI comes into the picture as like, well, that seems like it could either really accelerate that, or it could really sort of destabilize that. And then you ask the question of, what do you need AI for in that context? And I think the other part, the other way of looking at it sort of comes to the same conclusion, which is like, well, what does Europe currently not have? And the answer is, it currently doesn't have

1:20:28frontier AI systems, which turn out to be one of the most important economic inputs of the future, and also one of the most exciting parts of like strategic and economic competition right now. And the question is, well, what do you do about that? And I think there you quickly realize, well, building these systems ourselves is just too expensive, it doesn't actually work. So the next best thing we can think about is how do we get access to frontier models in a way that is short and secure, and allows us to build what I sort of described as the first approach around, like, how do we reduce the geopolitical risk of just doing the things we're good at? How do we make

1:21:01sure we have a short access to frontier systems and don't get cut out of this like AI, AGI conversation while we do the things we're good at? And I think these all sort of come together to, you need something to incentivize selling frontier systems, you need something to make the Americans not nervous about selling frontier systems, and you need some productive way to use the frontier systems downstream to make something happen around that. And I think the strategy that we wrote basically answers, tries to answer these questions. And I think especially the parts that I most

1:21:33contributed to try to answer these questions. And I think just very briefly running through this like high level takes is like, the first thing is this compute for access idea that I first wrote down late last year, early this year, and started sort of shopping around with a lot of countries in the world ever since, which is now also one of the sort of pillars and one of the main sort of asks of the strategy, which is to say, we built data centers for American or like in cooperation with American hyperscalers. In exchange for the favorable conditions we provide these American hyperscalers and labs, we get a short access to the models that run on these data centers. And as long as the

1:22:08Americans keep giving us the model, they continue to get access to the data center. If their site goes back on the deal, and we're cut off from access to the frontier models, they lose access to the data center. This is the sort of incentive part of the conversation, rebuild the infrastructure and get access in return. The second part is, how do we make the Americans not nervous about doing that? Because done wrong, this is a security risk, right? You can't run this. You've done it with the UAE, for God's sakes. We should be able to reach a deal with Europe. Yeah, well, I mean, I think the UAE thing is like kind of fragile. I think the UAE is kind of worried

1:22:39about, well, what is the future of that? Will there actually be frontier weights hosted on UAE data centers? And I think that's very unclear. I think they're going to run some inference on them, but maybe it's just going to be like Haiku inference and not like Fable 6 inference. And I think that is an open question. So the question is, how can we get the security alignment to work out in a way that the Americans aren't too worried about hosting the models there and giving the model to the European economy? And I think that just has to do a lot with aligning with the US on a lot of these security provisions, building out the data centers to be just like secure on the

1:23:11cyber and physical side, building out sort of KYC regimes with European firms, just make sure that the Americans don't have any well-grounded national security worries that were put against the incentives from compute for access. And I think the third thing is, let's get a little bit more self-assured about the assets that Europe does have, right? Europe has both sort of broad economic assets and just like a very powerful economy, still in absolute terms, if not in terms of growth trajectories, but still in absolute terms. And Europe has a lot of assets in the semiconductor supply chain, right? Like ASML and SAIS and all these things that play into building frontier chips and

1:23:43different frontier models. Let's just think about how we can be strategic about that. Let's set up an anti-coercion instrument of sorts that says, well, if everyone plays nice, we'd love to feed these assets exclusively into the American supply chain. We're willing to align with export controls vis-a-vis China. We're willing to just, you know, we're willing to be good friends and good partners to the US. And also, if the US ever doesn't decide to use its ability to cut off frontier models as a means of coercive action, then we're also willing to use the supply chain bottlenecks that we have as coercive action in return. And I think between those three, frontier access is pretty assured. And then you're at a

1:24:17point where you're sort of back to where we were before AGI, which is like, Europe still has a lot of structural economic problems. We still have to solve them. But at least we fixed the geopolitical problem of being cut off from frontier access. And I think that's maybe step one. I think that's the things that I'm excited about getting done in Europe in the next year or so. What's the hardest part about it? Is it just getting data centers actually cited and built? Or is there other challenges that you think would be bigger than that? I mean, we can have this conversation and there's like a shared understanding that

1:24:50the suggestion I make interface with like a realistic future that we think might happen and that is worth preparing for. This is not the case in many rooms with policymakers in Europe. I think there is deep skepticism of the continued trajectory of capabilities of US built AI models. There is a lot more optimism about the broad availability of open source competitors that can basically do everything as well as the American models. And there is just a more fundamental question around, are these models that powerful? Is it that important? Is that as big of a

1:25:23geopolitical issue? And then there is also a bigger question around, well, if the models are that important, if everything that I and we and people say is true, then like, why shouldn't we just build this ourselves? That surely can't be that expensive, right? We'll find like a more clever way to do it. The Americans are sort of wasteful, high on their own supply anyways. Like, we'll just like spend a few million dollars and like surely we can like whip something up. And I think just cutting through that, which I understand to be just like complete misunderstandings of the material reality we find ourselves in, just cutting through that and like making the point of no, this is not actually,

1:25:55this does not accurately describe reality. You need to think about this in sort of clear-eyed ways that respect that this thing that is happening in America is real and the Americans are fundamentally right about a lot of the aspects of this. That's the biggest barrier. And then I think if that awareness existed, I think there are still political things to figure out. Like for the ASML thing, it's going to be some amount of triangulating between the Dutch government interest, the ASML interest, the interest of the other member states. That's not quite easy. In terms of data centers, there's going to be some domestic skepticism against American tech firms and working with them.

1:26:26That's not going to be quite as easy. In terms of the security alignment, there are going to be people that are going to be more excited about hedging toward China and just like trying to stay between worlds a little bit. I think all of these are surmountable, very, very easily surmountable, if you just get the alignment and the awareness of what's happening here, right? And I think that's the main challenge. So what's, one of the things I did notice in reading the report was, you know, the authors, you and your co-authors sort of are willing to dream a bit in terms of

1:26:57how the authorities might act, right? I mean, there's, at some point, there's basically a statement that like doing this in the sort of half-assed or kind of, you know, highly bureaucratic, you know, everything's a committee to nowhere mode that European governance, at least by reputation, often acts would be maybe worse than not doing anything at all. So how do you think, you know, how realistic is it that you can actually get this sort of action? And like, what's the mechanism

1:27:28for doing it? I mean, it's still downstream of urgency and awareness of the situation, really. I think there's always a trickiness in writing for, I think, for national governments generally, really, which is you try to write something that isn't quite within the Overton window of what they're willing to do, but that also isn't so far out there that they'd never do it, right? I think like, one failure mode is you just write it, well, we just have like 20 million in the budget. So like, let's just think of the maximally AI-pilled way to allocate the 20 million. It just turns out it

1:27:59just doesn't matter. Like, either way, just think you can just burn it, you can throw a party, it doesn't matter. Like, it's not going to change the conversation. And the other failure mode is to be like, well, we're going to be maximally honest about what we think should be done. And we just write exactly that up. And then, you know, you guys can tell us, oh, I'm so sorry, we were wrong later on. And then it's going to be too late to do it. And then we're just going to shrug and be like, well, you know, we told you the honest thing. I think that the art or the trick of getting this kind of thing right, and I hope we struck a decent balance, is AMAT just a little bit more ambitious

1:28:29than they currently are, accounting for the fact that they will get more ambitious and that they need some nudging toward being more ambitious. And I think that is the sort of calibration that the strategy tries to reach. I'm optimistic about that. I mean, if I had to give you odds of this strategy as a whole being implemented within the next year, it's not that high. If I had to give you odds on elements of it, making it into serious policy attempts, and actually getting set up, I think it's pretty high. And I don't know which ones of these that they're going to be, I could make

1:29:01bets on which one of ones of them are going to be more popular and less popular. But I think some of this is going to happen. And I think that's the sort of hallmark of a well calibrated strategy. We have done this before in Europe. There have been times where Europe has managed to very quickly act very decisively in a way that has delivered results as quickly as it has anywhere in the world. I think the thing that I used to work in German policy and politics for a bit, especially in energy policy, and in the immediate aftermath of the start of the war in Ukraine, there was an absolutely heroic effort of the German government to buy this fleet of shadow LNG tankers

1:29:34all over the world to get them to transport alternative gas supplies to Germany once the pipelines were cut off. There was a justice heroic effort to get like LNG terminals built out in some of the most nimby parts of the countries. And within like six months, the German government, with all its capacity and all its urgency, managed to consolidate resources to, we made it through the winter, no problem. Max, like complete fine energy, no problems at all with just like lack of heating or anything. That was like the big doomsday scenario. It just didn't happen. There was like a massive heroic effort and it just, it just worked. I think before that I worked in COVID policy when, when that was

1:30:08happening, there was like a lot of political pressure against joint vaccine procurement. It was very, very difficult to get the negotiations right. It was very, very difficult to get the member state interests aligned and Europe managed to procure vaccines fairly well. And I think the vaccination campaign in Europe went fairly well. We can talk about non-pharmaceutical interventions. They get a little bit more of a messy story, but I think that all went pretty well. And I think Europe can't do this. Europe just has to realize that this is sufficiently important. And I think we're not that far from realizing that this is important on a COVID slash Ukraine war scale. And I think if we just get there,

1:30:39then we can definitely make progress on the kind of recommendations we make in the, in the strategy. And I don't think that would be a big problem. Cool. Very interesting. Are there small countries that you think are worth calling out for taking a, you know, distinctive and potentially effective approach? I was thinking, I don't know anything about this other than that I know that their sovereign wealth fund, I'm thinking Norway, I know their sovereign wealth fund is a large and be like AGI pilled in its operations. Like they use a lot of agents and, you know, they're, they're

1:31:12kind of transforming themselves, but I don't know if that's translated to something like a national strategy in Norway, Singapore comes to mind as somebody that might be interesting. Who else is like doing interesting things out there, even if they're, you know, kind of small and carving out a, you know, a narrow path, perhaps. I mean, threading upon the needle, you might say.

Small countries and compute hubs

1:31:31Yeah. I think Singapore and Norway and the UAE are probably the three you did most obviously mentioned. I think Norway could do much more. I think the exciting thing that Norway could do is you could become an inference hub, haven, whatever for the entirety of Europe. You could, it's a little bit hard to invest domestically with the sovereign wealth fund itself, but you can conceivably come up with sort of like schemes to invest it into European consortia that then invest into compute building Norway. Norway turns out to be a pretty decent place to build a lot of compute, hence the

1:32:04first attempt to build a Stargate, now a Microsoft data center there. I think Norway could be more AI pulled about deploying these resources, but I think Norway has a lot of resources that could easily be deployed and pivoted towards that. And I also just think the base case of just like investing a bunch of your wealth fund into like basically just like recreating, like, I mean, maybe saying recreating situational awareness is no longer as en vogue as it used to be like two months ago or whatever, but like basically recreating like a fairly AI pulled portfolio and part of the wealth fund probably just lets you sort of ride on the coattails of the AI revolution for quite a long time.

1:32:37That probably works. The UAE play is building a bunch of data centers that basically just finding a way to turn money into something that is an asset in the new economy. And I think that's a good way to spend a lot of money if you have it. Now it's incidentally kind of a tough situation to be in that you're that close to Iran and that it's that easy to drone strike data centers. So I think that has sort of thrown a wrench into that plan. But I think structurally that was still a pretty good play. If they can manage to build the data centers quickly and if they can manage to secure the next generation against drone strikes and so on, I think that's still a play that works.

1:33:08I think Singapore is another interesting case. Like A, there's also wealth fund investment questions. B, there is just a massive amount of state capacity in terms of like understanding what's going on and engaging with that. I think probably no parliament with the greater density of readers of very AI-pilled, very insidery publications than the Singaporean. I think the same thing goes for their civil service. It's just Singapore is a little bit more tough because a lot of the Singaporean economy is very exposed to AI disruption. And I think if you're Singapore, you just have to

1:33:38sort of hope that the white-collar apocalypse doesn't look quite as apocalyptic because it's just hard to pivot an economy that is as big and as service indexed as Singapore toward a completely new way of operating. So I think there's a lot of capacity and interest there, a little bit less of an obvious AGI play to pursue. Are there any other countries that you think are well-positioned, maybe more well-positioned than they know, that should be doing something that are just kind of sleeping at the switch? I mean, I think Australia is kind of awake now, but I think for the longest time it was Australia

1:34:12because Australia is just such an insanely good place to build compute, both for just like data center construction, energy supply reasons, and so on, but also for security integration reasons. Like there's a really deep amount of like national security trust between the Australian and the American agencies. I think there is a strong understanding that Australia would not defect to China in any way, that they would be willing to play ball and all the alignment on China-focused export controls. And so that just makes Australia a great place to run like compute for access, data center, build out place. I think more recently we've seen more of that happening. I think that's

1:34:44very good. But I think for the longest time that was a sleeping giant. And I think it probably still is. You could probably still like 5x, 10x the data center ambitions and just run the inference for half of the world out of Australia. And that would not be an overly ambitious thing to do. And I think, you know, there's still a lot to be done there. I think maybe the inverse of this is the UK, where the UK has just the greatest density of like talent and expertise, both in government and outside, just outside of the US. And it's not quite entirely sure what exactly they're planning to do

1:35:14with it, or if they can do anything with it, just because the broader political conditions of the UK, the skepticism towards US alignment, but also the sort of damaged relationship to the European Union and the rest of the middle powers. It's just very, very difficult to figure out what this incredibly talented cluster of people is actually supposed to do in the UK. So in a way, the UK and Europe really have inverse problems, where Europe has amazing assets that it could use extremely well to actually have like very like live player position in this AI conversation.

1:35:45And it's just really hard to get them to, whereas the UK has all the awareness and all the expertise in the world. It's just not entirely sure what they should even be doing with it. Because at the end of the day, if your starting hand doesn't include any cards that are really good for an AI future, then you can be as aware as you possibly want to. And it's still really hard to get something done. If you are the rest of the world, like, let's say you're Brazil, or you could, you know, pick your countries, or maybe you would put different countries into different positions. I hope this doesn't happen. You know, my, I'm hyperstitioning better US-China relations and some, you know,

1:36:20form of collaboration and not carving the world up into spheres of influence. But one thing I've been wondering lately is like, because of course, there was this reporting that the Trump administration was like planning to do something along those lines and tell countries like you're either with us or you're with China, you know, pick your, pick your camp. If you were put in that position as, say, a Brazil, or again, pick your country, how would you decide? Where would you go? Well, I mean, I think the more important you think AI is, the less justifiable it is to go

1:36:52with China here, just because there is no Chinese AI expert program right now, right? They just don't have the chips. So I think if you, if you think your economy needs access to AI systems, and then you can sort of figure out the rest, then I think you just need to go with the US because they, only they can give you access to the computing capacity, which is why a few colleagues in, I wrote this, wrote the paper, which was in itself a follow-up to another paper, which are both called the Closing Window to Win, which are about the sort of American AI export ambitions. And just like, China will eventually be better at offering these export deals as China has been in the past in

1:37:26like a bunch of international initiatives that they've run in South America and Africa and Central Asia. But currently they're not because they can't offer any data centers in any chips. So they can't actually offer a full stack export that can match the US ambition. So right now, I think if you're sufficiently AI-pilled, you just have to pick the US. At some point, China can probably throw in enough like non-AI related things that the deal looks a little bit more attractive. But just in terms of, well, is there any sort of like hedging strategy to be had in AI specifically? I just currently don't think there is, as long as China doesn't have the chips. That might change in a few

1:37:57years. And I think then the decision is going to be much harder. But currently, it's basically just, well, how reluctant are you going to be about buying US systems, I think is the realistic question that a lot of these companies face. And I think ultimately, like that is a great position for the US to be in strategically. And the question is just, can the US actually offer a deal that these countries will think it will stick to? And I think that's maybe the main strategic challenge for the US, which is like, yes, everyone is tactically and strategically incentivized to take the deal, no way around it. But they still don't like getting a deal that they feel like the US can renege

1:38:31on at any point in time. And so the US has to figure out some way to commit to these deals in a way that's credible to these countries. Building data centers is part of it, sort of deep industrial integrations are another part of it. But I think that is something that the US just has to think about much more, which is like, even if the deal is necessarily the only deal the other country can take, it might still irrationally defect, if you're a sufficiently unpleasant partner to make a deal with. And so the US just has to think a little bit more about how to be like a slightly more pleasant and reliable partner. And I think it's not that far off. But I think, yeah, yeah, you should do

1:39:01We've got the guy for the job. So perfect. One thing I have gone back and forth on quite a bit over time, because I'm a very AI focused person, of course. And then as, but China sort of, you know, became the end point for a lot of conversations that I was having, I became a little bit more of a China person. I'm still not much of a China person, really. But I always had this question of like, how is this strategy where we have these export controls? And, you know, like so much of

1:39:33what you're saying really depends on timelines, right? If you believe in super intelligence in two to three years, you got to be on Team USA, because there is no Chinese export. I agree with that. At the same time, if that is the path we're going down, the chips are made in Taiwan, you know, it's real close to China, it's real far from us. And I just don't see a world where all this is allowed to reach its culmination point, you know, a la Machines of Love and Grace, where it's like, now we're going to make some sort of, you know, deal with the Chinese that they can't refuse,

1:40:08essentially, and realize eternal 1991, without them just being like, fuck, no, you're not like, we're taking out the, the fabs. How do we not end up in a world like all these sort of things, they seem to be taking, as far as I can tell, they seem to be taking us to a point where China's going to hit a breaking point, and they're going to be like, the fabs are going down. And I don't know how we get around that with the strategy that we are playing, like, we can't defend them, right? So I mean, yeah, I mean, it's like a super sensitive asset. You know,

1:40:42it doesn't take a lot to, from what I understand, like a piece of dust or a skin flake, you know, can ruin a batch. So they can presumably not really be defended. Like, how do we not end up there? Sure. So I think I'm also not a big, like, US-China geopolitical competition, how do we win this guy? I think I have 194 other countries to focus on, and I think that just, like, hasn't, hasn't left me enough time to really, really think about this in as much detail as others have. I think, like, very briefly, there's, like, three ways to avoid that. The first is,

1:41:12the AI systems just get so powerful that it doesn't, like, and, like, the US is so far ahead that basically escalation around Taiwan is suicidal for China even more than, like, accepting some amount of US domination. It's very unclear to me what exact shape of AGI is so powerful that that would be the case. But I think if you are sufficiently sort of, like, ASI and, like, really sort of super intelligence killed, at some point, you might actually think that, yeah, that's just, like, a dominating, dominating advantage, and you just actually

1:41:44can't go to war with a country that has this kind of system. Maybe that is part of it. The second thing is, well, that's, like, maybe the fabs being blown up is just not that big of a deal, because, yes, it obviously destabilizes the entire supply chain. Obviously, that's, like, the end game in terms of US-China competition, and who knows what happens then. But, you know, you'll have some indigenous capacity in Arizona, and you'll have all the chips already up and running. So maybe if you're already sort of, like, halfway into your intelligence exploring by then, then it turns out you can just run all of this on the chips that you've already built.

1:42:17Yes, if, like, TSMC gets taken out, like, the chip supply in a year really takes a hit. But maybe AGI can do a lot in a year, especially if it gets TSMC Arizona. And I think the third thing is, well, China also didn't have indigenous capacity, and it's not entirely clear that, like, going to war with Taiwan in a situation where they already think they're behind in the AI supply chain is the best way to escalate the conflict, especially if there is still, like, some TSMC capacity in some way, shape, or form indirectly

1:42:49ending up in China. If that is the case, then China might just think it's best, like, catch up hopes, revolve much more around, like, domestic and industrial integration. And, you know, like, that's the, like, least AGI-appilled version of the future. Or it's just, is it really worth going to war at this specific point where the Americans have this, like, decisive technological lead on AGI, and the Chinese diffusion play and the semiconductor engineization play hasn't really worked out yet. Can we just, like, kind of work in their shadows a little bit more, indigenous and more of the capacity, and then deploy later on? Not sure whether that's the best strategic take, but I think that might also be one strategic

1:43:21approach they take. But I also think it's a massive vulnerability, and I think, like, I think any, like, reasonable AGI endgame has to account for the fact that the Taiwan situation just might blow up in our faces that I think a lot of them aren't. Yeah. Again, to talk about threading the needle. How about, this has been awesome. How about maybe a little lightning round to close? Yeah. You mentioned hedging, and also in the context of the Norway sovereign wealth, like, you know,

1:43:53buying the right equities to get through the AI transition in a successful way. A challenge I've been wrestling with a little bit lately is the Tyler Cowen challenge of, if you're so doomer, what are your shorts? I've been trying to come up with, is there an actual answer to that question? Is there some way, and I'm not a total doomer, but I'm like, I think he should be taking it more seriously than he is. So I want to have an answer that either, you know, I either hear my shorts, or I really tried and I can't come up. And that's kind of where I'm at right now. I cannot come up with a

1:44:24way where I think I can get rich in the doom scenario. Do you have any suggestions to answer Tyler? Yeah, I think this, I think my, my general sense of that is just like Tyler imagines a much more continuous and smooth on ramp into like actual doom that makes it so that like, sort of just betting on volatility and betting on like near misses and like pretty catastrophic disasters that aren't quite doom make, make a lot of sense in that world. And I just think, I'm not sure whether

1:44:57that's true. I think a lot of the ways in which things go badly are just like things go extremely well in the market just all the way until they go really badly. And then the only situation where you cash in is when you're dead. And I think, I think that's the least convincing part of his argument to me, this idea that like, no, no, you'll get all these near misses and they'll already also like all an expectation created the stock market. And I think there is just a very reasonable like doomer view that concentrates basically all of the probability mass of doom of like things going well all the way until doom. In fact, if you look at a lot of the sort of canonical like doom scenarios and like people who have talked about like serious like existential risk and catastrophic

1:45:29risk, many of them give the scenario of like everything's like looks like it's going really well and strategically great and economically great and looks kind of weird, but it also looks economically great just until the point where there's like the takeover happens or the big incident happened. I think that's the main problem with that. And I, I mean, I think also just like many people just have coherent worldviews that they also just don't bet on and that they just don't take financial bets on even if they're committed to them. I think that's a sort of like more boring meta contention to that. But yeah, I think my main contention is just like, well, I don't think it's a smooth distribution of probabilities. I think a lot of this is just like either this goes very well or it

1:46:02goes very badly. And there is not really a lot of world where it's like volatile and goes kind of badly for a while and then kind of well for a while. And so I think I also don't have a good trading strategy, sadly. Yeah. Okay. Well, I appreciate you for thinking it through. Do you maintain a P Doom number? I'm sure you've been asked many times. I haven't heard you ask, though. Do you have an answer? I just think it depends so much on what you include in Doom. If it's human extinction, it's very, very low. If it is, if it includes like all of the sort of catastrophically risky scenarios, including gradual, like sort of like the end stage of like gradual

1:46:36disempowerment and stable heteritarianism and that kind of thing, I think it's probably around something like 10%. But I think that really is only if you account for the sort of like political Doom outcomes in the broadest sense. And I think my probability of like technical extinction is substantially lower than that. So you're counting in Doom, like we live in a sort of Chinese plus plus state and life is pretty good, but we like don't have political freedom.

1:47:10Yeah, I think like, I mean, maybe life isn't even particularly good in a lot of consumer levels. It's like extremely disempowered, extremely low human agency, extremely low human economic participation, call it the permanent underclass if you must. But like, I think in the actual permanent way, right, I think like you shouldn't like include too many just like all prosely sort of bad outcomes in the Doom number. But I think the sort of old portfolio of like existential, like long-term risks used to include things like stable authoritarianism and like stable economic disempowerment. And to the extent that that's actually like a locked in path for the human

1:47:43future that you can't see a conceivable break out of, I think I'd include that in Doom in the broader sense. And then yes, but not just like, oh, well, the economy kind of sucks. So my pre-Doom is like very high because I really think the economy is going to suck. Yeah. Okay. Maybe

Robotics and labor market disruption

1:47:58something I should have asked earlier, but I think I know the answer. Everything you're projecting assumes that like robotics really works. It doesn't necessarily have to be humanoid, but like we're going to get highly flexible robotics that can be deployed in all sorts of contexts. Yeah. Okay. Yeah. I think, I mean, it's going to take a little bit longer than I think. I think I don't see like a super crazy industrial explosion very soon, but I think eventually this is an engineering problem. This is a scaling problem. And at some point we're going to scale it. And at some point we're going to resolve the physical bottlenecks. It's going to take longer. So physical bottlenecks are going to matter longer than software bottlenecks,

1:48:31for example, that eventually they seem eminently resolvable. So does that mean like, let's say 2030 is the over under for when people start to have domestic service robots in their homes? Would you take the over or the under? Sure. People start to have in 2030. Yeah. I think that sounds roughly right. I think it might take a little bit longer than, than that just because of like idiosyncratic, psychological and political resistance. But in terms of technical maturity, that sounds about right to me.

1:49:02Yeah. Okay. How does all this change as compute goes to space? Well, it just puts, so I think there's two versions of compute going to compute going to space. The first is space is one of the places where we can put compute. I think that's going to be the case in 2029, which is going to be like, there are going to be some data center setups that are going to be worth putting in space. Inference more so than training, for example. And I think it's also just going to be gated by launch capacity. So we can't put all of our compute starting in 29 in space. We might also conceivably have different chip supply chains for chips that are suitable for going in

1:49:36space and racks that are suitable for going in space and terrestrial chip deployments. So some compute goes to space in 29, I think that changes some things and concentrates some more computing power effectively within US jurisdiction. It makes SpaceX AI a lot more powerful. It makes launch site governance a little bit more relevant. These are all like interesting marginal shifts in how the conversation moves. That's part one. Part two is what happens if every marginal chip goes to space instead of to any terrestrial data center. And there's basically no terrestrial competition for data centers anymore. And I think things get a lot crazier then.

1:50:09I think anti-satellite weapons become a really important part of like deterrence and geopolitical stability for one. Because that's the only way you can threaten the deployment of like a super intelligence system because it is in space. And if you can't shoot down the satellites, then then good luck, you're not stopping the super intelligence. In much the same way that the AI 2040 plan sort of talks about making like data centers sort of bombable and visible and allowing for this sort of intervention into this kind of sabotage. In the same way in that world, we just want satellites to be hitable from the ground and the compute to be vulnerable for geopolitical

1:50:41stability reasons. Another effect is all the control concentrate. Do you think that's the default scenario? Like my understanding is we could probably shoot down satellites without too much trouble. We just don't really, but we can, right? I mean, yeah. I think the question is who can? Like the United States? Yeah. Other countries? Perhaps not. And I think they need to develop the capacity to do that. I think there are ground. China probably. I would think China has it. There is some other, like the French have the beginnings of a program. The Indians have the beginnings of a program.

1:51:12There are already programs. It's not that no one can, but it is in the same way that nuclear power needs to have second strike capability. I think there is a sort of geopolitical stability sense in which like a sort of sovereign nation might want to have anti-satellite capacity and not all of them do just yet. And the other question about this is this like sort of crazy Kessler syndrome conversation where like there is some sort of, some amount of mutual deterrence of like ever shooting down satellites. Because if you get to the point where you have like that much debris in space that keeps creating more debris because things keep colliding and so on, it's just going

1:51:44to be really difficult to launch anything into space at any point in the future. So insofar as everyone is kind of disincentivized from doing that in the same way that everyone is kind of disincentivized from like creating nuclear winter or something, I think there's also, that also makes the satellite weaponry math a little bit more difficult. I think the other part of this goes back to our middle power conversation, right? We talked about these compute for access deals and like deploying data centers and so on. That all hinges on this idea of the US is interested in building data centers in other countries because it wants to build data centers somewhere. If the US builds all its data centers in space instead, then the incentive of putting the data centers into the host countries is

1:52:17just so much lower. And I think as a result, that does sound pretty bad for a lot of these compute-based strategies. So I think the most actionable and meaningful consequence of the prospect of data descent is going into space is the compute for access and the compute build-out strategy that middle powers are starting to pursue has a time limit. It stops working at some point in the somewhat near future. And you should start thinking about what your end game beyond the compute thing is in case the space thing works out as sort of as like space takes a matching systems. A more terrestrial concern. Why? I mean, we talked a little bit

1:52:53about like bottlenecks. So I don't mean, you know, and human kind of inertia around why adoption hasn't happened as much as it obviously could in theory have happened so far. But if you take the flip side of that, and you just look at like the AIs and their capabilities, clearly, there's something that is missing relative to, you know, the experience of hiring a human to do work, right? It feels like that's thinner and thinner all the time, almost to the point where I'm like now having a hard time putting my finger on like, what is it about like Fable 5.1 or Astra that's actually worse than hiring a human? Do you have an answer that is

1:53:32like, what that is? And, you know, what, what additional like marginal capability gain you might expect to actually create labor market disruption? Yeah, I'm not sure it's capability gains at this point. I think it's integrating with proprietary data, it's integrating with proprietary workflows. I think, obviously, not the entire task profile suite of humans is currently covered by models. But I think, in specific tasks, they're better at humans. And I think you can drop them into specific task profiles, at least, especially in software engineering, I think, and also in some other like, sort of like general white collar activities.

And I think the question is just like, the labor market just takes its time to rearrange around that, and to get to the sort of like, augmented, like sort of mutually beneficial, because you can't just like, you can't just fire the guy who's sitting at his, at his desk, and like, plug in Astra instead, you need to be slightly more sophisticated. And well, we now need like, instead of three guys, we need like one guy who tells the agent to do the things that the agents do the tasks.

And but that guy has to be like, a little bit better at all the things that the agents can't do. And

1:54:32So I think it requires some institutional organizational reconfiguration. And I think that just will take some time. But I think the capabilities are there, that I would expect them to have this sort of not displacing, but at least disruptive impact that changes how teams are built and how productive they are. I'm a little bit less sure that this specifically leads to displacement in the short term. I think it also does create additional demand and does create additional like, things that human workers can do on the margins. But I think, in terms of disruptive effects and reconfiguring effects, I would, I would, agree with the capabilities are there. And it's just, it's just latency and lag and bottlenecks and frictions.

1:55:07What do you think would happen in a hypothetical world where Tesla decides to license its full self-driving and within 18 months or so, you know, let's say we make a priority of it. So we're into a little bit of a fictional scenario here, but all of a sudden, like, basically all the cars drive themselves and the four or so million Americans who make their living driving don't, are needed to drive anymore. That seems like one, you know, pretty clear displacement story that very well could happen. Like, do you think the economy can absorb those people? Where do they go? It seems really tough when you actually get down to like, okay, this dude has driven a truck for 25 years. He's not ready to retire, but the truck now drives itself. What happens to him?

1:55:50Well, I think part of it is going to be political responses, wage insurances, reducing their times. I think there's also going to be a political necessity to add some frictions to this happening. I think, frankly, there would be human in the loop laws, there would be sort of like, you have to have a driver in the, like, the sort of like Harley and like New York and whatever idea of like, well, even if the thing drives autonomously, there still needs to be a driver in the seat and whatever. I think we'd see a lot of these political reactions and frictions introduced before anything happens. Yeah, that's one part. I think the other part is, yes, eventually, I think the economy would probably be able to absorb at least a decent percentage of that.

1:56:26Not necessarily in better jobs, not necessarily in better paying jobs, probably in worse jobs. But also, I will point out that in that specific story, you found one of the like, very few jobs that clearly just have like one specific task to them. And no mutually sort of synergetic way of engaging with like, sort of, you should like fear it in like the one technology like one to one replaces like a specific kind of worker. I think most automation and like AI specifically just isn't like that, in that it just like gets at specific tasks and leaves other parts of the task profile up so that it lends itself to a sort of, at least like in the medium term, almost like augmented coexisting structure in a way that like specifically driving isn't like.

1:57:05But yeah, I think that will be very hard to absorb. We'd see a lot of political frictions as a result. We'd have to cushion a lot of it, which is like social spending. And I think some of them would find jobs, most of the jobs would be worse. Your idea, not your idea, but your reference to the idea that somebody might be required to sit in the car, even as the car drives itself, reminds me actually of a novel that my dad wrote about a pretty dystopian, but highly AI enabled future where everybody is kind of out of work, but they need the dignity of work.

1:57:37So they're required to show up and stand around all day. And they're known as standers in his imagination. Yeah, I just really hope we don't get there. And I think that's like, yeah, yeah, well, the fingers crossed. But I think, you know, you could make the cynical observation that some jobs in the real world already are kind of like this. But yeah, we'll hope we don't get there. A few bullshit jobs out there. But if people are still with us two hours in, they'll be interested in your thoughts on this. What do you think America should be looking around the world to learn?

1:58:09One thing that I came away from China really thinking about is there's a lot of upside to surveillance. Don't want to have it for a lot of different reasons. But I'm kind of like, geez, it really sucks to leave all that upside on the table. Is there such a thing as surveillance with American characteristics? I'm interested if you have a thought on that. And also just like what other things, when you look around the world, do you feel like the U.S. not just like should envy, because I don't think we can like copy the high-speed train from China. But what should we actually be trying to import and realize our version of?

1:58:42So I think on surveillance, it's really hard for a specific reason, which is I think I'm just very worried about perfect enforcement of laws. I think the American laws specifically just aren't made to be nearly perfectly enforced. If you enforced every law on the books in America, I think this would just be a draconian oversight regime. I think I understand people's motivation of like, well, this is like massive, like perfectly disincentivized, any sort of illegal action. But man, a bunch of things are illegal. And I think like a lot of punishments and a lot of the sort of like criminal codes are like specifically structured around the idea that you catch one of every like 100 or like 10 or 1,000 criminals, and then the deterrence is calibrated to that.

1:59:23The downside of surveillance, to my mind, and I think more broadly, the downside to like more AI integration, I think the most well-taken point in the flock debate as well, is we're just not, legal systems aren't set up for perfect enforcement. And just like surveillance maxing, even with American characteristics, is just such a clear pathway to perfect enforcement of laws that were never meant to be perfectly enforced. So maybe we can talk about that once we've like completely sort of revamped the entirety of the criminal code and whatnot, and the practices of enforcement around that.

1:59:54Before that, I'd be extremely worried about going that far.

Trade offs and importing institutions

1:59:57What to import from the rest of the world? America is a very idiosyncratic and like very specific place that is like done very well with like a very weird balance of institutions and economic activity that have for some reason served it very well. And I think usually when American politicians look around the world into Europe specifically, and they try to import like one very specific part of a society, they sort of underrate how much of that society is just like in a completely different sort of equilibrium and balance. That's just like, if you just transported like specifically welfare spending from the Nordics, I think that, you know, you'd have to do things with the tax system that disincentivizes a lot of other commercial activity that then in itself just like forecloses other avenues of like providing the same services.

2:00:37And I think the same thing is true for the political system where like, yes, there is something to be said for the stability of a party and parliamentary system that isn't quite as, that doesn't swing back quite as much between administrations. But then you also get a much less decisive government that is able to do much less and that gets paralyzed into gridlock a lot more. And I think there are countries whose concepts and structures are wholesale, perhaps preferable in the way that they deliver services and outcomes for their citizenry to America, perhaps not. I think that's a tricky conversation. But I think I basically don't know a lot of like high level, really good things about how things work in a very like structural, like political level in any other place in the world that you could import into America without disrupting a broader part of how this country works.

2:01:24So I'd be very skeptical of like doing that sort of piecemeal. Does that mean we're living in the best America? Because it sure doesn't feel like we're living up to our potential in that ways. I think it's Pareto optimal in some ways. And I think that's different from the best, right? But I think like everything that you would improve would come with a trade-off against something else. And I think I'm very skeptical of this notion that there is, there are like clear Pareto improvements to the way that America starts. I mean, you can scroll through the like Institute for Progress's website on like policy interventions and you'll probably find like 10 like marginal fixes to laws on the books that would just make this clearly better.

2:01:58And some of them are inspired by other countries. But I think meaningful changes to how the country runs are probably not Pareto improvements, but are like tricky trade-offs that I'm not sure would sort of on balance make the country run better. Which is not to say that like this is the sort of shining city or any other country is the shining city. It's just that, man, it's all trade-offs and it's all really difficult to get right. And I'm just not so sure that there are like easy fixes too much of anything. Yeah. Last big question. And we've talked about a lot of different challenges, obviously, and, you know, vexing conundrums of all kinds, conundra, conundrums.

2:02:34What would you say are like the most important needles that we need to thread?

Threading the needle and the future

2:02:39You know, what are the top couple few things that you think are kind of absolutely most critical? And then I'd love to hear just what do you think life is going to look like on the other side of this in your kind of 90% where it goes well? Like, why, you know, for people that are like, this whole AI thing, why do we even do it? Isn't it just stupid? Like, paint the upside picture to inspire that audience. Yeah, I think I'm just not going to fix the problem of the AI story, of the AI lab's like big narrative concerns. I think I'm just, I just hope we can kind of continue on the sort of positive trend that I think history has been on for the longest time.

2:03:17I don't think I take that much of a sort of fatalistic view about the current or like the past trajectory of society. I don't think we need AGI to bail us out of much of anything. I just think it's the next thing we do. It's the next thing there is. It's the next big innovation. It's the next like innovation that we'll need to stay on track with these sort of like compounding economic growth that we have. I think in a lot of ways, it's going to look very crazy. I think some of it is going to involve space. Some of it is going to involve robots. Some of it is going to involve a lot of automation. But I think some of it is also just going to involve like fairly prosaic future economic growth that just makes us all a little bit richer and a little bit wealthier.

2:03:49And our institutions a little bit more functional by the day, just as they have been, at least since the start of the Industrial Revolution. I think that's my main hope. And I don't want to, I don't dream much bigger than that. I think things might get a lot crazier than that. And then we'll have to find ways to deal with that. But I think that is my hope and my dream for the future in a decade or in two decades or in five years, if things go very fast. And in terms of what do we have to do for that, I think just like genuinely just muddle through, like make sure that the balance of power works out and continues to work out and the balance of wealth continues to work out well.

2:04:22Make sure that the labs don't pull away in terms of power and control from the U.S. government. Make sure the U.S. government doesn't centralize and control the entire sort of like flow through of intelligence through the world. Make sure that some other countries have a stake in this, both economic and in terms of just like power and influence and leverage. And just, you know, whenever things seem to go off the rails in terms of there's too much power amassing in one place and things look like they're going wrong, pull it back a little bit again. Keep it on course. I think that's what I want to do.

2:04:53And I think it'll just be like a long exercise of doing things like that on the very small margins. And I think then we're probably going to be fine. Well, that is an unreasonably reasonable worldview. And I appreciate you for spending a couple hours sharing it with me today. This really, I think, has been an excellent conversation. So anything else you want to leave people with before we break? No, I think I enjoyed it very much. Thank you so much for having me. It was great. And I liked. Thank you for being part of the Cognitive Revolution. Thank you so much. Thank you so much.

2:05:53Thank you.

2:06:23Thank you.

2:06:53Thank you.

2:07:23Thank you. Thank you.

2:07:53Thank you. Thank you.

2:08:23Thank you. Thank you.

2:08:53If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution, the Cognitive Revolution, the Cognitive Revolution, the Cognitive Revolution. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more.

2:09:24We're produced by AI podcasting. We're produced by AI podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AIpodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.