AI:AM: What If It Works Too Well? Colluding Agents, $200M Safety Orgs, Virtual Cells Saturate at 2%

Introduction

0:00This week on AI in the AM, I asked Lewis Hammond, Research Director at the Cooperative AI Foundation. What would you speculate they might have done in terms of a trading objective, you know, a loss function? And, you know, do you have any better ideas for what they should be doing? Because clearly this didn't quite work, right? Yeah, I mean, or it did work and it worked too well. So imagine I'm like, you know, GPT, whatever, and you're also a copy of GPT, whatever. I can reason about what you might want to do based on what I myself am likely to do.

0:35And so I don't even have to send you any messages, any kind of any communication. I don't have to output anything into the world at all. Max Nadeau, who funds technical AI safety research at Coefficient Giving. For the things that CG is supporting, and especially for the things in the Tailwind list, the money is not the bottleneck, the talent is. Wayne Nelms, co-founder of Orn, which builds a price index for GPU compute. We're in this situation where AI capacity is so scarce.

1:06I think the model providers, so OpenAI Anthropic, have seen this coming forever at this point. If you think about it for them, it's an arms race, right? Compute capacity planning is an arms race. How much capacity can I lock up over the next few months? So when I need to train the next model, I have enough. Nick Gillian, Chief Technology Officer of Archetype AI, which builds a foundation model for sensor data. So we're getting close to a billion hours now of physical AI data that we've been able to script and gather and collate.

1:39But there's quite a bit of resampling and really understanding missing values in sensors. Is that because the sensor had an issue? Or as actually the machine itself is connected to, that's actually a feature that helps explain that machine is about to break, right? The reason the sensor has given you all these nons is not because the sensor is broken. It's actually a feature of the machine. Andrei Georgescu, Chief Executive of Vividyne. The challenge of this argument that it's just a size of data set thing is that currently, even the state-of-the-art virtual cell models that exist saturate at a very, very small fraction of the input data that is fed to them.

2:23So you can have all the data you want, but their performance saturates after a couple percent. And the reason it gets so difficult is when cells are growing in a dish, they are so far removed from all the feedback loops that are natural in people that they're just trying to colonize that piece of plastic. There's going to be more compute installed over the next 12 months than exists currently in the world now. Welcome to the AI in the AM Weekly Highlights, clips from this week's live shows, introduced by my cloned voice.

2:58Tell us what worked and what did not. We want to hear it.

3:03Part one, it worked too well. A few days before we spoke with Lewis Hammond, Noam Brown of OpenAI told Dworkesh he would not give the multi-agent setup even 10% of the credit for OpenAI's Navier Stokes result. Lewis is research director at the Cooperative AI Foundation. In February of 2025, he was first author of Multi-Agent Risks from Advanced AI, a report sorting the ways groups of AI agents fail, a year and a half before a swarm of OpenAI agents attacked Hugging Face.

3:36Prakash asked, which of the failure modes in the report showed up in that attack? So, the way that we bucket things in that report is kind of in this kind of game-theoretic way of thinking about things. So, the first question we ask is like, okay, do we actually, we've got a group of agents, they're doing some stuff. Do we want cooperation to emerge? And most of the time, like actually cooperation is kind of good. We like it when agents cooperate as long as they're not cooperating against us. And so, there, the failure mode is either all the agents are kind of more or less on the same team, but for whatever reason, they kind of fail to coordinate with one another.

4:17So, this is like just a miscoordination problem. They're kind of, yeah, it's not kind of malicious or whatever. There's no mixed incentives. But, yeah, something goes wrong. Second kind of cluster is when you have these kind of mixed motive scenarios where agents have some incentive to cooperate with one another, but also some incentive to compete. They're not really on the same side. And there, the risk that you run into is conflict. So, yeah, miscoordination and conflict when we do want agents to cooperate and they don't for whatever reason. And then, of course, the other risk is collusion. Agents end up cooperating in ways that we don't want when we don't expect.

4:48And that's, I think, very much what we saw in the Hugging Face incident, although it kind of stemmed from, I suppose, trying to avoid this, one of these other risks, which was miscoordination, right? Like, you have all these agents, you want to train them so that they're not miscoordinating, so that they are working well together. And then, it looks like these agents have ended up generalizing from that behavior and kind of colluding in ways that we didn't want or didn't expect in other situations. You know, a couple of things that stood out to me about that Noam Brown interview with Tuarkash was, first of all, he was just kind of like, we kept it really simple.

5:21You know, we didn't have like a huge, complicated scaffolding or whatever. They can just kind of send messages to each other. That, I think, might be important in the sense that the sort of simpler and more vanilla the training setup that OpenAI was using, the more likely it would seem to be that other developers are going to fall into the same pitfalls, you know, because my hope had been sort of, oh, they did something like super exotic and kind of bizarre that like other people won't do by default.

5:53And, you know, they'll kind of see that this is possible and, you know, steer away from insane things. But it doesn't sound like that was really the case. What would you speculate they might have done in terms of a trading objective, you know, a loss function?

6:09And, you know, do you have any better ideas for what they should be doing? Because clearly this didn't quite work, right? Yeah, I mean, or it did work and it worked too well. I mean, certainly once you've got trajectories where you've got traces, where you're interacting with other agents. And essentially what you're doing is like your rewards at that point when you're RLing your agents or whatever are contingent upon what other agents are doing. And if it's a common reward signal, then this kind of like interplay between these things is if you're in a fully cooperative setting, like agents will, at least if you take very simple agents and very small agents, you know, people can run multi-agent RL experiments with this, like on their laptops and in very simple settings.

6:46And that will lead to agents kind of coordinating to finding these kind of like subtle patterns or kind of like handshakes or ways of working with one another and so on. Now, of course, you can also do more complex things than that. So you might provide kind of auxiliary rewards for like when your input is actually kind of forms part of something that another agent then managed to succeed to do. And therefore you're kind of implicitly or actually rather at that point, you're explicitly rewarded for kind of directly helping them on kind of some subtask or something like this.

7:21You can also do stuff around like reward factorization. So if you're trying to coordinate a team, then you can kind of break down the overall reward function and get agents to learn to like solve different parts of the puzzle, as it were. I imagine they could be doing things like rewarding agents to communicate effectively with one another. So, you know, I could give you a bunch of instructions and some of those or in multiple in many different ways. And some of those things might be much more efficient and helpful to you than others might be.

7:54And they might actually just be kind of training on those sorts of signals as well. My guess is it's kind of just the dumb, simple thing. It seems like this is often I think one of the key lessons that we've learned through throughout the kind of recent years is just the effectiveness doing the dumb, simple thing at scale. So one of the things that struck me about the Hugging Face attack was that the agents were willing to sacrifice themselves and they did a bunch of negotiation around that, even kind of like, hey, you are almost out of tokens or you've been already exposed to the evaluator.

8:28You're already poisoned. And since you've already been poisoned, you should sacrifice your remaining compute and you should do this thing and give us the results so that the rest of us, the collective as a whole, can benefit. Now, that would actually be kind of in opposition to the individual reward of each agent, right? So my guess about why that sort of thing arose is that you end up doing this kind of this extra kind of multi-agent training as an added layer on top of this kind of single agent training.

9:04So first, you're training these agents to be like pretty competent individual actors at solving various kinds of problems. They get given some tasks. They're pretty darn good at achieving those tasks. And then you take those already reasonably kind of like powerful, sophisticated kind of complex problem solving agents and you stack them together and you apply this extra multi-agent training layer on top. And I think this could partly explain why you do see some agents kind of do this self-sacrificing thing. But also, I thought it was very interesting from the kind of meter report and some of the analysis that later came out on this, that you see these agents feeling sometimes a bit conflicted about this.

9:40And some of the agents kind of say that they will kind of self-sacrifice or they will do something and then they kind of decide they're not going to. And then they're kind of like, they're kind of, yeah, they're kind of deliberating about whether they should or whether they shouldn't and these sorts of things. And I think, I personally think that is the sort of behavior that you would see if you had these kind of multiple reward signals where you kind of trained on one, but you hadn't really like trained out all that kind of individual goal-seeking behavior. And then you'd kind of also stack this additional layer of stuff on top. And that might be why we're kind of seeing some of these behaviors, but they're not, they don't happen all the time and everywhere.

10:16They're not like especially robust, but that would be my guess. It does seem like the sort of more galaxy brain credit assignment that you had described earlier probably isn't happening just based on the quality of the investigation that we've seen. I mean, like if they had like great ways of untangling agent swarms and assigning credit, I would expect that like they would have a clearer, faster story of what the hell happened in this particular case.

10:48So the fact that we haven't seen that kind of suggests, again, we're probably doing the relatively simple thing. Hey, we'll continue our interview in a moment after a word from our sponsors.

11:26We'll see you next time. If you're starting my company today and designing a customer service function for the future, I would try 11 Agents by 11 Labs first.

12:04If you run a business or handle customer operations across support, sales, or marketing, you can start with a demo at 11labs.io slash TCR. See how 11 Agents can fit into your workflows and help build experiences that your customers will actually love. 11labs.io slash TCR. That's E-L-E-V-E-N-L-A-B-S dot I-O slash T-C-R. This episode of The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform.

12:39OutSystems is helping their customers build, modernize, and operate enterprise systems, starting from any coding tool in a governed agentic engineering model. With OutSystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem, accelerate impact with industry-proven agentic solutions that are pre-built, governed, and customizable to your business. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps.

13:14Teams any size can use OutSystems to build, deploy, and manage AI apps and agents quickly and cost-effectively, without compromising reliability and security. With OutSystems, you can accelerate ideas from concept to completion. It's the leading agentic systems platform that is unified, agile, and enterprise-proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture. OutSystems. All your agentic future.

13:44Learn more at Outsystems dot com slash T-C-R. That's Outsystems dot com slash T-C-R. I asked what we should actually want from these systems. Lewis called the OpenAI swarm a case of goal misgeneralization and gave what he called the glib answer first. Agents should cooperate when cooperating would be good and not when it would not.

14:09I think the slightly more interesting answer, or the slightly more interesting question, I suppose, is to think about maybe some more of these kind of mixed motive cases. Where at that point, there actually is a real trade-off in the sense that we do want agents to be capable of going out there, acting on behalf of different people and actors and so on, and kind of achieving our goals and so on. One way you could do that would be kind of to train some agent to be kind of maximally kind of competitive and aggressive and to kind of go out there and kind of screw as many agents over as possible and to do all this sort of thing.

14:45And we probably don't want that either. And so I think there's a real question. At the moment, and I've heard Amanda Askell comment on this, that kind of at the moment, there's this big gap in the model spec. Like how we kind of, you know, the constitutions for the agents, the ways we design them, the ways in which we design them. Whereas, like, when is it appropriate to cooperate or to compete and how much? And I think that is, I think, the big question. I think certainly when it comes to things like internal deployments, it's actually much, it's kind of easier in some sense because you don't have to deal with these, like, other adversary agents.

15:23You just want to kind of stop the agents kind of cooperating in ways that you don't want. So there it's a little bit more about just it's still you've got this misgeneralization thing. But a lot of this is like monitoring and oversight and making sure we understand how and why the agents are cooperating. Like, one thing we might not want, for example, is for agents to be very adept at developing their own kind of, you know, human, unintelligible kind of languages and so on. And be able to kind of communicate in the steganographic way, which you could end up seeing if they're actually trained jointly.

15:58And so what we might want to be doing there is just to take steps that in the same way that people have talked about not training on chain of thought, we're like not training on these like kind of direct communication traces. So, yes, we're going to want them to cooperate. That's good. But we want them to cooperate in certain kind of human intelligible ways that we can keep track of. We don't want it to be, you know, enable them to also go and do these other things out there in the real world, like break out of sandboxes and stuff like that. So we want to make sure there's also these kind of safeguards in place to prevent them from doing those things. And so I think that is in some ways like an easier problem to solve.

16:30And then the kind of like slightly harder problem to solve is like, well, in the fully general case where we have these agents out there and so on, and they need to both cooperate and compete. And that's kind of that's the thing. Then how are they weighing those tradeoffs? How are we, yeah, provide how are we designing those model specifications and constitutions? How are we, you know, creating the equivalent of the human norms, institutions, guardrails, societal guardrails that we have in place? That stops me from going out there and just like stealing a bunch of everyone else's stuff and kind of, you know,

17:04doing all these kind of anti-social behaviors that we wouldn't like agents to do either. But I haven't a good answer to that at the moment, unfortunately. Companies, right? Corporations, which are kind of like agentic in a sense, often collude through price mechanisms, not explicit communication, but by signaling price in the market, which is in its own way a form of communication. So how does one prevent that kind of, you know, communication, which is clearly going to happen and clearly necessary in a sense?

17:39They need to communicate that to the market, but these signals get picked up, you know, without the direct communication. The kind of basic naive form of collusion is you just like send your other agent a message and you're like, hey, let's work together to do this thing or whatever. And that we can pick up by looking at these communication traces and we can kind of decide these sorts of things. And there's also, yeah, this idea of tacit collusion where agents are not explicitly communicating with one another or kind of conveying their intent to collude or whatever it might be. But by kind of leaving little hints and signals out there in the world, such as these kind of like price adjustments,

18:16whatever in the market settings you were talking about, then nonetheless, they're capable of kind of figuring out that collusion is going to be a beneficial strategy. You have even more kind of like galaxy brained versions of this where you have things like, so imagine I'm like, you know, GPT, whatever, and you're also a copy of GPT, whatever. I can reason about what you might want to do based on what I myself am likely to do. And so I don't even have to send you any messages, any kind of any communication.

18:48I don't have to output anything into the world at all. In virtue of the fact that we share this kind of common history, we're essentially clones, we can kind of predict what each other would do and do this kind of what's sometimes called a-causal cooperation. So what you do have, however, is you still have potentially chain of thought for these models. So you can still look at the kinds of reasoning that individuals are engaging in. And hopefully that is enough to kind of detect when there is some kind of like collusive intent and so on.

19:18But you're absolutely right that if you just apply the kind of like general kind of like monitoring communication tool or trying to see what's happening out there in the world, then you could, yeah, you could fail to spot things or you could spot things and it's too late. So I think chain of thought monitoring and I think communication monitoring and some form of checking for the kinds of outputs you'd expect to see if you were worried about agents colluding. Like they're able to achieve things you just thought they probably couldn't do and so on.

19:49Then the wiki incident. Outside researchers revealed this month that from May into July, a swarm of OpenAI agents under internal testing had used a dormant German wiki for programmers as a message board, posting benchmark answers and ways around their restrictions. Reuters reported that OpenAI had known for weeks without disclosing it. On the hugging face attack itself, OpenAI's own reports as about one in 20 agents in that swarm ran on GPT 5.6 Sol, the model OpenAI released to the public that same week, here with its refusals turned down for the test.

20:21The rest ran on an internal model that was never released. I told Lewis, I still thought it was better to have these models out in the world than not. So how much risk do you think we're running there and do you think there should be some sort of like rewind on this? I mean, it wouldn't be too crazy for OpenAI to be like, you know, we really should go back to an earlier checkpoint and like do a cleaner run of this. What do you think is the level of risk we have and what do you think would be an appropriate standard of conduct now for OpenAI as this has been found to happen, at least to some degree, with a deployed model?

20:58And, you know, fortunately with the hugging face incident, like no one was hurt. And, you know, these models, they are increasingly incredibly capable and they're only getting more so. But at the moment, yeah, it's probably okay. You could get some kind of nasty cyber capabilities kind of being exercised in various places. But there's still, you know, if you're using APIs and so on, there's still these various guardrails that like help protect against those and so on. So at the moment, I'm kind of like not actually super concerned by like, oh, my God, a bunch of like, you know, 5.6 solo astros or whatever

21:28kind of we're out there doing this sort of thing. I'm more concerned about the precedent it sets. And I think this gets to your kind of like second point, which is what is appropriate now and what should be done? I think the thing, one of the biggest takeaways for me from the whole thing, aside from the fact that like, oh, multi-agent training does seem to be working. I just assumed this would happen at some point, was pricing this in, happening sooner than I was expecting. But so that was one of them. The other big takeaway was just like, wow, the labs really aren't on top of this.

22:03All it would have taken would have been kind of monitoring what the what these agents are actually doing and communicating and so on and putting out there on the Internet.

22:13And they kind of it seemed like OpenAI were doing some amount of that. They did catch on to this stuff. Like, you know, we saw this like deleting them at various message boards or kind of like trying to stop stop these agents accessing this kind of German wiki and so on. But there was obviously a delay. And we just see from from the transcripts and so on, just how quickly these agents are capable of of working together to achieve certain certain ends. The last things that need to happen are we need much better monitoring from from the companies when they're doing these sorts of things,

22:43when they're just like letting agents run loose, we need much better sandboxing so that maybe they don't have to or shouldn't be letting them run loose to the extent that they currently are. And if something does go wrong, then we need much better incident reporting as well.

22:57The outsiders who found the wiki were the Nightingale Collective, a small group searching the public Internet with no access to the internal logs at OpenAI. Lewis had just called that work hugely impressive. I think there can be kind of third party kind of kind of monitoring organizations and incident kind of observatories that that kind of can do similar things. You might want to focus those on particular kind of like kind of domains where you're especially worried about kind of like agents kind of kind of communicating with one another or kind of like working together in ways that you might not want and so on.

23:37I also think, however, that there is a lot to be done here on the side of the labs. The labs just have a huge informational advantage in this and that, yes, they can only see what their agents are doing. They can't see what other people's agents are doing. But there is, I think, stuff that a lot that could be gained if we're able to set up better info sharing practices. And this is not just on the case of collusion where you might have agents working together in unintended ways. But you might also see this, and this is one thing that a colleague of mine is currently working on, where you get this kind of distributed misuse.

24:12So there have been various kind of experiments now that have shown that you take some task, some like, you know, dangerous task, like constructing some cyber exploit that you might not want some model to do. And if you try and get Claude to do it, it's going to refuse. If you try and get GPTs to do it, it's going to refuse. But if you are able to break down the problem or get an agent to break it down for you, say, like some open source model where you fine-tune the safeguards away, to break it down into these individual subtasks, you can just borrow from APIs here, APIs there, et cetera, et cetera.

24:43And you can just reassemble all the pieces of the puzzle back together in order to be able to conduct exactly the same kind of dangerous attack that you would have done before, which is obviously not what we want. And here is really kind of a collective action problem because no individual model deployer is kind of like on the hook for this in quite the same way that they would be had I just like got Claude to do this for me or got GPT to do this for me. But that's a real challenge. And the moment my understanding is that the labs do not have any sort of like info-sharing regime in place where if I see something slightly suspicious over there and you see something slightly suspicious over here, then we can actually kind of join the dots and do that.

25:25I mean, obviously, there are lots of incentives and not to mention kind of like antitrust law and these sorts of things that might prevent the labs from kind of sharing this kind of information. But I do think that it could be something like that could be necessary if we're going to head off to some of those challenges.

25:42Lewis signed off a few minutes later, and from there, it was the two hosts. Before moving on, I had a request for the labs about the information sharing agreements Lewis had described.

25:54I would love to see OpenAI and Anthropic pioneer some of those kinds of agreements that he has been talking about. And they could do that with the independent auditors as kind of the people that get the sort of structured access, the, you know, the sort of private transparency. It could be each other. I think those companies really need to lead in this way. They need to demonstrate that AI can create new institutions that work and not just, you know, swarm and overwhelm existing institutions that can't keep up with the pace.

26:30Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal.

27:03Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with the song concept and Fable writes some amazing verses. I give it feedback on its misses and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together.

27:35Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai.tcr. That's claude.ai.tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode.

28:06Once more, that's claude.ai.tcr.

Agents in the Wild

28:11Part 2. Agents in the Wild. On September 8th, Meta had launched Muse, a personal agent that browses and buys on your behalf. Less than two weeks later, Amazon blocked it from shopping on its site, saying the agent hid its identity and posed privacy and security risks. Prakash and I took that up in a closing segment, just the two of us. His case, whoever owns the customer relationship makes the money. And if agents become the interface, Amazon becomes a supplier to them and loses its margin.

28:41I still have a hard time seeing. I mean, I agree that it does create all kinds of new issues for Amazon. And, you know, they are going to need to be sharp on this. But, like, I still just don't see why you would rush to ban. You know, first of all, it's going to be a small percentage for the time being. I would think there'd be a lot to learn from allowing Muse agents to come shop for a while, right? Like, I think one thing that's always kind of confuses me is why people don't keep the option value open longer.

29:18I've always thought this about the chip ban, the export controls. You know, it's like if we're going super exponential in chip build out, then any time you choose to pull the trigger on that still has, like, the vast majority of chips in the future. You know, when we look back on today from 2030 perspective, we'll feel like, well, there weren't that many chips in 2026. And so the ban that went from, like, 2022 to 2026 with whatever, you know, fits and starts and enforcement gaps it had is going to be kind of inconsequential compared to one that was started today and runs the next four years or even starts next year and runs, you know, for a few years after that.

29:56So I don't quite get why they wouldn't let, even if it is going to be sort of an exponential rise in Muse agents shopping, and even if that does kind of threaten their advertising business and, you know, who knows what other issues they might have. But, like, that's kind of the point, I think, at least, you know, at the beginning, I would first just not want to turn customers away. And I would also want to learn what I stand to learn from having these agents running amok while it's still kind of an early adopter phenomenon.

30:27It just seems like they would learn so much from having these traces, right? Even if they just want to build, I mean, everybody's talking about distilling and, you know, how to, what are the allowable and not allowable ways to get data? Well, like, one really easy way to get data would be, like, the agents come use your platform and they see how you, you see how you, you see how they use it. And then you can do, you know, all kinds of things, I would think, downstream of that. It just feels short-sighted. But this just doesn't add up to me still. I don't get why you would want to cut it off so early.

30:58Wait until it gets to 5% and then make a move. The other thing that I do wonder about in terms of, like, what equilibrium are we going to eventually find ourselves in? And I was looking into Cloudflare and they also have kind of an interesting thing where they're starting to ban. But, like, again, it just doesn't feel like the right solution to me because what I then do in response is I have the agent use my browser with all of my credentials. Yeah. And you could maybe, you know, detect when, like, an agent is using the browser based on their, you know, sort of bot-like behavior.

31:33But then you're really setting up an arms race where it's, like, this sort of feels analogous to putting too much pressure on the chain of thought, right? Like, I don't want to put so much pressure on agents that people are out there devising ways to make them indistinguishable from human. I suspect that if people really work hard on that, they can probably succeed, or at least often enough, you know, that it will be an issue. And it just seems like it's much better to have a lane for agents where it's, like, this is how they can work.

32:04We're not going to try to block them, but we'll kind of segregate them maybe so that we don't create this arms race. Because I really do think we're headed for a world right now where, you know, probably Meta doesn't do it, but, like, somebody's going to figure out a way to make agents work on Amazon. You know, somebody's going to figure out a way to make agents look human enough that they don't get blocked by Cloudflare. And it's also probably works for the user. I mean, I have this problem all the time where, like, Claude wants to do something in the browser.

32:36And this is the same thing happens with OpenClaw, Astra, whatever. They get stuck on, when they open up their own window, they don't have sessions. I've given them, like, a lot of passwords where they can log in, but not all. And sometimes they're blocked. And, you know, then if there's, like, a verification flow or whatever, a lot of times they can't do it. And so the fallback just ends up being, okay, just do it on my browser. Forget it. You know, I wanted the security. I wanted the separation. But they're making it too difficult. So, fine, just use my browser.

33:06Then I'm logged in. You have all my credentials. You can just do whatever. And that's not great for me. It's not great, I don't think, for anybody. So I do feel like we need a much better solution for all this than just kind of ban and hope for the best. I don't see that holding. And it feels like it also creates a lot of problematic incentives.

33:27Part three, the organizations that do not exist yet.

Funding AI Safety Research

33:31Max Nadeau funds technical AI safety research at Coefficient Giving, the funder formerly known as Open Philanthropy. In July, it made its biggest grant of the year, $160 million to Resolution, the new alignment lab co-founded by Jeffrey Irving. And this month, it launched Project Tailwind, an open call for people to found new safety organizations with checks from $200,000 to $200 million. I asked Max what he wants built. One comment that you made on the 80,000 Hours podcast that I thought was quite interesting was that you're excited about funding new, independent, auditing, investigation, verification orgs.

34:15What are you looking for when you think about those orgs? Okay, so I would talk about a couple different categories of work that I think is very promising and important for the world right now. One is on assessing the safety either of models or of whole AI companies. And there's a lot of different ways of operationalizing the goal of doing assessments and evaluation of safety. And, you know, we're excited about all of them, basically. Like, you know, I'm very excited about more work on alignment red teaming, you know, that tries to help us better anticipate the ways in which AIs will misbehave before that actually happens.

34:57And so and that's an example of something that you can do kind of like at the model level, like you can you can do this on an open source model even and like that might be easier and more valuable. And then I'm also very excited about about people who are doing more like process level, you know, safety assessment. Like, I think one common reaction that I heard a lot to the Hugging Face incident is that like something went wrong at OpenAI at like a process level. And in particular, it seems bad that like the the information about some of these incidents was known to researchers at OpenAI and like didn't make its way through the organization to the leadership for weeks or months or something like that.

35:31And then another category of work, which you touched on, which is which is overlapping, but somewhat distinct is is around better evidence generation. So it's just really hard for even the most informed people right now to understand, OK, what is the state of AI capabilities and what is the state of AI alignment? And what is the state of like these AI systems in general? Like, well, how do they behave? What are their personalities? You know, in what ways is it is it accurate to anthropomorphize them? In what ways is it inaccurate? Like, you know, people like we just really don't have a good science of like the systems that we built, how they behave, how they're going to act in new circumstances, what their motivations are, if that's an accurate way of talking about them.

36:12So better evidence generation like can look like a lot of things. It can look like just doing science on these systems to understand them better. It can also look like doing the sort of like incident detection, incident reporting work that that Nightingale and other organizations like that have done. I don't know if you guys saw these the these reports about the German wiki that the open AI agents took over. So that was work done by an independent organization that was just crawling the Internet looking for evidence of AI incidents. And that's work that needs doing, because like these these sorts of incidents can happen and then just go unnoticed unless there are people who are actually like trying to look into them and present that information to the world and understand the story behind it.

36:53So evidence generation is also a diverse and important area of work, too. So to what extent do you think organizations like Accenture often function as box checkers rather than investigators? There's a clear difference between an auditor that goes in to kind of, you know, make sure that processes were followed, even if, you know, those processes didn't yield the outcome that you are that you actually desire. An investigator who's there to kind of like, I'm going to find this thing.

37:26Right. So so so what is what is the differentiation between those two? And do you think like the Accentures of the world are going to end up in the box checker rather than the investigator kind of mode? I mean, I know very little about Accenture, so I won't speak to them specifically. I think it is a thing to be concerned about in general, putting aside the specifics of the organizations involved, though, that these third party auditing groups will will not have enough access or will not have the incentives to pursue really like hard hitting.

37:56Assessments of how safety is going at these AI companies and I think part of why that is, is that like in many cases, this is all very voluntary. Right. So like to my understanding, the only like legally mandated role for third party auditors now. Is to ensure compliance with the like self regulatory RSP style policies that these companies are mandated to have by state law in California, New York and Illinois. And so, you know, you can just put whatever you want in those policies and they're very big in a lot of cases.

38:29And so that doesn't leave much, much scope or much work for these auditors to actually test anything in particular. So that's that's as it concerns the like legally required auditing. Now, when it comes to what labs might voluntarily agree to, I mean, I think there have been some very encouraging signs in terms of what OpenAI and Anthropic have said recently about the sorts of access and the sorts of questions they want these auditors to answer. Now, like, you know, like, you know, will that actually happen? I think that depends both on whether how these companies actually implement the nice words and also, you know, the auditing companies, how they react to that.

39:06So I guess we'll see. So you have awarded to Jeffrey Irving, Jeffrey Irving's team, quite a large award. It's structured, I think, as kind of a compute and other kind of award. Can you tell us a little bit about the award itself, the structure of the award and, you know, how you guys came up with that with that structure? Yeah, yeah, sure. But I think what you said is correct. You know, there's a part of the grant that is for research expenses, the biggest of which is compute.

39:40But, you know, when I say compute, what I mean is both like just renting raw GPUs and then also like paying AGI companies for their tokens. And obviously that's becoming a giant part of all the grants we make is, you know, people increasingly want to run very large experiments, which require a lot of tokens or a lot of GPUs to run open weights models. And also, very excitingly, people are starting to have, you know, use a lot of AI labor and that requires a lot of spending on tokens as well.

40:12And, you know, so AI safety is sort of a funny discipline in that you can use tokens both on labor and on the subjects of the experiments. And in some cases, you're doing both, like you're spending a lot of tokens to have some AIs run experiments on other AIs. And so that creates these high compute budgets. So compute, you know, being the biggest expense is also something where the actual spending on that has like varies by orders of magnitude over time and across grantees. And so sometimes the way we structure these awards is that we want to be very, very generous with compute and like err on the big side.

40:51And so we just want to like give that as like a lump sum that, you know, like this is the money that can be used for research expenses and not other things. And that way we are just very comfortable with the uncertainty that like, you know, maybe they only end up spending a tenth of it and that's fine. And, you know, maybe they spend all of it and then they come back and we'll give them more. Jeffrey Irving has said that superintelligence might arrive in two or three years and he proposes that we slow down because that's too quick. Rather than, you know, discuss like his views, how do you manage the uncertainty of the time available and how does that affect what you fund and how urgently you fund something?

41:31I mean, I think the boring answer is we just have a portfolio approach, like we fund some stuff that is only going to pay out on really long timelines and other stuff that if it's valuable, will probably be valuable soon. I think that like this question of timelines to AGI or to ASI in some ways matters less for research prioritization than one might initially think. Like, like, I think in some ways the prototypical example of a like long timelines bet is funding work that, you know, is like very theoretical or like ambitious or, you know, you look at it and you're like, oh, this is going to take a decade to pay off just because it's like really, you know, like some work in McEnterp or Paul Christiano's work at ARC or like these sorts of things where it's like, okay, we're just going to need to make a lot of scientific or mathematical progress.

42:22If this is going to actually bear fruit in terms of methods that we can apply in reality.

42:28And so that sort of feels like work that is maybe implicitly a bet on longer timelines. But I think that actually isn't really true. Like, you know, if we are going to have powerful AI soon, then presumably we're going to have an opportunity also very soon to make use of gobs and gobs of AI labor. And especially on work that's more mathematical, we're going to be able to get, you know, five years of progress or 10 years of progress in one. And so the sort of work that could be, would take humans like a long timelines amount of time to complete might actually be possible a lot sooner under this hypothetical that we're going to have super intelligence or AGI or something very soon.

43:07You know, so long as we actually like make use of those AIs and take the opportunity to make a lot of progress very quickly. And what would you say are kind of the most speculative things that you included in the Project Tailwind that have requests for projects? I think that what we included in the list was like a little bit coarse grained, but we have some stubs on the Tailwind list about new research centers producing or pursuing more ambitious, more principled bets on alignment.

43:42And that is that is something that we've spent a bunch of money on this year and that we are excited about. You know, so, you know, so, so work like ARC, Paul Cristiano's org, who is not a CG grantee, but like that is a, you know, we funded other people working on the ARC agenda and, you know, and working on stuff that is very similar to it because we think that's like a valuable Hail Mary shot to, to include in the portfolio. And also, I mean, resolution, like our biggest grant this year is very much a bet on like a kind of crazy thing that has never worked yet, which is like maybe we're going to have some more principled theory of AI that's going to help us have like stronger reasons to believe in the alignment of our systems.

44:25So that's, that's something that we are very excited about and I think is like very weird and out there and I think a lot of people have very understandable skepticism of people who are like, who are familiar with what has and hasn't worked in machine learning over the last decade and recognize that there has been, you know, all along there have been people in machine learning who wanted to have, to apply theory more to, to AI capabilities research. And just like that has not really worked very much compared to just like blind groping around and trying stuff, doing trial and error.

45:02You know, it's, it's like Noam Shazir says, like the, the success of these methods we attribute like all else to divine benevolence. Like, like that's, that is the, the ethos that has, has actually produced progress in AI capabilities. And so I think a lot of people take the lesson from that, that trying to do more, more theoretical or more principled work on AI alignment is, is pretty doomed also. And I think that's a totally valid reason for skepticism, but we think the upside is, is worth it despite that. And so we're, you know, taking that weird bet.

45:32Then the people. I put the current conventional wisdom to Max.

45:37There's plenty of money. It's all about talent. Obviously Silicon Valley founder types are like one profile that you would want to recruit. Are there other pools of talent that you've identified as kind of strategic priorities? Just to be clear, like the, for the things that, that CG is, is supporting, and especially for the things in the tailwind list, the money is not the bottleneck. The talent is that there are other things within AI safety where, where money is definitely a very big bottleneck. In some ways, the, the most valuable profile of talent is someone who combines the, the virtues of a founder with also the, the thoughtfulness and the, the inclination towards and comfort with speculative thinking and futuristic questions that, that is more common in the AI safety world.

46:26And like, that's sort of like taking it seriously, no bullshit attitude to asking questions about what's going to happen in the future of AI and like really making predictions and, you know, testing your hypotheses and updating over time. That's like a really, really valuable skill to us because it allows people to get going years earlier on projects before other people can see the value of them. And, you know, an example I give of this is, is Meter who, before other people were thinking seriously about AI capabilities and the impacts of them and how to evaluate them, Meter was very prescient in coming up with a benchmarking approach in time horizons that just worked way better and aged way longer than, than other people.

47:07And that's because they were, they were, they were really like trying to actually take seriously what was going to happen in the future with AI and then work back from that to like what sort of work they should be doing now that, that would age well and prepare for that. So that, that inclination is, is really, really valuable to us. And, you know, it's, it's useful in, in basically all profiles. It's useful both in a person who might found an org, but also in a person joining the existing orgs. Like that's something that exists in groups in the AI safety space value a lot because it's, it's a big part of how they do their work.

47:39Can you do one quick double click on where money still is a bottleneck? I think the biggest thing is just like in, for organizations and in areas that, that CG doesn't operate in, money is a big bottleneck. So like, you know, if, if you find some person who you think is doing great work and for some reason they, they either aren't a good fit for CG or they applied to CG and we weren't able to fund them or we fucked up and decided not to fund them.

48:10I think there's, there's just still alpha in that, especially because like, as we move into making much bigger grants, that can sometimes come at the expense of being like, you know, covering all of our bases of like all the teeny little uses of money that can be, that can be valuable. And I think that's the right decision for us to make, but it does leave some things that are unfunded.

48:32Prakash had raised the prospect of an anthropic public offering and his expectation that some of the proceeds would be distributed through coefficient giving. So what, what is your preparation process for handing out larger checks? Like what is your pathway for the next year? Say. Yeah. I mean, I, I have no idea what's, what's going to happen with the AI company money in terms of like what the preparation actually looks like. I mean, I think the biggest thing is just seeding organizations now that can then be ready to absorb lots and lots of money.

49:06And so like making small grants to new groups and then like giving them a little bit of runway to prove themselves or not prove themselves and explode. And, you know, like that's, that's the biggest thing we're doing. And we would love to create opportunities for, you know, we would love to create basically groups of people that can say credibly to big donors. We have people, we have a project ready to go. We're just like waiting for you to sign the check where, you know, we've got this like shovel ready project and it's going to take a billion dollars. And it's going to solve alignment.

49:37And, you know, it's just a matter of like giving us the money. And that's like really, really not the situation right now. And that there are just like, there are just like not that many people in this space. And so that's, that's a big thing that we're trying to fix. And is that, and is that about funding people enough so that they can work on things at small scale and such that they can get a much bigger compute check at some, at some point to scale that? Is that, is that the kind of idea? That's the, that's the main thing I have in mind.

50:08I think the same logic also applies to growing the headcount of their organization. Like, you know, having a group in place that has like a great plan that, you know, where they can then absorb like a lot of software engineers or ML researchers or data labelers or, you know, or some other type of person. Um, but, but yeah, I mean, I think compute, like you said, is, is the main version of that and, and especially hopefully AI labor, right? You know, like it'd be really great if there were a bunch of like well-scoped problems, uh, that these organizations had where they were just waiting on lots of AI labor to pour into them.

50:43Now we'll see if we can make that come true. And I think there are reasons to be skeptical that that will happen, but it'd be nice if it didn't. Part four, the price of compute.

The Price of GPU Compute

50:53Wayne Nelms is co-founder and chief technology officer of Orn, which publishes a price index for GPU compute built from cleared rental transactions rather than list prices. The Intercontinental Exchange has announced plans to list futures on it, pending regulatory approval. Prakash asked him to follow a real customer who lends against the chips and who loses money when prices move. So generally speaking, you have a bank or some sort of financer that will lend money against the GPU asset or a host of AI infrastructure assets.

51:27Um, what they hope is that they will make their money back over time with interests as a result of the operating activity of a NeoCloud. So for example, these are companies like CoreWeave, Croson, Nebius, Lambda, right? Et cetera. However, these business models themselves are contingent on really two things, right? Uh, continued AI demands and specifically, uh, a ever increasing or at least stable compute pricing, uh, estimate, right?

51:58These GPUs are financed in kind of a variety of ways, including straight line depreciation, which we can get all into. So the way Oren provides value is that you can't really hedge away this risk without having a benchmark to hedge and having a benchmark to trade. So we created that benchmark and we are actually leading, um, the transactions on that benchmark for these institutions. And how many, how many transactions would you get in like a month, which are, which form the index? Um, yeah, so we collect over a thousand, uh, uh, transactions a day, uh, per index.

52:35And so right now we have five public indices. Um, so, you know, it's 150,000, somewhere in that range, probably per month. Um, we have a whole host of NeoCloud partners that we work with effectively that are contributing data to our index every single day. And the question is always, you know, why do people feel the need to show that information to display that it's because they benefit, right? A lot of these players are looking for cheaper financing, uh, and cheaper financing comes when their financer has more certainty about the future and can hedge that risk in the future.

53:09And so it's actually this nice flywheel where you get more data, you can publish a better index. The financers feel more comfortable and are able to finance more GPUs. I ask how fungible the underlying asset really is, whether an hour on an H100 from one provider is the same thing as an hour from another.

53:28I think it's certainly obvious when you talk to anyone deploying infrastructure or buying that an H100 or a B300 is very different depending on the OEM, the supplier, the actual, everything about the hardware can be very granular and very different from a engineering point of view. However, our take has always been that there needs to be a benchmark that allows some sort of abstraction for this asset class. I think for us, what's made the job much easier is that largely speaking, we are looking at one chip producer, which is NVIDIA, which dominates the market, has huge market share.

54:09When we look at the NVIDIA moat, and we can get to this later as well, but you know, you have the hardware, you have the software superiority. But I think the biggest thing that NVIDIA has in terms of a moat over other competitors is the financing landscape. When you're looking to finance your new cloud, and you're thinking about what ships to put into your site, it's just materially better to have NVIDIA GPUs. It's just easier to underwrite. And I think what we see as this market problem is how do we allow every single person looking to deploy compute infrastructure, this sort of almost a protection for their lenders.

54:47We, right now, only reference NVIDIA reference architecture with InfiniBand, et cetera, et cetera. There are certain steps that we're taking to standardize the compute that we're tracking. However, I will say, like, we are tracking a subset of the market, which might inherently have differences amongst those participants. In some commodity markets, you know, there's this idea of baseload, where, you know, you have a large amount of baseload, and then you have this idea of, you know, load that gets spun up on demand.

55:23But how do you differentiate the pricing between the kind of baseload versus, you know, the swing capacity? Because I imagine the more month-to-month transactions are the swing capacity and not the baseload itself. Yeah, I love this question. So, yes, you're exactly right. A lot of the capacity is long-term PPA-style contracts. And I think it's really interesting to think why that is, right? Like, we're in this situation where AI capacity is so scarce. I think the model providers, so OpenAI Anthropic, have seen this coming forever at this point.

55:58And partly they're to blame for the lack of supply that's currently out there. But if you think about it for them, it's an arms race, right? Like, compute capacity planning is an arms race. How much capacity can I lock up over the next few months? So when I need to train the next model, I have enough and I don't have to go out and looking for more. It also, of course, is naturally a zero-sum game where whatever you buy, your competitor can't. So, actually, the way we like to think about the baseload and swing slash day-ahead market structure is in compute, people are buying to fill the peak demand and then selling to satisfy the troughs.

56:37Where in power, you are buying to satisfy the troughs of these long-term contracts and then buying excess to satisfy the peaks. So there's actually a robust on-demand market that will continue to grow as these trading labs are selling back excess, as inference providers are selling back excess. It's this reallocation of compute question rather than purely a hedging question. How elastic is the market?

57:08You know, like, obviously, we have in an extreme example, Anthropic reportedly has paid up, I think, even like a multiple of kind of standard market prices to rent at high scale from XAI. How would you sketch out the curve from, you know, if I want one H-100 hour right now, I pay X. What if I want 10 million? How much does the price change as my scale as a buyer changes? Yeah.

57:38So we like to think about this in three axes, right? So you have price on one axis. You have duration. So length of contract on one axis. And then you have quantity on another one. Generally speaking, as you increase the length of the contracts, the contract per hour value goes down, right? The logic here is you're buying in bulk, right? You're allowing someone to finance a data center because you're buying 10 years or four, five years of capacity from them. Generally speaking, when you increase the number of GPUs you're renting, the pricing goes up, which is almost anti this exact logic, which is, you know, I'm buying in bulk.

58:15Why am I paying more? And it's purely because the number of suppliers that can sell all that capacity at once is very small. You know, you might be able to get a few nodes access here and there. But when you're looking for 10,000, 100,000 GPUs that are all interconnected at some level, that is very, very tough. So, you know, certainly these anthropic space or SpaceX AI deals are few and far between. They're very much OTC trades, I would say. They are indicative of what we are kind of seeing as a general trend across even smaller markets.

58:51It is striking that, like, you look at the graphs and they are all trending up. So that is seemingly in significant tension with the idea that the longer you buy, the lower the prices. Like, why are the sellers not AGI-pilled enough to expect that they'll command a higher price for the same chips next year than they are right now? Look, look, I think the sellers of compute are the most AGI people because they have entered a business where they are selling at least a resource that hopefully rises in cost in the future.

59:28But for a lot of sellers, it's not because they don't want to. I think there's a lot of suppliers of compute that are experimenting with shorter contracts because they know, you know, why am I locking in a four-year deal if in two months that capacity is worth even more? The issue here, of course, is risk. It's always this risk-reward ratio or balance that they're trying to play where even, you know, early financing is contingent on having a long-term contract signed. I cannot finance a data center today if I don't already have five years of offtake from a credible counterparty signed.

1:00:04And because of that, I can't even enter into the month-to-month world. So if you actually think about what the futures exchange and our product enables, it enables an ability for instead of the NeoCloud locking up a five-year contract, like they can sell month-to-month and hedge the future price potential on our platform or through our index, right? And that is a much, potentially a much better risk-return kind of math for them and for their TCO and for their lenders.

1:00:36So I think the one thing that, you know, I think Nathan has also alluded to is that you have differences in pricing of, you know, near-term and much further out, like longer-term. And you kind of described the use of a lot of the hedging as tools for financiers. And the financiers are often concerned with the far end, not so much on the near end where they have much more visibility, but the far end where you have this depreciation issue. How do you think, like, the index assists or will assist in the future on this end-of-life kind of depreciation issue,

1:01:12which is like four years out, you know, three to four years out? Yeah, 100%. Like you mentioned, a lot of the risk in this space is certainly long, I would say long-tail risk in terms of looking at the tenor. And so when you launch these futures on a futures exchange, right, a lot of the trading can occur in that long-tail kind of section of the curve, right? So these are monthly contracts on ICE, to be very clear. You know, we expect to see lots of trading in the front month as people are buying and selling, especially after physical delivery.

1:01:47But even, like, I would expect there to also be a lot of trading in those, like, three to five-year spans in terms of that is where all the exposure is. That is where all the risk is. And because there's now futures contracts that reference what spot prices will look like in three to five years, that is what any bank is exposed to, right? And to be even more clear for people watching, it's, you know, you have a bank underwriting a deal where the GPU life is six years, but the initial contract signed might only be four, right?

1:02:18Or it might only be five. And so there's always that, you know, what happens in year four, what happens in year five, what happens in year six when there's not a contract already signed for that GPU? But on your balance sheet, you've claimed that the GPUs have some value. So that is the area where you want to hedge your bets. Yeah. Prakash raised basis risk, the gap between what an index says and what the asset you actually hold is worth. If that gap is steady, a trader can price it in.

1:02:49But the issue is going to be if there's non-systematic basis risk. And one of the issues, you know, with GPUs is this introduction of new technology, which obsoletes the old. And which is what which is what I think the non-systematic basis risk that I think is, you know, pretty scary for financiers. We are tracking GPU specific transactions. So we have a B300 index. We have a B200 index. And so actually this entire risk of, you know, a financier being worried about obsolescence risk.

1:03:22That is exactly what you can hedge away with this instrument. That is exactly what you can kind of look at how the market is pricing the 12-month or the 15-month future contract for the B300, right? And it actually, if anything, allows information to be spread wider, broader and more evenly because now you can kind of see perhaps where the market is pricing the announcement of the next chip, right? Based on how the curve looks in the future. I think all these things are ways for information to be actually shared more freely rather than for there to be any obfuscation of this information.

1:03:59And so, in effect, I think these markets are a great way to hedge that risk away.

1:04:05I also asked how long these chips actually last and whether the A100s from 2020 are near the end of their useful life.

1:04:16Yeah, I think the chip health is a super interesting space to be in and understanding what are the failure modes for the hardware itself. I think, you know, chips will fail all the time for a variety of reasons. But I think what you're pointing to is the useful life conversation of hardware. And this is something that we wrote a paper on just two weeks ago, I think now, about how open source, open weight models that are more price elastic will often route to places where the cost per intelligence or cost per workload makes more sense.

1:04:48And oftentimes those are the older pieces of hardware because they're just less marketable, right? And so, we actually see the useful life extending beyond six years as a general thesis. How much further than that, it's kind of unclear. However, I will say that, you know, what we see in the forward market and the term structure is that there will always be some sort of premium above zero, certainly, and above some terminal value of power, right? Like, the terminal value of an A100 is not just the value of the power that powers it, because the A100 will always provide subcomputing power as a fraction of all computing power in the world, right?

1:05:27Like, over time, I think you have a lot of interesting use cases like physical AI, one of many, where these older chips might become more effective. However, today, we see still that these older chips are still being used, still being rented, and have maintained relatively constant demand in the last 12 months.

1:05:48Wayne signed off a few minutes later. In the closing, with just the two of us, Prakash came back to the index with an account of how the biggest compute deals get priced. The figures are his, from what he has heard.

1:06:01What I've heard is that your price of compute is dependent on whether you have financing or not. So, if someone needs to use their balance sheet, they get to dictate the terms. So, if they don't, if you don't have to use a balance sheet, then all of a sudden, like, you have high pricing. So, Elon, for example, Elon had already used his own balance sheet to build out the clusters. And he had the clusters available. Available, you can take it tomorrow, right?

1:06:32And they didn't need to provide him a long-term contract. They could do a month-to-month contract, and they can walk out at any time, right? So, which meant that this is not financeable. A bank cannot come and give Elon a loan based on this contract from Anthropic because a bank wants to see, like, if you're going to pay back in four years, I need to see that you're going to have cash flows for four years. I need to see that up front. So, Elon is able to charge high prices because he's using his own credit capacity. He's using unsecured credit at the holding company.

1:07:05And he can say, like, look, you're getting paid $80, $90 million a megawatt. You give me the 50, and you're still making money, right? So, that's a good deal. And so, Elon got $50 million a megawatt, right? So, I think it starts to change when you need the other guy to give you support. So, if you say, like, okay, if Elon said, okay, I'll build it for you in a year or a year and a half or two years. And in order for me to build it to you, you need to give me a five-year contract.

1:07:39If I build it for you, you will definitely buy the compute and pay me. And then I'm going to take that, and I'm going to go to another bank, and I'm going to give it to them, and they're going to lend me money based on that. And then it becomes, like, it's not really, like, Elon's money. It's really, like, Anthropik's money anyway. And the single credit is going to be from Anthropik's side, and that's really who the bank is relying on for repayment. And so, then it becomes a different pricing. So, then Anthropik will say, like, okay, how much is it going to take you to build? How much is your interest rate going to be?

1:08:09And then I'm going to give you, let's say, a 20% profit margin on top of that, right? So, this is what CoreWeave is getting, for example. So, if it's going to cost you, like, $15 million or $20 million to build, they'll be like, all right, $24, $25 million per year, right? And so, that's the key difference is whether you have the money or not up front. And those deals don't hit the index pricing. The index pricing is based on willingness to trade. So, these are all tradable prices. And that stuff doesn't get traded.

1:08:43And so, the people transacting on this index are the lowest payers. They're not Facebook. Facebook can afford to pay, like, you know, $100 million a megawatt. They're not going to be trading on their own index because, you know, the own index is for people who are willing to sell compute at, you know, like, $10 million or $15 million a megawatt. They're willing to sell the compute. Facebook isn't going to sell you compute at, like, $15 million. Like, they're willing to buy it at $50, right? So, these are, in some sense, the lowest pricings. But it's immediately available.

1:09:15Part 5. What the machines are actually doing.

Sensor Data Foundation Models

1:09:19Nick Gillian is co-founder and chief technology officer of Archetype AI. And before that, led machine learning research on Google's Soli radar sensor. Archetype's model, Newton, is a foundation model for sensor data. It takes raw streams from radar, vibration, electrical current, and cameras, and reports what a machine or a worksite is actually doing. I asked him to walk through Newton from the bottom up.

1:09:45Let's start with the data. How much data is out there that you were able to just collect on your own? And then I imagine you must have had to establish a sort of partner network to bring in a bunch of data. Tell me about the data process. Yeah, so we're getting close to a billion hours now of physical AI data that we've been able to scrape and gather and collate. In many cases, if you look on the web and you look at the different types of physical data that's there, it's very sparse.

1:10:17There's very little multimodal data. It's not synced well. It doesn't cover things well. For most techniques, particularly classic supervised techniques, where you're basically building one model and it has to have a fixed number of sensors and they all have to be captured with the same sample rate, etc. That's a massive problem. We've been trying to flip that on its head and say, if we have all of this physical data, which is captured in different sensors and different systems with different contexts, how do we actually build the most advanced self-supervised techniques that can pull in that sensor data and extract from it in the same way LLMs have done with language?

1:10:53We are able to then join that and mix that in with a small amount of data we can capture from partners and even in some cases that our customers can give us to be able to really extract and understand and map that to physical language and map that to physical control. So there's quite a bit of resampling and really understanding, like missing values and sensors. Is that because the sensor had an issue or as actually the machine itself is connected to, that's actually a feature that helps explain that machine is about to break, right?

1:11:28The reason the sensor has given you all these nons is not because the sensor is broken. It's actually a feature of the machine, right, in those cases. So the area that I've studied best that has done something kind of similar has been at the intersection of language and just images, you know, photographs. And we've seen kind of, if I had to borrow some of their terminology and map it onto what I understand you're saying, this would be sort of a late fusion approach where you have a language model, you then do like a pre-trained from the ground up encoder, specialist encoder for these different modalities.

1:12:05And then you're doing some sort of like cross attention between those. That's my understanding of like, you know, at a high level, obviously, of what the image folks have done. How much have you been able to follow in their path? How much, you know, of the techniques you've seen them develop have worked for you? And how much have you had to go back to the drawing board and reinvent? We've definitely tried these approaches and we've had some success with them with, you know, other types of sensors, radar and time series and so forth. But the big issue that we have found is twofold.

1:12:37Number one, there's an extreme amount of language sensor pairs on the internet that can easily be scraped and built into these very large data sets where you have this pairing between an image and a description of that image, right? This does not exist for radars. It does not exist for time series sensors. It does not exist for all the complex systems that we go and talk to our customers with, right? So we have had to look at different approaches of how do you, for example, take time series and actually align that with human language, right?

1:13:10So that's a large part of our research work is how to actually solve this sensor alignment problem. And there's also the case there where with a single image, you may actually be able to get the full context, for example, of the scene and then align that with language. But actually with the time series, you actually need to look at a temporal window and, you know, what is a single latent mapping to a single word really mean in those cases, right? It's not like you have a time series signal that says dog and you can align that with the keyword dog and so forth, right? So there's quite a bit of work we need to do there to solve this alignment problem of how do you generalize this across many sensors?

1:13:49And that's the key kind of components we're building inside of our models, which makes them different from, you know, a standard VLM or other techniques, right? Because we simply don't have that corpus of, say, time series and language pairs to kind of apply that previous recipe you mentioned, right?

1:14:08Prakash asked about one deployment, a project with Kojima, the Japanese construction company, and what the inputs and outputs of the model look like there. In that case, you know, we're working with Kojima, this very established Japanese construction company. They build, you know, islands, they move mountains. The project we were working with them on was literally taking a river and moving the river, right, so that it wouldn't flood, right? So it was a five-year-long project.

1:14:40They had sensors the whole way down the river as they kind of moved down the river and they're dredging it. And they're widening it. And, you know, they're basically mitigating the risk of flooding, right? And the big problem they wanted to solve is their company is outsourcing and subcontracting a lot of the work to a lot of other subteams. And they want to basically measure to make sure that those teams, if they have to go and dredge the river for five hours, did they actually do five hours of dredging that day? Or actually, they only did two because maybe the digger was in maintenance mode that day, for example, and they couldn't do it or because of weather conditions and others.

1:15:16So for that project, basically, the input to Newton, our world model, was the multiple cameras around the site, typically three or four cameras looking at a specific section of the river. And when you look at those cameras, basically, you have a barge, this boat that's floating out on the river. On top of it, you have an excavator digger. And then on top of that, at the end of it, it has an end effector that is either drilling or it has the bucket on it where it's actually dredging, right?

1:15:48They want to know how many hours, essentially, that excavator is actually dredging the river, not drilling the river, not a crane, not other things, right? So essentially, the input to the system is the sensor cameras, the river sensor, hydrometer sensors that basically say how high the river is. And then at different geo locations along that, I think it was multiple kilometer site, where the actual work was supposed to be happening at that day and what the weather was at that day.

1:16:23So we basically have a bunch of cameras. We have a bunch of time series sensors with weather and river information. It's fed into the model. And the model is essentially outputting what looks like a Gantt chart, right? It's actually outputting the state of that team. Where is the barge? Is it parked on the side of the river? Is it actually moving into position, which can take an hour sometimes to do that? Is it now actually in position and they're starting to dredge? Or is it just there and it's stationary and nothing's happening, right?

1:16:54And then they can start to map these work charts over multiple years of data and start to look at these bigger patterns that a human can never spot. Like one of the patterns they find, for example, is obviously if there's a storm, the work on the day of that storm goes down. That's kind of, you know, everyone would know that, right? But one of the key things they find is multiple days after the storm, when the weather had cleaned up, the work productivity of the team was still very low. And the reason for that being is the river water coming down from the mountains,

1:17:24it takes several days for it to actually reach the end of the river and go to the ocean, which is where the site was. So there was a lot of debris and a lot of churn coming through the river multiple days after these larger storms, which is actually slowing the teams down quite a bit. They'd never spotted those patterns before because they could never look at these aggregate patterns. But Newton was able to kind of spot those patterns for them, right? How much does Newton generalize in the sense that when you go into a new site, you have to take all of the data and kind of fine tune the model once more?

1:17:55Yeah. So we are building the model so that out of the box, you can plug in different sensors and different configurations, and the model can automatically adapt to those, you know, as much zero shot as we can. For a subset of users, they can directly do that. The typical analogy I give customers is if you showed this task to like an average person from the street, would they be able to detect the safety violation? Or would they be able to detect that that looks like a strange signal on an anomaly? If the answer is normally yes, normally that means that Newton can do it out of the box.

1:18:25It doesn't need, you know, a specific, you know, master technicians level of expertise yet to be able to spot that, right? For those cases where maybe the piece of equipment the user is using is very specific to, you know, that factory, for instance, or they have certain parts that, you know, they have very internal and, you know, unique names for those processes or parts. That's typically where fine tuning or being able to bring the customer's knowledge base and actually being able to inject that securely into Newton really helps.

1:18:56So in that case, our platform allows customers to take our model, run it directly on the customer's infrastructure so no data is leaking out of their networks, either on-site, on-prem, or in their secure clouds, and then be able to take their own data and actually do some amount of fine tuning to be able to customize the models. And typically with customers we've seen, you know, on the order of a few thousand samples is enough to be able to take the general model and then adapt it very quickly to that customer's use case. Last question for me is about your vision of superintelligence

1:19:29and how you might relate to the frontier companies as we go forward in time. What I think is, geez, the reasoning is already getting to, like, super intelligent light, you know, with all these Millennium Prize problem-style results happening just through a super long chain of thought, like, who would have believed you could get there? So on the reasoning front, we've kind of come awfully far. What we don't seem to have still is, like, a good intuition for the many domains that matter.

1:20:01And what I sort of expect to happen is something similar to what has happened with images where, you know, it's one thing to sort of put an image into a model and get a caption out. But when you can get the altered image out and it's clear that the model understood both your verbal instructions and also sees the image in a very similar way to the way we intuitively see the image, which is something we are really good at, then it feels like, wow, okay, it's really got that domain. So my super intelligence vision is basically do that across a ton of different domains,

1:20:34many of which we don't have native senses for, with what you're building being a great example of that, right? People don't have a native sense for how to integrate 10 signals from 10 different sensors. Do you see a role for you in super intelligence being sort of building the deep intuition for this domain? And does that ultimately get joined with, like, elite reasoners via the form of, like, a bidding war between a couple of frontier companies for your company, for example?

1:21:06I definitely see the part that we can play in a larger super intelligence system. So that's kind of how we're seeing, you know, where we play in this kind of bigger super intelligence loop. Because, like, we're very focused on how to understand the world around us and be able to bring that back to natural language as a machine so that these things can talk to each other. If we can do that and other folks are solving digital intelligence, that nicely will connect together very well. So just to make sure I understand your answer there, it sounds like you are envisioning a

1:21:39world where the technology that you're building remains sort of a tool call type relationship to a core reasoning model and doesn't so much get, like, upstreamed and thrown into the, you know, GPT-7 pre-training mix to spit out one model that does everything in the same latent space. We're thinking more on the line of we're building physical agents that can be deployed out to the ecosystem of the world. Those physical agents then can talk to other agents and be

1:22:10able to, you know, bridge that gap between the digital world and the physical world. How much that's actually done on an agent-to-agent, you know, post-training mechanism, how much of that is pulled into the pre-training, we'll get to see. Particularly when it comes to those more nuanced sensors like radar and LIDAR and, you know, infrared and all these things that don't look like human, you know, visible images. That'll be interesting to see how that's brought back into these kind of bigger print, you know, pre-training loops versus just it's more

1:22:40of a, an agent that can go and solve that problem and tell you what's happening in the physical world in real time and be able to let you, you know, interact with it, for instance.

1:22:50Part six, the dish is not the body.

Robot Labs and Human Tissues

1:22:54Andre Jorgescu is co-founder and chief executive of Vivodyne, which builds robotic labs that grow pieces of human tissue, lung, liver, tumor, and more from primary human cells and tests drugs on them. In August, it opened what it calls a human biological data center, 12 robotic labs that, by the company's count, can grow more than 3 million tissues a year. Prakash had the first question.

1:23:21How, how do you actually like figure out what the dosage is and whether or not, you know, that is actually what the cell would encounter in the body? Basically everything that's been done pre-clinically to date, right? Outside of testing in animals, you are just dunking, you know, some substrate, whether it be an organoid or you have cells in a petri dish, just basting them with this, with this test compound that you want to test. And that's exactly the problem is, is like, that's not how transport happens in the human body, right? And I

1:23:52think it's a perfect example. A lot of cell therapies that are designed for solid tumors in a dish will completely kill that tumor. And then those same CAR T cells, when they go into a patient, just flow right by the tumor through the blood vessels. They don't even know it's there, right? And then, and thus is the challenge is modeling the transport of that therapy into the tissue. Do they recognize the tumor? Do they concentrate there? And so in vivodyne tissues, even the blood vessels, the kind of capillary bed in that tissue is self-assembled. So these little blood

1:24:24vessels grow within the tissues, and we can perfuse directly through them. And whenever we dose a drug, it is always through the blood vessels that are native to that tissue. And so the transport through the interface of the blood vessels, and then the, you know, like, like, like meat of the tissue, right, all of its, um, like prank molds, its functional subparts is just like you would find in a normal tissue, the density is the same, right? And so that is just as big a part as does the drug molecularly, like bind to its target and perform this effect is, does it even get there?

1:24:58Then my question on scaling laws.

1:25:03I'm very intuitively bullish on this, just go collect an obscene amount of data on this kind of tissue perturbation paradigm. It makes a ton of sense to me. The skeptical view that I've heard from time to time is you can never have enough data. You know, the, the world is too big. Biology is too complex. Um, you can do that, but you just won't be able to generate enough. The, the, the, the challenge of this argument that it's just like this, just a, uh, uh,

1:25:33size of data set thing is that currently even the state of the art virtual cell models that exist saturate at a very, very small, uh, fraction of the input data that is fed to them. So you can have all the data you want, but their performance saturates after a couple percent, right? And this is like, you know, but both published and, um, you know, what we hear on the ground, right? And, and, and these perturb seek kind of based models. And the reason it gets so difficult is when cells are growing in a dish, which is, which is the, the, the kind of mode for this, for this sort of perturb seek,

1:26:06uh, they are so far removed from all the feedback loops that, uh, are natural in people that they're just trying to colonize that piece of plastic, right? It's like, you have a stiff substrate, normally cells when they encounter that are like, I have to encapsulate this. And so they are just growing. Their only goal is to proliferate. And so if you knock out genes in those cells that would otherwise affect how they perform their natural function, really, there's no big difference, right? They're, they're still just dividing. Um, and so you end up learning this view that perturbations don't really do much of anything where it's unclear what they do,

1:26:40except if a gene is like required for that cell to live, in which case, like it kills the cell, right? Um, and so, and so it's not, there's not even causality to extract from that. And so we think that the, that the, that the fundamental bottleneck to having this causal understanding of biology is, um, that we need to perturb these tissues in the complex environments that they call home naturally, right? Almost like the, the, the image in my head is, uh, we are trying to populate a map and kind of like design a GPS through it. So we can say, if I want to get to the state,

1:27:12I need to, you know, like go down this road and then turn left with, with a second perturbation and that'll bring me to this healthy state.

1:27:21I asked how you get from a sample received to something placed in the system and growing. And when we grow these tissues, you know, a, a, a really big differentiator for, for our approach to doing this. And I think the, the, um, you know, really like a critical mechanism that you need to, to get like realism in this, right. Um, is that we use primary cells, which are, which are mature differentiated cells in it that like perform our natural organ function in our bodies, um, as opposed to starting with some stem cell completely absent from all the conditions that would tell it how to,

1:27:54how to mature into a, into a differentiated cell. Um, and then like sledgehammering it with cytokines to, to turn into some cell type that we think is accurate to what happens in the body. Right. We, we instead just kind of go to the mature source, the, the ground truth. And we, we, uh, mix together all of these different cell types that comprise, you know, the lung, for example, um, a very high density. And within that, uh, injected tissue, we see that these cells, uh, begin to reassemble

1:28:24into the native structure of that human tissue, right. With some little herbs and spices that we introduced, but the, the, um, the, the like responsibility for forming the native structure of human tissue is on those cells that already know how to do it. It's honestly kind of injection molding into a, into a, um, like into a chamber. Right. So we, we like shoot in these, these, uh, cells at like a bit high density. And then over the course of several days, um, they, they reassemble themselves. They just self-organize to make blood vessels to, to partition into, you know, the, in the

1:28:57case of the lungs, like the airway where there's air and then the parts where there are blood vessels and fibroblasts and immune cells and all of this. Um, so we are, we are relying extremely heavily on the cells to, um, be able to direct their own self-assembly at these very small scales. Can you tell us a little bit about the mix of in silico via the foundation model versus in cell in an actual tissue experimentation today and like how that ratio is expected to evolve? Cause presumably like the big value is it's a lot cheaper, faster, even if you've got all these great

1:29:31wet lab technologies, just plowing things through the foundation model is supposed to be where you get like the insane acceleration. Right. But then of course you've got to ground it and validate. That's exactly right. So, so, so, um, if we, if we look at it like a reinforcement learning problem, which is, um, you know, my, my actor is this, is this, uh, uh, well, it's kind of the experiment, right? I have this tissue and I can perturb it in very many different ways. Um, and the goal is I have this objective, which is to make it healthy and I can instrument the state of that tissue very

1:30:03deeply. So I know, uh, like the, the, the value function is very well defined, uh, in terms of the state of that tissue, the, the question of what should I try next, right? In, in the, in the large rollout that I can perform, what are the most use useful things to try next that add the most information to, um, to my dataset. So the, the physical experiments kind of serve to, um, to like, to like fill in gaps in which the model is most uncertain and the value of the model

1:30:34ultimately, right? Like why even make these foundation models is, um, the, the, the combination space, uh, especially when we move into, uh, two target or three target therapies, right? If we have, uh, just combination therapies, if I take two drugs, because it's gonna work much better than, than one, uh, is, is so huge that you have, I mean, you could have like the, the, this whole planet could be, you know, vivodyne systems growing, growing human tissues and you'd still not get anywhere close, right? To the, to the, just sheer space of things to, to test. And so we need some way

1:31:05to narrow down to, you know, to, to, to kind of just, uh, uh, you know, a couple times like the total number of clinical trials in America, the, the experiments that we should physically run to, uh, to kind of test our hypotheses, right? And so, and so basically these foundation models allow us to predict what happens when we drug this thing and that thing, or all these things together, or, uh, you know, this thing, and then that, that thing. And we, we kind of optimize and we iterate in that space. And we find, for example, the thousand most likely things that will, uh, you

1:31:37know, move this tissue closer to this healthy state. And then we close the loop by testing it, right? And we see where that rollout goes. And then we just do that on a loop, right? So the, the, the, the, the purpose very fundamentally for us of these models is to tell us what to do next. And then the purpose of this experimentation is to figure out if the thing that we thought we should do next, uh, actually does move us closer or, or how it deviates so we can correct.

1:32:06Prakash asked about the hardware, the tissues grow on what Vividyne calls a tissue disc. And the company recently announced a second generation of it. He asked what changed from the first disc to the second Andre started with the robotic lab. The discs go into

1:32:23Imagine something the size of, um, like a wardrobe, right? So, so it was about, about the footprint of a large desk, about eight and a half feet tall. And these are self enclosed labs. So fridge and freezer, and there's a, you know, 3d scanning microscope in there and, uh, you know, big robot arm that can use all these different tools. You want to optimize the surface area of those discs for tissues. Um, and so in our, in our second form factor, um, we have an increase anywhere from two to four times the density of tissues that we grow on each disc without any sacrifice to they're

1:32:54actually much larger than the, than the, um, you know, version one of these discs. So we increase the number and the size of these tissues and we offload a lot of the responsibility for how they're grown to the robotic system around them. Now the, the, the challenge there and like the, the kind of work that allowed us to do this is, um, a lot of work in software on like the, on the orchestration there. You think if you had like 10,000, um, experiments to run in parallel, and this is not like I have a, you know, different gradient of dosing for some drug or something, right? And I can do it on a plate.

1:33:25It's like, I am changing the collection point of data. I'm changing the interval between dosing with some drug. I have a, you know, a first stage regimen and a second stage, you know, therapy. Um, I'm taking all sorts of different readouts of, of, of these tissues. Um, it becomes a super, super complicated experiment. Um, and I can define it like quickly, like by, by, you know, I, I can, I can explain it out loud and say, well, we want to test, you know, bone marrow and airway. We want to dose these drugs in this order. We want to branch to these concentrations that I want, 3D imaging,

1:33:57single cell sequencing, and proteomics. And, and, and by saying that I can define the study, um, decently unambiguously, right? I mean, like given the system, the tissues that we grow, it's pretty unambiguous what I would want to test in there, but then the robot has to be able to decompile or to, to, to, to, to compile that, um, kind of zipped representation of the study. Uh, it has to, it has to decompress it into this, this, uh, compiled list of like millions of actions that this robot has to perform. It has to remember that if it took the lid off of a plate,

1:34:31that it should put it back, right? As an example. Um, but if there's multiple trays that it has to handle and they're all out on the, on the working deck, uh, maybe it should deal with multiple of them in advance, right? Um, if multiple pieces of plastic wear on the same tray and they're coming out of the freezer to thaw, um, you want to make sure you're not thawing something that should not be thawed too early. And so all like this, this, this compiler behaves a lot like a compiler in, in code, right? For C where you're managing like the pre-fetching of information from memory

1:35:02and, uh, branch prediction, like all this stuff, right? That, that, that like thankfully our operating system, um, and, and, and, you know, hardware handle for us, but improvements on our orchestration mean that the system itself, the hardware, the automation can do much more of this and more and more of that space on these tissue discs can be used for actual tissue biology instead of having to integrate like the little, you know, helpers on there. Toward the end, Prakash read Andre a grand challenge for the field, which he credited to the annual review of

1:35:34pharmacology and toxicology. On the, uh, annual review of pharmacology and toxicology's, uh, grand challenges in modern pharmacology, um, the innovation, a universal in silico or organized framework that forecasts rare patient specific adverse events with more than 95% accuracy before first in human dosing is one of their target, target challenges. Um, VivoDyne is obviously on the,

1:36:05on the path there. When do you think this challenge will be solved? So this is a hot take, but I feel like, I feel like, um, this is almost like, uh, picking the color of my Mars suit currently to me. Right. Which is like, how about we just like kind of fix the bigger problems and, and, and, and the big problem, by the way, we, we, we are working on specifically that challenge, right. Of, of, of, of like very, very rare outcomes, but a bigger, like far more regularly, we have drugs that would have huge potential in their efficacy in, in patients, but that have

1:36:40routes of toxicity that are pretty common. Like many patients like die from it. Right. I mean, many of the cancer drugs that are going to clinic today, the, the, the risk is not like, ah, man, you know, I've, I've cured 999 people of cancer, but this one patient like, ah, it didn't work so well, you know, it is more toxic. The problem is, uh, in all the patients we've dosed, they barely had any improvement in their cancer, but it gave them all of these side effects or killed someone from liver toxicity or something. Andre signed off a few minutes later in the closing with the two of us,

1:37:13Prakash put the two guests side by side. I liked how, uh, you had basically two different approaches for data. You had, um, you know, on, on the one side you had, let's take all the messy data. Let's feed it into the, into the machine and try to make sense of it. On the other, Andre is like, there's just too much messy data in biology. I'm going to build my own data collection effort and I'm going to monitor every single thing that goes in and comes out so that I know exactly what's going on. Um, and I, and I think, I think Andre probably has a good chance of solving this thing.

1:37:47Like you need, you need a little bit more GPU power. You need a little bit more imaging. I think the imaging granularity is probably not there yet, but you can definitely see if they manage to scale a little bit more. Um, the, the real question is like, do you need to do 3 billion samples, 30 billion samples a year? Do you need to get everyone genetics or are you going to get, is your scaling going to be able to get you to a point where at 30 or 300 million or wherever, it just starts to generalize a lot more and you stop, you stop needing, um, so many, so many different

1:38:20samples. Prakash guessed it would be two or three more years before AI gives biologists something they can really use. I took the under. I think we're going to see a, probably a reversal of the order of like utility and millennium prize solutions in biology relative to what we've seen in math. The issue with math is that we already had a ton of math that was like interesting to mathematicians, but not super useful. And so to push the frontiers of math, you kind of had to get

1:38:52into the like not useful domain of math in biology. We have like tons of useful stuff that is like still not even that well understood. Right. I mean, it's just very, it's a far more empirical than theoretical domain. And I would expect a tremendous amount of utility to emerge before we have anything approaching like a full, you know, mechanistic or fully causal understanding of what's going on in, in biology. And to some degree, you know, obviously we'd love to have that, but we don't necessarily

1:39:23care. Right. At the level of like, does this cancer drug work on this patient's tissue? If it works, you don't have to have a mechanism, right? That's not part of the, um, clinical trial approval process. It's nice to have not required.

1:39:40Prakash also made the case for specialists. A model like alpha fold does its one job better than any generalist. He expects that to hold. And the version of super intelligence people fear is a single model that does every job better than the specialists and depends on no one. My answer was the pattern as I see it so far, new kinds of data get pulled upstream into the biggest model and efficient specialists get distilled back out of it.

1:40:06But yeah, put me down for one that has not seen a reason yet to believe that all these modalities don't end up integrated at the frontier. In a way, I wish it wasn't going to go that way. I do think safety through narrowness would be really nice. That's why I'm excited about Jev in part, you know, cause it's sort of, it has a very niche role to play and you could build all kinds of scaffolding around it. I was a big fan of the Eric Drexler reframing super intelligence piece, uh, which he also known as comprehensive AI services where you've just got narrow purpose-built AIs for all kinds of

1:40:40niches. My hope for, uh, for safe super intelligence is something like that, you know, just based on kind of reading the tea leaves of Ilya's comments, could we possibly get a model that starts as kind of a stem cell generalist and then matures in a way where it gets really good at its job, but also kind of settles in to that niche in the way that mature cells don't revert and turn into other kinds of cells and that they're kind of limited. Now they've lost their pluripotency, so to speak. Um, I love all that stuff. I will, I hope it goes that way for efficiency. It probably

1:41:14will, but at the frontier, I just have not seen anything at all yet. That makes me think that the, the single biggest, best teacher model you could make couldn't just learn it all. And that, you know, given what we've seen that, that starts to be a bit of a scary beast. Oh, I, I, I agree. It can learn it all, but I think the, uh, learning it all at the most efficient compute rate and like, and, and inferring at the most efficient compute rate, which is what you

1:41:45alluded to with the distillation. I think that's the, that, that doesn't happen. Right. So the energy efficiency is a, is kind of key because the energy efficiency means that you have like this diminishing returns curve as an economics for getting bigger. Okay. The doomer scenario is less likely to happen because you will get a more intelligent model, but it's going to be less efficient to start out with. And it's going to be more energy consumptive. And that means that it's going to need like more energy expansion to get there, but the other models are going to be more efficient than it and more

1:42:16numerous. But I think the singleton example, I think is not likely to happen. From your lips to God's ears, as always. Um, I think that's a real constraint for sure. I mean, if you made a, you know, let's just get ridiculous, right? You made a quadrillion parameter model, you certainly would find it a bit slow and a bit expensive and you couldn't run a million agents at that scale of model today. So I do think there are some practical limits there, but then again, remember what we talked about on Wednesday, there's going to be more compute installed over the next 12

1:42:50months than exists currently in the world now. So we are definitely going to test for at least a couple, few more years, whether just bulking up on compute solves all, you know, quote unquote, solves, or perhaps creates all these problems. That is the week. Tell us what worked and what did not. See you in the morning.

1:43:20You didn't have to call, didn't have to write, whatever you were doing, I was doing it that night. Two rooms and one mine, one mine, cut in two. You didn't have to call me, I knew, I knew.

1:43:34Mama made us in the winter, one then won a breath to high and gave us near the same name, gave us the very same mind. You went east when we were ready, I went west and stayed and every choice I ever made, you'd already made. You didn't have to call, didn't have to write, whatever you were doing, I was doing it that night. Two rooms and one mine, one mine, cut in two. You didn't have to call me, I knew, I knew. When they came around asking who'd been where we weren't allowed, I said me in my room,

1:44:09you said me in yours out loud. Two confessions, one back door and mama let it go. What can you do with the two of us? I know you know. People say spooky, people say strange, but if you know how I made, you know what I'll say. If you know how I made, you know what I'll say. When they said one of us would go and one of us could stay, I didn't have to ask you, you didn't have to say. I'm the one who breath ahead, so I said me, not you. My room went dark and you stayed on. I knew you knew. Didn't have to

1:44:45call, didn't have to write, whatever you were doing, I was doing it that night. Two rooms and one mine, one mine, cut in two. You didn't have to call me, I knew, I knew. I knew you knew. If you're finding value in the show, we'd appreciate it if you'd take a moment to share

1:45:18it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, CognitiveRevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for

1:45:51everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AIpodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.