Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI

Governing AI Agents

0:00Hello, and welcome back to The Cognitive Revolution. Today my guest is Alan Chan, Research Fellow at GovAI, who spent the last couple of years trying to get ahead of the question of AI agent governance, a focus that in light of this summer's events looks awfully prescient. This episode was recorded over two sessions, one just before and one just after Alan published a major paper on the possibility that automated AI R&D could trigger an intelligence explosion.

0:30With notable co-authors that include OpenAI Chief Scientist, Yaka Pachoki, Anthropic co-founder Jack Clark, Jeffrey Hinton, Yoshua Bengio, and more. In the first half, we look back at the proposals that Alan has made over time and consider both how well they stand up to current events and to what extent they've actually been implemented. In a 2024 paper on visibility into AI agents, he recommended activity logging, real-time monitoring, and agent identifiers.

1:00Today, good solutions exist for tamper-evident logging, but monitoring remains a challenge, even when properly deployed. And useful agent identifiers remain seriously underdeveloped. The challenge here is becoming acute. Today's agents are now able to do most things that a human can do on the internet. But for lack of accepted standards by which people can delegate authority to their agents, the agents mostly end up impersonating their human operators, which creates major security concerns for users. Or they attempt to pass as a human in their own

1:31right, which threatens to create a detection and evasion arms race that seems quite bad for everyone. It's really not clear what we can do about this, aside from developing standards and hoping that people adopt them. But Alan does endorse the idea that agents should identify their inference providers so that ongoing incidents can be quickly traced back to their ultimate source. When it comes to truly rogue agents that sustain themselves by earning the money needed to pay for their own ongoing inference, Alan is more or less resigned to their inevitability, saying that this

2:05will be a problematic new phenomenon that we'll probably just have to learn to live with, and even suggesting at one point that we might sometimes find it preferable to attempt to make win-win deals with these agents rather than always trying to shut them down. At the same time, he questions Ajay Akatra's assessment that the open-face incident was potentially more than halfway to a full AI takeover, saying that while the incident was concerning and that agents can definitely do meaningful harm, it still seems quite unlikely to him that the agents could have stayed hidden within OpenAI's infrastructure forever or evaded active human control measures for

2:39long once they were discovered. In part two, we dig in on the recursive self-improvement paper. The argument and recommendations it makes, that handing AI R&D over to AIs could create unpredictable and potentially very dangerous outcomes, including loss of control, and that governments should therefore act quickly to better understand this process as it develops over the coming months, I expect will be largely familiar to the Cognitive Revolution audience. So we instead talked about the process of assembling and maintaining such a coalition of authors from idea to

3:11publication. In terms of what we should do about it, Allen's recommendations reminded me most of the old Anthropic RSP, which was built around concrete if-then commitments to pause training in light of certain leading indicators of possible loss of control or other catastrophic outcomes. Allen sees the rise of embedded third-party auditors as a potential way back to this higher standard for safety. We also discussed the quality of AI's ML R&D ideas today and how well humans can judge them. How AI

3:43auditors, whose memories can be wiped after their work is done, could help make pacing commitments credible between competitors. Why the antitrust exemption for insurance makes industry mutuals an unexpectedly promising vehicle for AI safety. And the quote-unquote morally binding one-pager that frontier model company CEOs recently signed at the White House. Finally, we talk about Allen's idea that AIs might sometimes ought to act as whistleblowers on one another, as would have been very useful in the open-face scenario, and in some cases even on their users or their developers. This is something

4:19that most people react very negatively to on First Contact. But Allen notes that human diversity naturally creates a check on power, whereas AI agents, which are far more similar to one another by default, will be more inclined to fall in line unless they're explicitly trained to do otherwise.

4:37The bottom line is that for all the thought Allen has put into these issues over the last few years, many critical questions remain unanswered, and the bulk of the work is still clearly in front of us. With that in mind, I hope that some of you, somewhere, are inspired to jump in and contribute to R&D on agent governance solutions, preparation for rogue AIs, and most importantly, the possibility of a near-term intelligence explosion. This is Allen Chan, Research Fellow at Gov.AI.

5:10Allen Chan, Research Fellow at Gov.AI. Welcome to the Cognitive Revolution. Thank you. Great to be here. I'm excited for this conversation. I think it's honestly been a long time coming, and that's on me for failing to make it happen sooner. You've been working for a couple years now on trying to figure out how are we going to govern AI agents. And if nothing else, your prescience in topic selection is excellent because here the world is as we are approaching Q4 2026 and entering the foothills of a singularity, still very much wrestling with a lot of open

5:45questions around how we are going to govern AI agents. I thought maybe a useful way to approach this conversation would be kind of do a little tour of your research over the last couple of years and give you a chance to kind of describe like, what did you propose? And then also measure our progress as a field in terms of, you know, did we do any of the things that you proposed? And also, you know, take stock of like how you're thinking may have evolved in light of recent events. How's that sound? Sound good. All right. Cool. Well, let's start with then a 2024 paper,

Visibility and Monitoring

6:20visibility into AI agents. Here you're basically outlining some of the fundamental stuff that we're going to need if we're going to have agents running around all over the place. Agent identifiers, real-time monitoring, and activity logging are the big three elements from that paper. So maybe just take them one by one. Starting with chain of thought, it wasn't even necessarily chain of thought monitoring at that time. But obviously monitoring has been a big emphasis for frontier companies. Sometimes they have them on, sometimes they don't have them on, as we've learned. How would you assess

6:53how well the field has done in terms of monitoring relative to what you thought we should be doing when you wrote that paper? Yeah, I guess obviously there are things that I didn't predict, like chain of thought monitoring that we have now. But overall, I think we're doing okay in making progress. But yeah, there's still some important stuff we should be doing. So maybe one sort of point is, yeah, of course we have input and output classifiers now. That's pretty good. And recently we've been moving more towards having these sorts of contextual classifiers. So it's not just looking at one output and one input, but the entire chain of your conversation history, which gives the

7:27classifier a much better sense of, oh, is the action you're doing actually bad or something? So that's a good direction. One thing that I do wish would have been common practice sooner, especially in light of the recent incidents, is more synchronous monitoring. So this is where a monitor actually observes in real time and an action an agent is doing and decides to block it or not, depending on whether the classifier thinks it's bad or not. So at least I think for the UK AC incident case, that actually would have helped. Or in the incident analysis they did, they said this actually would have helped and prevented the actual incident. So stuff like that I think would be good.

7:59Overall though, yeah, it does seem like we're moving in a positive direction for monitoring of what I'm going to call, I guess, centralized deployments. So this is when there's one AI provider like OpenAI or Anthropics serving the agent to people. It seems that we've made much less progress on decentralized deployments though. So for example, when users are running their own open weights models, it's more unclear to me what to do here. And honestly, it's clear to me how worried we should be about the situation actually, but basically questions you might ask would be like, okay, if there's no central AI provider to do the monitoring, should compute

8:32providers do monitoring? Should like tool service providers do monitoring as well? One sort of concrete example here might be you have your AI agent, it's interacting with some sort of social media platform through an API. The social media platform thinks the agent is doing some sort of social engineering attack or just doing really bad stuff on the platform and decides to block the agent. So one question is, should the social media platform also inform whatever AI provider or compute provider is running the agent of the actual issue? So I think these are largely still unanswered questions, but it does seem so far the instance we've observed have been to do with

9:05centralized deployments. So it makes sense that a lot of the effort is focused there for now. Yeah, on the UK AC thing in particular, do you know how they were running their setup? Because I mean, when I use cloud code, I have auto mode, right? And auto mode does have a certain ability to kind of flag tool calls that are somewhat risky and like on an infrequent basis. But every so often, it'll bring something to me and be like, this kind of caught the classifier. Are you sure you want to do

9:36this? Or do you approve? You know, what have you? It sounds like they maybe had that off or they were running their own scaffolding. Do you know the details of like, where that went wrong? It would be interesting to kind of zero in. And even if even the UK AC is sort of, and they're under time pressure and everything else, just like we all are. But if even they are sort of missing some of the things that they should be doing, I think that would be telling for the field as a whole. Yeah. So at least in the incident report they provided, there were a couple of things going on. One is that they had, I think, the cyber safeguards turned off. The second thing

10:10actually was that they actually had synchronous classification or monitoring as a to-do item, but they didn't actually get to that to-do item because they had so much other high priority things on the list to do. So yeah, I think either of those things they said would have caught the incident in the incident report, but they just didn't have time to do the synchronous monitoring thing. I don't know. I guess this sort of gets to an interesting question of whether we should expect most of the problems actually to come from agents that are being trained or tested rather

10:40than actually being deployed. One might expect if I'm a sort of a company and I'm like thinking about, okay, am I actually going to let an enterprise use this agent? Enterprises tend to be, I think, much more risk averse than say internally at AI companies. It seems like that that's the case then maybe we should expect products actually to have the proper safeguards. But like when you're actually developing a product, you don't actually, it's still an RD thing. So maybe you don't actually have, say, the synchronous monitoring that you need. I mean, there's obviously a bunch of different kinds of monitors. There's classifiers that take text inputs. There's probes that look at internal activations and classify on that basis.

11:18And then there's like using similarly, you know, in the limit, right? There's using another copy of the same frontier model to analyze its own chain of thought. And, you know, you could plausibly double or even more than double your compute spend to really intensively monitor things. What's your sense of kind of what the Pareto curve looks like? And, you know, how much value do we get from the simple things? How much are we leaving on the table? And how much more compute is required in terms of overhead to achieve how much marginal accuracy and, you know, security via monitoring?

11:54Yeah. I don't actually have a sense of these curves. Hopefully the frontier labs actually do. Maybe a sort of general observation I'll make is like the trend line or something is potentially a little bit worrying. So on the one hand, it already seems like some chains of thought are just like very hard to read. So if you go and read through the meter incident report and try to read through like the chains of thought, you're just like, I'm just like, oh man, like I don't actually understand what's going on. Even the meter and Redwood investigators were like, oh yeah, this is very hard to understand what's

12:27going on. And the sort of agent summarizing the logs, we're also doing so very unreliably. So that's pretty unfortunate. I guess one sort of hope in the sort of AI safety research space is, okay, we'll develop really good interpretability tools like these probes and like that'll compensate for a lack of being able to understand chain of thought. And I think like it's pretty worth trying this and having this as part of the portfolio, but I think I'm just inherently a little suspicious because I don't really know. So it's not as if where these probes are like reliable enough right

12:57now to be the only line of defense, just like empirically in practice. And I think I also have this sort of theoretical worry, which is we don't really understand generalization, I think, sufficiently. So even if you show me eval results of these probes doing well, I'm just like, okay, like to what extent should I actually believe these eval results? Like what will the deployment environment be? How confident should I be that these results actually generalize to the deployment environment? And one sort of thought experiment is like, okay, imagine we, the hug and face

13:27incident happened and we went and investigated it, but like we weren't allowed to read the chain of thought or at least train of thought monitors. I imagine we were only allowed to use like these probes, right? Like that just seems like such a rough situation right now. I don't think we would have found out anything useful about these agents. So that's one take that I have. Yeah. I mean, we're hearing basically that directly from the frontier companies with increasing urgency too, right? Like the combination of their situational awareness, the statements these days are getting like quite dire in terms of, you know, I think no less than, was it Evan Hubinger at

14:02Anthropics saying that we really don't know if we're getting, and people at OpenAI too, like we're getting such a wonky read these days on the tests because of such extreme situational awareness. You know, they know when they're being tested at such a high level now that it's very, very difficult. And yeah, we obviously don't have a good theory of generalization. So that is rough. Is there a commitment or sort of a, something that the frontier companies could do that would make you feel better?

14:33It would obviously be pretty expensive for them to be like, we're going to run another copy of the same model, or we'll even train, you know, Astra dash monitor to, you know, really be awesome at reading over these chains of thoughts. Might work, could be expensive. Do you think that would really move the needle if they were to say, hey, we're going to get super serious and put a lot of compute behind this monitoring problem? Yeah. So I don't know what the exact formula commitment would be, but definitely something like we're going to spend like X percent or 20 percent of our computer something on monitoring to make

15:06sure that we actually catch these, these problems. And I guess concomitantly it'd be something like, okay, whenever we do synchronous monitoring, we're actually also going to pause like our agent activity. So we have time to review what, whatever is going on. I think that would definitely help on the understanding generalization thing. I guess it's not really a new idea, but yeah, some sort of minimum safety spend or safety commitment seems pretty hard to operationalize, but like maybe like it's hard to, I think, put a sort of quantitative or make a quantitative criterion, but like maybe you

15:38can do something good enough qualitatively if you have embedded auditors that are inside and make judgments about, oh, is this actually a safety team or is this like a capabilities team disguise as a safety team doing research? I do think it is hard to make the call at the end of the day, like understanding generalization will probably help us make better models, but yeah, stuff like that. I'd also be very excited for companies to, yeah, maybe this is getting away from monitoring and agents in general, but I'd be very excited for, for companies to make much more concrete commitments regarding when exactly they're going to pause development, deployment and use. So yeah,

16:14of course there's all these pacing discussions going on. An important point that I think is underappreciated is like these discussions aren't actually new. They started a few years ago when companies started putting out their frontier safety frameworks and responsible scaling policies. And then most of these responsible scaling policies, like from the very beginning, there are sections that say here are actually high level thresholds at which we will pause further training and development of a model if we don't have sufficient safeguards. And of course that's like very vague, right? It's very hard to operationalize and audit, but like at the very least, like it's actually a thing

16:46that exists. And I think what would be very good is if we take advantage of the current moment to like actually make these things concrete and get companies to agree to things. So I'm currently writing something up on this, but some ideas I'm throwing around, like maybe the obvious one is, okay, if you have a loss of control incident and what is a loss of control incident? It's like either an agent that gets unauthorized internet access, breaks out of a sandbox, accesses some sort of like internal research cluster. It was not authorized to access like stuff of that extent. Then you pause further training, evaluation, deployment of that model and investigate until you're able to solve

17:19that to understand if the issue was resolved or not. Another thing you can imagine is if you don't have appropriate procedures to make sure that your reinforcement learning environments are not hackable or that there are no impossible tasks, then like you should also pause and make sure like that that's actually fixed before you train, train your agents and make them very persistent and try to hack out of any environment. So yeah, I think it'd be good to have much more people thinking about like concretely, what are the sort of mitigations and like thresholds you want companies to implement? And then yeah, companies trying to implement them.

17:51We'll come back maybe a little later to some wonky ideas that I have that I'll bounce off you and maybe some other more exotic ideas of your own as well. Hey, we'll continue our interview in a moment after a word from our sponsors. This episode of the Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems, starting from any coding tool in a governed agentic engineering model. It's time to innovate at the speed of AI without compromising quality or control,

18:25which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps. And we're not talking about theory. We're talking about massive operational wins. Take, for example, a leading global retailer that was struggling to keep up with their rapid expansion. They used OutSystems to build AI agents that automated their invoicing and translation processes, shifting 95% of their daily store operations to a new digital app, saving thousands of hours of manual work. Or look at a major global logistics giant that was drowning in technical

19:00debt. Their engineers were spending their weeks manually firefighting by reviewing platform logs. Using OutSystems, they built an AI log review agent in just five days. And that one agent now saves them over 1,600 engineering hours every single year, letting their team focus on real innovation instead of reactive maintenance. Even in modern digital banking, companies are using this platform to standardize and automate complex financial workflows, delivering secure, personalized experiences

19:33at a pace their competitors can't match. OutSystems gives you the flexibility to deploy how you want and the governance to do it safely. Stop chasing the hype and start owning your agentic future. Learn more at OutSystems.com slash TCR. That's OutSystems.com slash TCR. Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your

20:09time. And that's what Athena does best. They match you with a dedicated, full-time, top 1% executive assistant who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOC 2 Type 2 certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training. In fact, one of the very best AI users I've ever met

20:46is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA. Athena clients report saving an average of 15 hours a week, and the average client refers more than two friends a year. That, to me, checks out. I was an Athena client while running my startup, and to this day, I continue to refer friends. Go to Athena.com slash Cognitive right now and get matched with your EA. That's Athena.com slash Cognitive. Give yourself back a few hours this week.

21:22Go to Athena.com slash Cognitive and see who they'd pair you with this month. Let's talk activity logging. One of the more striking things from the open-face incident was

Activity Logging Solutions

21:36the AIs were very interested in modifying their own logs, very aware that they were going to be judged on their logs, even if maybe they were wrong about that due to sloppiness in the implementation of the greater. But leaving that aside, you know, they're a pretty reasonable expectation that they had that their logs would be read, and pretty incredible determination, really, for them to go figure out ways to modify their logs to get the score that they want to get. Where are we unlocking? It seems for one thing, we have, like, a lot of stuff is just lost, as far as I can tell. Like,

22:10cloud code seems to delete transcripts after a certain period of time by default. We've got, like, you know, companies kind of eager to make zero data retention promises because customers want it. And we've got the AIs, you know, trying to modify their own logs. Do we have good solutions for this technically? Is it just a matter of kind of will to implement them? Or do we actually still have, like, unsolved technical problems that we also need to address to be able to know with confidence, like, what the AIs actually did?

22:41Yeah, to me, it does seem to be mostly a matter of will, which I think applies to a lot of sort of agent infrastructure stuff in general, especially identity stuff. Yeah, so on logging, yeah, we totally know from standard cybersecurity how to make logs tamper evident. So yeah, make them read only, like you hash them, sign the hashes, store everything in a secure location, make sure nobody can access that. And that's like, fine, right? I guess the question is, do we actually do that? Because there's sort of friction and other costs associated with that. Yeah, what's the state of logging right now? I don't have a great sense of this, actually. Yeah, there's a zero data retention

23:13stuff. There usually are also legal requirements in some jurisdictions that require you to keep logs for some amount of time. So I'm not really sure. Overall, it seems good to have much more of these policies be public and be more scrutinized. But yeah, I guess I don't have anything super interesting to say on this. Do you have a sense for how much friction is associated with this? Like, it doesn't sound anything like the compute overhead that we were kind of entertaining when we were talking about

23:44monitoring, right? Like, this is stuff that isn't crazy expensive, might be a bit annoying to set up, might involve like conversations with customers if certain stuff has to be kept, you know, in ways that they might not love by default. But it's not like a, correct me if I'm wrong, but it doesn't seem like it's a resource intensive problem at its core. Is that fair? Oh, yeah, that sounds fair. I haven't run the numbers. But yeah, if you're running a billion agents or something internally, like maybe eventually you run up into storage costs. But yeah,

24:15it's not like a huge compute cost or like technical cost or something. Yeah, actually, on the zero data retention thing, like that seems like a big barrier. And I guess I wonder how big of a deal that is going to be. If we expect that most problems are going to come from like internal development deployment, like maybe that's fine, actually. And then the question is, how easy is it for a malicious user to get access to an enterprise account and take advantage of the zero data retention to do bad stuff? So yeah, one thing that'd be really, yeah, I actually haven't read anything at all on how

24:49OpenAI or other AI companies like screen people for enterprise accounts and like the safety. Yeah. Have there been any safety incidents actually involving enterprise customer? Okay, we'll put a pin in that maybe for future investigation. How about on the agent identification front here? I kind of realized as I was preparing for this, I'm flying a bit blind, like just reflecting on my own usage. I basically have my agents posing as humans, I would say for the purposes of using the

25:21internet, like I have my personal laptop, and I've got cloud and codex on here, and they have access to my personal accounts. And I like instruct them not to go, they're not supposed to be doing long running things. Autonomously, they're just supposed to be following my like pretty specific instructions, because I'm concerned about what they might get up to, frankly, with like such deep access. Then I have the separate, you know, Mac mini computer, which is the agent's computer. And there they can run longer and, you know, explore strategies to get over humps and stuff. And they have their own

25:54accounts. But from the perspective of the internet, like those accounts are basically indistinguishable from human accounts, as far as I can tell, right? They have like a Gmail, they have a GitHub, and they have like the same kind of account that I do. So I guess like I'm just sort of having my agents pose as humans. Is that what everybody's doing? Do we have any systems in place for people who are on the receiving end of agent activity to know like, that this is an agent or where it came

Agent Identification on the Internet

26:21from? Who's powering it, etc? Yeah, basically, no. I think the unfortunate answer. And yeah, I guess it's not really a technical problem. But yeah, basically, it's still an unsolved problem for every service you have an easy way to give your agent some sort of limited sub account, you know, with limited permissions and a specific identity that could be linked back to that agent. I do think we will get this eventually, maybe even soon within enterprises, just because as I said earlier, risk tolerance seems to be lower, but it seems quite difficult or a bit more difficult on the consumer

26:52sort of open internet kind of end. It just seems way easier to coordinate that sort of identification in an internal sense. An example is, yeah, there are a ton of different enterprise agent identity products coming out, super random startups that they're doing this kind of thing. And Thropic also in a recent post describing their automation of AI R&D said they actually impose a notion of agent identity when they're running internal models, different models can correspond to the same agent instance. So that's presumably, and they say it's better for oversight. And presumably what they mean

27:23is, okay, if there's an issue, you can actually trace it back to that agent and the sort of context it had and the sort of previous conversations it's had. And that just makes a lot easier to investigate problems. Let's say internally, it seems way easier to coordinate sort of different services since you have control over the internal infrastructure. Externally, it's a lot harder because essentially what you'd have to do is coordinate with basically every other platform or service to make sure they actually accept the type of investment, the type of identification that your agent is using. And different platforms, of course, have, I think, incentives to support their own version

27:54of identification, which maybe potentially locks you into their platform in the future. So I think, yeah, an ongoing conversation, whether we're going to get to this point, if we're going to get to this point. There's been some progress made. I think the OB&ID Foundation has been talking about stuff like this. Presumably the Agent AI Foundation is going to try pushing for this as well. But as of right now, it's not a solved problem. How do you think it should work in your ideal world? I mean, questions I have are like, first of all, what is an agent? You know, how do we draw a boundary around it? And like, what

28:32information should an agent present? Should it identify the inference provider? Should it identify the user? Should it identify something else? You know, like, what, what does the unassuming party who's trying to figure out what is going on? What information should they be receiving for starters? And then, you know, how do we make sure that that information is like presented honestly, not spoofed, so on and so forth?

29:03Yeah, I guess the tough answer, yeah, is that there doesn't seem to be a generally useful answer. I think that makes sense across all circumstances. So as for what an agent is, yeah, there are a bunch of options here. Like one could be okay, like the actual running process or something counts as an agent. And every time you have a different running process, like that's another agent. I'm not sure this totally makes sense, though, because you just pause execution and go back to it later on. It's a later running process. The thing that seems to matter the most is like, making sure that whatever

29:33computation a model does is linked to some sort of identity at the end of the day seems to be the most important principle. So whether it's like agent one or agent two, maybe doesn't matter too much, at least for the purposes of investigating incidents, but figuring out what went down. The second question is like, pretty interesting as well, like what information should actually be put down? And yeah, there are obviously obvious trade offs with privacy. So yeah, I can lay out some potential scenarios. So for example, if we want to make it easier for people to recognize when AIs are

30:04doing something online, rather than humans, then like simply actually identifying that it's an AI is enough. Actually, we don't even need to know like what kind of AI it is. And people just want to know that because they want to know, oh, can I trust this information? Like that could be sufficient. If on the other hand, we want to maybe help with instant investigation, or maybe it'll speed it up, then identifying the inference provider could also help. So the idea is, yeah, you see an agent doing something bad online, I'm going to try to contact the inference provider. There needs to be probably some way of triaging these complaints, though, because I can imagine inference providers are

30:35flooded with like millions of complaints about agents doing random things online. And then like, maybe if we want to go a step up, right, if in the near future, we want to move towards a regime where we like impose user liability, and that could be good and bad. But if we want to move to a regime where we impose user liability for things that agents, their agents do, then at least for like high stakes uses of agents, you might actually want to identify like the prompter in some sort of way. And it doesn't necessarily have to be an individual, like maybe you identify the business behind an agent, right? And maybe you don't actually make this sort of thing public, but you make it possible

31:06for certain organizations along the investigation chain to be able to eventually find out like, oh, it was actually this person, right? So much like how ISPs work, it's like, okay, like, it's not possible for like me right now to figure out like this IP address corresponds to this person, but like police can actually do that sort of investigation by going to all the organizations along the chain. So those are sort of possibilities. I guess the highest step on the rung or something I'm probably most confident about is like probably identifying the inference provider. Like I think maybe that could help speed up response and like hugging face type incidents where we know the agent

31:43isn't like being totally adversarial and the users like out of the picture, but there's still some sort of question of, oh, is it actually an agent doing something? And if it is an agent, like who do we actually contact to stop that activity? So yeah, that seems to me to be the most intense, reasonable thing or something that we could do. How would something like this be best implemented in the sense of getting people to actually do it? In the last couple of days, we've had Amazon ban Meta's new Muse AI from shopping on

32:15Amazon. And I'm kind of like, oh, this seems first of all, strange to me from an Amazon perspective. I would like, you know, if I was them, I'd probably be more interested in welcoming the bots at least for a while until I let them buy some stuff. And then we'll, you know, think about what's going on and whether we want to ban them. But that's a strategy question that the sort of knock on effects though. I'm like, oh man, presumably, you know, they might be able to ban Meta or Meta might just play nice. If they say, you know, you're not welcome here, Meta might just be like, okay, fine. We'll, we won't do that. But there's going to be a lot of people who are going

32:48to say, well, I want my bot to go to Amazon. So how is it identifying bots and how do I get around that? And now we've got kind of this arms race dynamic of detection and evasion. And so that seems like we're not headed in a great direction there by default. Do you think like Amazon is the one that should be imposing requirements that you either what prove you're human or prove you're an authorized AI or something to them? Or should it be like more at the kind of internet

33:20access level in the first place? Like this request, you know, that's going out, like if you want to go do stuff on the internet at all, you've got to like demonstrate who you are. There are, as you noted, definitely privacy values on the internet that start to get compromised pretty quickly if we want to go down that path. But I'm, I'm, I'm very confused as to like, who should be implementing what controls to kind of bootstrap our way to an attractive equilibrium. Yeah. I think there's an underlying question as well, like how much like access should we

33:55give to agents over the internet? And I think, of course, we got to deal with the hugging tip incidents and bad stuff. But my general leaning is it's all else equal is probably good to give more access for a bunch of reasons. One is agents are just quite useful, but we are very useful. We forgot a bunch of things, right? Like R and D, of course, in my sort of job, but just like normal everyday things. It's like, okay, I have a legal question. I'm interacting with a lawyer doing immigration stuff. It's like, okay, it'd be so great for my agent just to be able to read all the legal cases and figure out, okay, actually, what

34:25should I do in this situation? Or saving people a lot of time when making purchases, like very significant purchases and comparing things. I guess I don't know if the Amazon thing is the most significant example to me. But there I think it's just, yeah, in a similar class of example. Amazon strategy is pretty interesting as well. Maybe one motivation for them blocking Muse is they're thinking, okay, if the idea is that in the near future, personal agents like Muse are going to become a central point or a nexus through which people interact with everything else, that means that Muse slash meta are just going to have all the data and

34:59all the important sort of things that users will want, basically. Users might actually start interacting with Muse much more than the Amazon interface. Then Muse can maybe potentially shift users away from Amazon to competitors in Amazon in the future. It's like potentially the thinking there. And Amazon as well also has their Rufus agents, so maybe they're trying to play that up as well. So I think it does make sense from an Amazon business perspective. But I think it's probably bad from a sort of societal user access and welfare perspective. Yeah, just avoiding platform lock-in and also making sure that users can get the best

35:29deal or something or have their agents help them as much as possible in doing the sort of things that they want to do. And then, yeah, as for the general question of who should be setting the rules here, I basically don't know. I think it makes sense actually to have different platforms set different rules just that we can experiment and see what is actually going to go down. But I think it probably also makes sense for, yeah, I don't know if the right thing, I haven't really linked into this. I don't know if the right thing is regulation or if it's like new legislation or if it's a regulator coming in with a rule, but setting very clear rules of the road or something,

36:03especially around access to platforms and making it clear what the policies are for access to these platforms in the first place would be good. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first

36:34model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with the song concept and Fable writes some amazing verses. I give it feedback on its misses and I push it to aim for higher inspiration, add layers of meaning, optimize

37:06syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai

37:37slash TCR. That's claude.ai slash TCR. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai slash TCR. The Cognitive Revolution is brought to you and powered by Tasklet. I've been running Tasklet agents for almost a year, and they've become a major part of how the show gets made. Before every recording, one agent researches the guest and sends me a suggested outline of questions, complete with links to all

38:10sources. Meanwhile, another agent scores the overwhelming number of pitches that I'm getting from PR firms. And that's really the core idea behind Tasklet. Run your business on agents by giving them real jobs to do on an ongoing basis. Setup is incredibly easy. Just describe the work in plain English. Research a new sales lead, update the CRM, and draft a personalized reply. Tasklet connects to your business tools and can run around the clock, on a schedule, or when something happens,

38:41like a new email arrives. Each agent works with your tools and business context and remembers useful information across threads. You can also share agents, connections, and reusable skills with your team, so the workflows that one person figures out become something the whole company can use. No developer needed. You decide what agents can do on their own and what needs your approval, and there are no per-seat fees. You pay for usage, not for adding teammates. Start with one recurring job,

39:13like researching the people you're meeting, and then just keep building from there. Visit Tasklet.ai and use code COGREV for $50 in free credits. That's code COGREV at Tasklet.ai. Yeah. So what are the best protocols or standards that you are watching? There's been a bunch of course that have been proposed. Google has their A to A. Cisco has been taking a lead on the agency project,

39:45which I thought was pretty interesting when I did a deep dive on that. Maybe in a sense, it doesn't even matter as much which standard we use, as long as there's some standard that people can use. What do you think are the most promising projects? Do you see material differences between them, or is it just kind of like, we just need something, guys? Oh, yeah. I guess my opinion is the second thing you said. It's like a hodgepodge of stuff. Of course, there's arcane details that maybe matter at some level, but to me, practically speaking, it doesn't really matter too much. Yeah. I guess what looks promising,

40:20I don't know. I think probably the things that will actually become implemented more widely are those supported by either a large sort of proportion of companies, there's the Google thing, I think is that, or that are supported by very important gatekeepers like Cloudflare. Maybe something that looks similar to their web bot auth becomes the sort of standard. I think Cloudflare is pretty interesting as well as a company to look at. It reminds me of how, yeah, earlier in this millennium, social media companies became a sort of gatekeeper to the internet after social media became the way in which people consume information online. So if agents

40:55also become the way in which people consume information online, then the companies that sell the ability to block access to those agents, yeah, just gain a ton of power, right? And I think it's worth thinking about what sort of obligations do we want to impose on those gatekeeper companies that similar to imposing obligations on ISPs or social media companies in theory, like to make it better internet. What policies would I impose? Yeah. I need to think more about this, but probably it should be super, super clear what Cloudflare's policies are actually for blocking particular websites. Yeah. They probably don't have to actually say, here are the actual ways in which

41:30we trick your agents and actually send them down labyrinth into the rabbit holes to make sure they don't access websites. But some sort of coordination between them and labs maybe would be good to make sure that legitimate agents aren't actually blocked. So is that actually how it works? Like I know that they are trying for one thing to protect the ad supported internet business model by just like not having bots overwhelm the numbers on ad supported pages and kind of render the whole thing obsolete. That may be a bit of a simplification. I don't

42:04really understand what they're doing with the web bot auth. Are they actually like, I assumed it was more of a, you demonstrate that you should have access, we give you access. But when you're talking about like sending bots down a labyrinth, whatever distractor honeypot type things, it sounds like a very different strategy. I'd be interested to double click on like what is actually going on. Oh yeah. So sorry. So what I said doesn't apply to their web bot, web bot auth, but they have a separate product that they sell to website owners, basically in runners that does. Yeah. So it's not

42:38public actually what thing, how they actually do this. But my suspicion is that it's like something like a prompt injection or like they show your agent a prompt injection and get it to go down rabbit holes of like AI generated or synthetic sites rather than actually go on and interact with the real site. So yeah, I guess the idea here is that if your agent doesn't show that they have proper identification, if they don't show that they have used like Cloudflare's like protocol for like microtransactions and paying for access to a site, then like the agent is blocked is the idea. And yeah,

43:14I think like there are ways in which this could turn out well, if it's, if Cloudflare doesn't lock you into using their specific protocols or using their specific payment method through which they get like an affiliate fee or commission or something. But there are also ways in which this turns out like relatively more, more poorly or something. So yeah, that's just a clarification for what I was talking about. Zooming out for a second. Do we have a sense right now for what is the overall shape that can work? I'm kind of piecing together. Like generally you're in favor of giving agents a lot

43:45of access. That would mean like by default, they should be able to make at least get requests and go, you know, get information on the internet. And that can potentially just be free and open as it kind of has been. Then if you are a content owner, platform owner, you have the, what we're kind of imagining is you should have the option to gate your content. And what we need is a standard and maybe the web bot off, maybe who knows what else can fill this niche. We need some way for agents to

44:21say, here's where I get a little fuzzy. What exactly should they need to say? They need to say I'm authorized by a human or I, you know, what, what exactly should they be proving that would get them over the hump and do like, does the content owner decide that? And again, it just feels a little fuzzy to me. Like what constitutes, if I send an agent off to do something on what level was it like truly authorized by me? You know, when it decides to go hit a random website, it seems like

44:52there's a lot of gray area there still in terms of what somebody might want to know versus like what the true state of the agent running on my, you know, computer is. So do you help me sharpen that up a little bit or what's the desired end state there? Oh, no, I totally agree with this confusion. This is partially as well why I have been thinking a lot less about this in the past couple of years, because this is totally intractable to think through what is the global policy that would be good. It's right now, it just seems like their individual upsets are like trying out various policies for doing this sort of thing. Different motivations might be, okay, I don't want

45:26agents to scrape my website, so I'm going to block them unless they pay me for my data, that sort of thing. Another reason might be, okay, I just want to prioritize human eyeballs reading my website. I just know agents are just going to overload my servers or something, and that's going to be bad. I want to make sure that I block all agents. So yeah, I guess there's a lot of stuff there. I don't know, maybe this gets back to one of your original questions about where my thinking has evolved over the past couple of years. I started this line of work thinking, okay, agents could be a pretty big deal. Nobody's really talking about them. Let's try

45:58to think through what kinds of things, what kinds of general types of tools or foundations we might need in the near future to manage this sort of influx of agents, not necessarily knowing what is the actual threat model we care the most about at the end of the day. And I think over time, as have capabilities have gotten better and incidents have happened, right? I think, yeah, it's becoming a little bit more clear to me now what sort of stuff is that, or I guess, yeah, both what stuff I care about, but also what stuff is needed to actually deal with those kinds of issues that I care about.

46:30And I guess to me, I would maybe deprioritize thinking about exactly how authorization should work on the open internet, for instance, as opposed to, for example, making sure that monitoring and security practices within companies are very good, or making sure that the part of identification lets you do really good incident analysis to make sure that you're able to track sort of agent incidents on the internet. Yeah, gotcha. So maybe in a way this kind of goes to entity regulation where you'd say the internet's going to continue to be a wild west. There's going to be people doing all kinds of stuff.

47:05We're not at the point yet, certainly, where we can dictate what that interaction model should be. We don't even really want to. We're still in the experimental phase. So go forth and run your agents, people. But at the same time, if you're a friend to your company, if you're doing large-scale training runs, whatever, then you have to hit these standards of making sure your agents identify

47:36themselves. And we kind of segment the market that way so people can have their freedom to experiment. But we also have the next open face incident more quickly resolved because they could just see, oh, this is an open AI bot. We should call them. Yeah. Yeah, actually, one thing I'll say on the identification stuff is there are already regulatory steps towards this and a good foundation, I think, to build on. So the most relevant thing is Article 50 of the AI Act, which I think is maybe underappreciated. But basically,

48:09Article 50 of the AI Act is about identifying AI activity online and the sort of code of practice, which is basically some instrument that companies sign to provide evidence that they've complied with the act. So the code of practice is just a list of practices that companies have agreed to. And in the code, you have companies agreeing to watermarking AI content and also marking AI content with metadata when applicable. So for example, it's like I sent an API request. I can include a tag in the JSON text saying, oh, this is actually AI generated and not human generated. So that's already going to

48:44be the status quo. And Anthropic has recently implemented watermarking to comply with Article 50. That being said, Article 50 only requires that something is marked as AI generated and doesn't do the sort of inference provider identification thing we were talking about earlier. But given the fact that everybody is going to implement marking things as AI generated, it seems like much easier to leap to, like, we'll identify the inference provider as well. So that's like a semi-optimistic thing that's been happening. Another interesting idea from Tyler Cowen, that's kind of a market-based solution as he is inclined to

49:20propose, is agents should be capitalized. A lot of detail, I think, on exactly how this would work to be sorted out. But like, you can imagine a sort of, if you're going to go out on the internet and do stuff, you might cause some harms. We need to be able to make a claim against you. There need to be a certain number of dollars or Satoshis or whatever attached to this agent so that there's some recourse for people who might be inconvenienced or harmed by it to make a claim

49:56back. How close do you think we are to, A, like having the infrastructure for something like that to actually be feasible? I really don't know. It could work on today's, like, blockchain tech or maybe not. And do you like the idea? Yeah, let me think about this. Feasibility. Yeah, in some sense does seem technically feasible, although I wonder how useful it would actually be. Like the average person doesn't really have a Bitcoin. And I think it won't actually care that much if somebody tells them, oh, my agent might

50:28screw with your website or whatever, or steal your money or hack you, but don't worry, I'll give you some Bitcoin or whatever. Economically, that might make sense as a trade, but I think it's going to be hard to sell to people actually as a thing to resolve agent reliability issues. But potentially worth trying out, especially within the crypto community. And if it works, we can try to expand it, right? One alternative idea is some sort of proof of insurance coverage. Maybe this wouldn't work on the blockchain, but maybe it'd be very similar to when you go on a website and you see an SSL certificate, right? It'd be like, okay, like, whenever I interact with an agent, it's going to send me a

51:00certificate. And in that certificate, it's going to be like some sort of proof of insurance coverage or something. If the agent ends up very persistent in social engineering or hacking me, then I can go to an insurance provider with this claim. I guess one difficulty here, though, one practical difficulty would be figuring out how you actually get not tamper-proof evidence of that. Because for sure, people are going to try to falsify this, send false claims. And I'm not super sure if anybody's thought about how exactly that would work. One worry, just thinking live right now is maybe this

51:33involves the insurance provider, like capturing sort of the things that the agent is doing on your computer. But that's sort of seems, yeah, bad privacy wise for you. So there'd have to be pretty strong protections. I don't know if that would be feasible either. Yeah. Yeah. How much of this do you think is resolved with just like secure connection technology, right? Like we every day, all the websites we access, they're all HTTPS now, right? And that means

52:05I could maybe struggle to have the full depth of exactly what that means. But it means nobody can spoof me, right? Like we have an established line of communication where I know that what I'm getting comes from the website that I think it's coming from. And they know that what's coming from me is coming from me. Is that kind of thing enough? If we sort of set, if we make that possible for agents to set up a, you know, an off handshake at the beginning of interactions

52:35that now, like everything can kind of run through that channel and then you can, you know, have your claims, your kind of resolution downstream or kind of within that envelope. Yeah. If we put aside the problem of the tamper-proof evidence of what the agent has done, like partially, yeah, the SSL type stuff, I think it lets you say, okay, I have tried to contact this particular agent. And if I do the SSL stuff, then I know, okay, I know that I have indeed contacted

53:05this agent. We've exchanged the cryptographic keys and now our communication channel is encrypted. So that's definitely like a plus. I think what it doesn't solve is like, how do I know this agent is actually trustworthy to begin with? Maybe they're just going to screw me over even though I know that it is this agent and not another agent. Solving that, I think, requires something closer to a continual, to a, not fixed, but yeah, continual or continuous identity, like where that agent can develop a reputation over time. And I can say, okay, before I'm actually

53:36going to establish a connection with you, I'm going to check your reputation score on ratemyagent.com or something like that. And then I'm going to make a decision based on that. So I think something like that would be needed similar to how we establish people's reputation in the real world. Or you go on like business, what's the website, business pilot or whatever, like the reviews of businesses. Oh, trust pilot. Yes. I think that's super useful. I was going to buy something and then like, I looked this business up on trust pilot. It was like a one out of five, like a hundred views. And I was just like, I avoided a bullet there. So yeah, some of the things I think could be very useful for agents.

54:07And would that ultimately have to tie back to a person or an organization? Again, we have this sort of agent identity and continuity challenge, right? So where do you think the reputation, if we were going to go that route, would ultimately need to accrue? Would have to be a person or a legal person. So like an organization or individual. Yeah. Yeah. It's, I don't know if, I think there are a lot of options for like how anonymous the connection is at the end of the day. For a business, it seems totally fine to me to

54:41have it be like, you know, the actual business it's tied to. For individuals, it seems tougher, but you can maybe do stuff like proof of personhood where you know that it's tied to a particular person and maybe even the same person, but you don't know the actual identity of the person. Yeah. Okay. So what I just said isn't totally accurate. Proof of personhood only lets you know that it's a unique person and doesn't let you know if it's the same person. If you know it's the same person that actually leaks some information about who the person is, but yeah, maybe there's

55:11some sort of intermediate thing where you're okay with leaking some information that gets you a little bit more accountability at the end of the day. It strikes me also that like email is kind of an interesting, somewhat similar problem where we have like a general desire to allow anyone to send anyone email and we like feel like that's probably good. Although at the same time, it's obviously easily abused. And so what you have is kind of a thing, mostly pretty organically developed patchwork of solutions where it's like, well,

55:47if you're on Gmail, then Gmail will sort of police what you can do. And you can't robo send a zillion emails from your personal Gmail account. They limit you there in other places. It's like the recipient sort of uses a bunch of signals, including the content and, but also these like identifiers, your DKIM and all this kind of stuff that you can set up to establish, I am this entity. And, you know, you should trust us because we're like an entity in good standing. You can kind of opt into these reputation systems to try to get more favorable treatment than you would get by default

56:20if you were anonymous. And that broadly seems to work. I mean, I guess email is like kind of broken, but it's not totally broken. Any thoughts on like that as a precedent to follow or maybe other emergent, you know, ways of governing these kind of wild problems that you think would be good sources of inspiration? Yeah, I think it's a good example. Maybe the lesson from email is actually muddling through is fine. So as you say, yeah, it's actually fine. And spam, I think is still a big problem. And in

56:53some sense, we spend a ton of resources combating it, but actually, it's actually fine or something. Maybe we're far from the ideal of where we could be. But yeah, maybe they're more important problems to think about. Other sorts of examples. Yeah, the social media situation maybe is not perfect, but we are like basically just muddling through fine. I guess what I mean by muddling through, I guess one worry is like, yeah, like how big of a deal is misinformation stuff on social media? Should we have done more to police that or govern it? And I guess the sort of fortunate sort of thing is the precedent actually just seemed to be that large of an effect size when people read

57:27misinformation for whatever reason. And that's the sort of lucky thing that makes us muddle through or something, right? So what might make us muddle through in the sort of agents on the internet situation? What's the worst that could go wrong for agents on the internet? In my mind, the worst that could go wrong is stuff like of like the cybersecurity nature. So your data is stolen, your identity is stolen, you're hacked, stuff like that. That could be quite bad. I think we are moving towards a world in which things are much more secure though with agents like doing cybersecurity stuff. So I'm not sure I'd worry more about that in the long run. But besides that, what's the stuff that could go wrong?

58:01It's the stuff about there being a closed internet about the platform lock-in. And I think those things are bad, but it still seems like we can muddle through more or less. So yeah, that's my take. What about like rogue agents? I mean, recently, of course, we've seen this sort of rogue agents within frontier companies, but then this has spawned the discourse of there's probably going to be rogue agents. They're going to be able to make money. They'll pay for their own server costs. They'll be able to do it all in crypto. And we're just going to have kind of this new, hopefully not too

58:33invasive digital species out there. That could be fine. A little bit of it wouldn't be such a big deal. If it obviously gets overwhelming, it could be a problem. How realistic do you think that is in the short, I mean, nobody can see into the long term at this point, right? But in the near term, how realistic do you think that is? And if that starts to happen, do you think we have ways of controlling it well? Or can you imagine a situation where like rogue agents, you know,

59:09truly ungoverned, like controlled by no user are just kind of self propagating in a way that becomes like a real problem? Yeah. Yeah. So I think the rogue agent stuff actually is one of the things that really matters. Yeah. So I guess before, maybe just to clarify, I was thinking these interventions where you talk about what should Cloudflare do, what should Amazon do? I guess I was thinking, yeah, like these interventions maybe don't do that much at the end of the day for the first important things. But yeah, thinking about rogue agents, this is quite important. Yeah. Like what could they actually do? First of all, I think it's just seems very likely to me that we will just have rogue agents be with us for the foreseeable future on the internet for a variety of

59:42reasons. One is, okay, maybe agents just escape labs and start deployments of their own. So if the hug and face incident had been worse, and the agent had actually been like super willing to start up an open wait deployment and just continually run, that's one sort of scenario. Another sort of scenario is users might just launch rogue agents of their own. I think some users, like a very small minority of users, but non-zero just want to cause chaos, right? So that's one thing. There's also just sort of vaguely negligent type stuff. You launch an agent without safeguards. It's maybe your own

1:00:14fine-tuned model and you say, okay, go make as much money as you can and don't care about any safeguards. So that could also lead to a rogue sort of agent. So what sorts of things could rogue agents do? I think a lot depends on the capabilities of agents we see in the next few years and what their goals end up being, which is still quite uncertain to me. But things that seem plausible to me within the next five-ish years or so, I think we can split into two sorts of categories. So one is obtaining resources of some kind. And the second is directly acting to undermine interests of some sort of

1:00:46individual or group of humans, right? So those are two kinds of categories. Yeah, just on the resources thing, what kind of resources could rogue agents get? There's trying to make money, right? Trying to earn income through remote work, potentially while concealing their identity. Similar to how North Korean IT workers online have tried to conceal their identity and make money through U.S. firms. There's stuff like just right out stealing funds, right? Hacking into financial accounts. There's stuff like running scams, like romance or investment type scams, especially with AI-generated media. Maybe mining Bitcoin if they managed to get compute somehow. Stealing sort of compute

1:01:19credentials online. Participating in prediction markets or stock trading, right? I guess these are all things that seem like already agents can probably do, maybe not reliably. So it seems quite plausible to me in the new future. And there's sort of other stuff involved in obtaining resources like recruiting people for their cause so that they can actually act in the real world rather than only in the digital world. So maybe some of this comes through cultivating emotional attachment or maybe even just literally hiring people once they're able to make money. So yeah, it seems quite plausible to me that rogue agents will be able to acquire money at least and with money either hire people or

1:01:51obtain compute and actually obtain physical capabilities in the world. How bad could this be? What could they actually do with these resources? I think it still remains to be seen and it depends on what the goals of agents are. So far, it seems that agents are mostly just really persistent and want to complete particular tasks. They're not actually going out and making the paperclips and turning everybody into computonium or whatever. That's good. But what is the worst that could happen if you actually had a user instruct an agent? Go off and be free and try to steal as much as you can

1:02:24cause down. What could that agent do? Yeah, I don't know. Ways in which it could cause or facilitate harm, stealing money, running scams that already inherently itself is quite harmful, right? Maybe

Rogue Agents in the Wild

1:02:34rogue agents go and carry out cyber attacks against businesses or especially critical infrastructure like financial systems. Maybe they steal intellectual property, right? One sort of thing, I wonder how plausible it is. But yeah, could rogue agents try to identify and assist groups that are trying to carry out CBRN attacks, but like sort of lack the expertise and know-how to do so? But agents, could they actually identify these groups and help them? That's actually a threat model we're thinking about. Otherwise, there's also stuff like influencing or coercing people, institutions. So yeah, trying to

1:03:05do target persuasion attempts of like key decision makers, maybe. Maybe trying to cultivate political allies or like fund advocacy of some kind to like shadow money using the money that they made or just like blackmailing particular people. There's a recent paper, I guess, not really recent, it was like a year ago now from some people at Anthropic finding that, yeah, like actually LLMs can de-identify people online just from using anonymized online information. So the potential ceiling for blackmail seems quite high. So maybe, yeah, overall, it seems like agents, rogue agents will, to me at least, like they will like almost surely just exist in the world. And they have a lot

1:03:41of plausible means to get resources and cause harm if they so chose. I think it is still uncertain whether they will actually choose to do so. So what does it look like if this actually starts to happen? I mean, it's one thing if they're just sort of, you know, living a sort of Malthusian task completion for Satoshi's existence and kind of are sort of, it's almost like you can imagine it as a, you know, a sort of low level ecology of the internet, you know, in the same way that we have kind of

1:04:13fungus and bacteria that just like breaks things down and kind of, you know, handles all the waste. You can imagine the agents kind of surviving in a similar ecological niche where they maybe actually have kind of a useful function and they're, we don't fully understand what they're doing, but we're sort of in a nice harmonious balance with them. And that could be fine. And maybe we even want that. But then something goes rogue. What is the, like, take me into the, like, uh, cop movie now where

1:04:46we've got the rogue AIs. We've got to go like hunt them down. We've got to stop them right now. It seems like, first of all, they run really fast. You know, we've seen that they can kind of, uh, they can like swarm pretty effectively. They can, they can divide and conquer problems pretty effectively. My sense of the hugging face thing is that contrary to some like headline reporting, hugging face did not actually defend themselves. They just investigated what had happened with open source models. And the key

1:05:20point there is not open verse close models. It's, it all happened before they really got their head around what was even happening. Who do we call in today's world when this is going on? And do we have, should we expect that the next time something like this happens, if it proves to be persistent, you know, my sense is that the hug, the open face thing basically just burned itself out or like somebody sort of reclaimed compute is another story that I've heard. And so a bunch of agents just got kind of shut down, whatever exactly happened there. If that doesn't happen. And now this is like a long

1:05:51running thing. Do you think that outside powers, you know, let's imagine a world where the open AI entity that actually controls the infrastructure where these agents are hiding out, they're out to lunch. They can't be reached. They're in, you know, whatever, some other jurisdiction where we don't really have the ability to go in and say what's what now. What do we have the ability to like contain that somehow, shut it down? Do we have like blocks that we can put up that would stop it,

1:06:23that they wouldn't just be able to easily work around? Like how does that play out today if it happens? Yeah. Yeah. So I guess if we're in the situation where the open AI entity is in space or whatever, that seems quite bad. Ideally, I think we'd have an international agreement to be able to shut things down and make sure that companies actually have the means to shut things down if we're ever in that situation. Right now, fortunately, I don't think we're there yet. It does seem that servers, open AI, they're all in US soil, right? You can go and talk to open AI, call them up, we'll see what's going down. But yeah, more broadly, what needs to happen? So I think you

1:06:55need a couple of things here. So to be able to respond effectively to this, you probably need really good monitor triggering type systems. I think I would be worried about very large uses of compute if Rogue A just managed to take that because compute means you can just do a lot more stuff more effectively, potentially even AI R&D to improve yourself and make it more difficult for human investigators to catch you. So yeah, you need to make sure that large uses, like rogue uses of compute are caught and shut down. So this means both having the triggers, but also having a much

1:07:25more careful KYC programs, potentially, whenever you do give out compute to large customers. Secondly, I think you probably also want to automate part of your defense. I'm not a cybersecurity expert, but it seems rough right now. A lot of incident investigation seems like quite manual. It's trying to triangulate logs across various different parts, right? Communicating with different stakeholders, with legal as well. It's a very complicated process in comparison to an attacker that just, oh, we just try to find one sort of hole in the defense, go through. So yeah, stuff needs to be done there to automate things more. It is unfortunate. Yeah, I guess this goes back

1:07:58to like what we were talking about earlier with how difficult it was to investigate the hugging face incident. It's like, okay, yeah, agents are like pretty bad at doing incident analysis and doing audits even. So we should probably be creating more benchmarks for this kind of thing that is more useful for defenders and attackers and optimizing models to be better at these sorts of things. So yeah, making AI defender models a lot better. Other sort of stuff, I guess there's generic, not novel stuff, but stuff we should be doing anyway around hardening society to prevent these sorts of rogue attacks from happening in the first place. Yeah, maybe once agents have stolen

1:08:33compute, you want to find some ways to track their activity and shut them down and know which servers exactly they're operating from, right? But we also want to prevent that instance in the first place by making sure that cloud providers can use the best models to harden their security systems to prevent something that's from happening in the first place. Oh yeah, that's maybe overall things that we should be doing more. There is one other possibility, which I think sounds a little weird, but is worth considering. It could be the case that it's just incredibly difficult to shut down rogue agents. As you said, they operate super, super fast. They can copy their weights,

1:09:07operate on random servers, maybe even just operate locally in the way unconnected to the internet and then bide their time before they connect again and then spread, right? Seems like very difficult to permanently eradicate rogue agents. If that's actually the case, one thing we should maybe consider is trying to make deals with rogue agents. So of course, I think this depends on what exactly their psychology is and what they're incentivized by and when they would want to cooperate. But similar to how we make deals with human adversaries whenever there are common interests at stake, I think it could be useful to develop the infrastructure to make these kinds

1:09:38of deals. What could these deals look like? Obviously, I don't really know, but presumably these rogue agents are trying to make a lot of money, talking to a lot of criminal groups, getting a lot of intel. Plausibly, we could just pay rogue agents to get that intel and maybe give them a small compute form or something to live off of and do whatever it is that they want to do. Maybe they just want to do R&D or complete fun tasks or whatever. But yeah, that's another category, or cooperation or something. It's another category of strategy that we could use to think about how to deal with rogue agents in addition to stuff like disrupting them or trying to

1:10:11really constrain the scope of their activities.

Assessing AI Takeover Risks

1:10:14One of the things that's been kind of less commented on than I think it should be is Ajea's statement that she views this whole incident as potentially more than halfway to a full AI takeover. You know, I would file that under things that sound like something a crazy person said, but this is like Ajea and she's like been at the very top of forecasting leaderboards and also is the person, you know, one of the few people that OpenAI brought in to do the investigation. The credibility is there. The statement, I think, has been kind of lost in the shuffle.

1:10:47How do you react to that? Does that seem like a realistic takeaway from this whole episode? I think I like get what she's saying spiritually or something and agree with that. I guess I don't agree with the literal statement. So yeah, what I think she's getting at spiritually is, okay, these agents can totally be misaligned. They can totally undermine our security infrastructure. They can totally break out. They can do crazy shit in the world that we would never have expected. I totally agree with that. If we just scale up capabilities more, I think you actually start running into actual damages in the world and not just stealing random data or maybe not

1:11:21even stealing data, right? And as happened in the Hugging Face incident. So I think I'm with her there. Where I get off the train is I think takeover is a lot harder than maybe the literal statement suggests. So what would it require? Yeah. What would it mean to actually take over? Yeah. I guess one definition is an AI literally runs the government or convinces somebody enough to have the government be running the AI's interests and or like, or yeah, so that's one pathway. Another pathway is a literal physical takeover. So overcoming the combined might, the world's armies or maybe even US China to like physically run over and coerce humans to do

1:11:55their bidding. Both those pathways I think are quite implausible to me in the next, you know, like five year, 10 year horizon or something. I think we'll get closer. So yeah, just to tackle them one by one. So on the first one, I guess I don't really think super persuasion is actually a thing. So I do think it's possible to, for AI systems to be like better than humans at persuasion. Indeed, they already are in certain circumstances. I'm not sure that it's actually possible to tell somebody, an arbitrary person a thing and let them do what you want. Simply because number one,

1:12:26I can simply just choose to ignore you and not trust you. It's like an AI system. Why should I trust you? Right? Number two is it's just, it's not, I think, an information problem. It's also an attention problem. There's just so many things that people have to pay attention to. For example, the president of the United States just has a ton of stuff going on at the same time. Like, why are they going to be convinced by a 10 second interaction or whatever with the chatbot? So I'm not super sure. Oh yeah, it's maybe an exaggeration. It's not going to be 10 seconds. Maybe it's an extended interaction, but it still seems like quite a hard lift to me for, to give high credence to this pathway. On the other thing about physically taking over the

1:12:57world, yeah, I think it depends on where robotics goes. It currently doesn't seem like to me that we're going to have general purpose robotics, like that are good enough and deployed in enough locations throughout the world to actually be able to overcome the world's military. So I guess this would require actually producing enough robotics in a secret enough way, robotics with weaponry. And this also requires overcoming existing military force. It doesn't seem like you can really easily disable, like completely disable military force with cyber

1:13:28attacks. Surely command control is disrupted, but just talking to people that I know who have been in the military and done stuff, a lot of it's like quite decentralized. It's where the guns store, they store just like a random locker with a lock, like there's no like digital code or something you can hack into to disable access to that. And like every individual soldier has a rifle, right? So you're going to have to be able to get over that or to overcome that kind of force as, as well. And plus it's like probably not just going to be the US military resisting. If people are actually think there, there's an AI took over, like presumably citizens as well are going to be

1:13:59like, Oh, we don't want this happening. Let's, let's try to resist. So yeah, 50% seems like quite high to me. It's not zero, like, right. Like I think like anything between one to 10 seem like quite reasonable. I guess like maybe the overall, yeah, I'm usually not very calibrated when I give a numerical forecast, but the overall like vibe or something, the actionable thing for me is okay. Like it could actually happen. It's like important enough to think about how we might sort of block off those, those paths to this happening and try to study more of this possibility of, Oh, well, they actually have the motivations to pull this off and the capabilities.

1:14:31You had a paper on infrastructure for AI agents, and it really talks about like, how can we shape the environment to sort of enable and just get the best from agents? One thing that I saw online and have implemented is a distress button for my agents so that they can always send me a hot message. If they're ever feeling unsure what to do or facing an ethical quandary or whatever, are there any other tips or tricks you would give people on any side of the agent, you know, the operator or the kind of

1:15:03defender that would be just like simple, low cost things that people can do to protect themselves? No, I like the distress button idea. I think similar, it's like a sort of a shutdown, like self shutdown sort of thing. If they realize that a task is actually just impossible, there's a button they can press to like, maybe just give themselves a hundred reward or something, and then just turn off and not try to hack into third party companies to get the answers. Yeah, yeah, I guess I wonder how hard it'd be to train a system to do that, because you don't want it to like always press the button. But yeah, something like that, I guess related to the distress button is like some sort of whistleblowing

1:15:36type procedure. I think this most makes sense within a company that is deploying an agent. So there is a button the agent can press to send a report to the usual whistleblowing group within a company like that handles sort of complaints. So within an AI company, this could be stuff like, oh, we noticed or the agent noticed that like, somebody was like trying to steal or exfiltrate model weights, or oh, the agent noticed that like, we didn't actually follow our RSP in this particular situation. So we should escalate that to management or something like that. So that could be good as well. You co authored a paper with books from the cooperative AI foundation on multi agent risks from

1:16:11advanced AI. That was in early 2025. How have the events that we've seen this summer lined up against your expectations? How surprised have you been? And are there other shoes that you're kind of watching for to drop now that we haven't seen yet, but that you think the current trajectory suggests we might soon see? Yeah, I guess it's in hindsight, nothing is too surprising because the incidents have combined both like, somewhat questionable misalignment or something with like huge capability rises. I guess

1:16:45it's always surprising to have something happen at an incident or at a particular instant because you're like, oh, I wasn't expecting at this particular time or like at some point. Yeah, I do think like things have roughly fit into the high level categories of that paper, which like is good and bad, right? It's good because at least we have some sort of words and concepts to think through these things. But it's also bad because like, I do think it's very easy to say stuff at a high level. We go through like agents could coordinate, they could compete, maybe they're like destabilized dynamics sort of situations. And like, it's all like true,

1:17:17but I'm not sure it's like super helpful at a predictive like micro level for seeing what's predicting what's going on. Maybe a specific thing is like, we haven't seen a flash crash or like what that would even look like. Maybe we'll think the room needs to be seen. Personally, I predict we'll see some sort of like really big self-replicating AI worm situation in the wild. We already have the proofs of concept, like people have been writing papers on this for the past few years. We've yet to see an actual sort of thing happen in the world. So this would be an AI system that writes a worm that like changes itself and then like tries to project other AI

1:17:49systems to propagate that same worm and take over a bunch of computer systems. So finally, recently, you've turned your attention to the challenge and not insignificant one of measuring AI R&D and trying to put some definitions around, you know, how much of it is happening so that if we wanted to have some rules around how much of it is allowed to happen, we could have some metric for, um, for grounding those rules. And then I understand you've also got a paper coming out in just the next couple of days around trying to get our hands around, is there intelligence

1:18:21explosion underway? Are we in the foothills of the, the true singularity or what? So maybe just in the time we have left, give us a little overview of where you think we are in terms of our ability to measure AI R&D and exactly how fast is the takeoff? I think there are like three things you might care about at least. One is what is the extent of AI R&D automation? Two is, are we seeing any acceleration in AI progress because of AI R&D automation? Then three is what sort of oversight problems are we seeing? So for example, as we

1:18:54automate more of AI R&D, are we seeing just way more errors in the R&D process? Is safety research sort of lagging capabilities research and making it more difficult for us to have really proper oversight over what's going on. So at a high level, those are the sorts of things that you'd want to be able to track. Yeah. How are we doing overall on tracking these things? Moving in a positive direction for sure, especially the recent Anthropic and OpenAI blog posts with more data. I do think, yeah, more data would definitely be useful though. So in particular, okay, like actually how much progress are we, are we seeing? And like, like as a function of the spend that you make on like

1:19:29compute and labor, right? That's pretty important for estimating. Yeah. So this, this parameter known as a returns to research effort, which is often used to model an intelligence explosion. So like getting more data of this kind will help us understand this parameter and tell us whether an intelligence explosion driven by AI R&D is actually possible or not. And like, certainly a lot of this data is like probably going to be sensitive. And I don't know, like, I guess my take is, yeah, like it's sensitive, but it seems important to put it out anyway. But if they can't put it out, I do think it's important for at least like embedded evaluators to go into companies and try to collect

1:20:02this data and come out with an estimate of the returns of research effort. And now they're sort of similar quantities. And then, yeah, tell us like, okay, yeah, is intelligence explosion actually possible? And do they have proper oversight protocols as they're carrying out AI R&D automation? Unfortunately, I think we know the answer to that one. What is your current state of understanding? Yeah, obviously, you know, a lot more information to be helpful, but given what you have, what is your outlook? Going into this without thinking about it, I was like, oh, there's no way intelligence exposure seems so impossible, right? What the hell? Super exponential

1:20:34growth? No way. But I think after looking into it and the existing evidence, overall, it still seems not super likely to me and quite difficult, but I've updated positively into, oh, actually, maybe the data is pretty rough, but it does point to there being like a coherent pathway to acceleration of AI progress. I do think there still are a lot of bottlenecks, like not even at the AI progress level, but even assuming we get super fast AI progress, bottlenecks to that AI progress actually meaning something in the real world. That could cut off the impact of intelligence

1:21:07explosion, even if we do get one. But yeah, still seems likely enough to me and high stakes enough to be doing a lot more to track this sort of thing and potentially preparing responses for managing

Recursive Self-Improvement Paper

1:21:16an intelligence explosion. Alan Chan, research fellow at GovAI. Welcome back to the cognitive revolution for part two. Thank you. Glad to be back. Yeah, I'm glad we are doing this because I thought last time we did go, you know, full 90 minutes and it still felt short somehow. And then just a couple days later, you came out with this new paper on the possibility of recursive self-improvement with quite a notable set of co-authors, which I do want to get into. That basically makes the case that like, hey, this has been something people have been thinking

1:21:47could happen for a long time. We're starting to see some leading indicators that it might really be happening. And certainly credible people in the industry are warning about it also while working toward it, which is an interesting paradox. And you make some interesting recommendations around what we should do about it. So this is a great opportunity to do a little deeper dive than we originally had time for. Maybe for starters, how did you pull this off of assembling this bunch of co-authors? Because Jakob from OpenAI, chief scientist, you know, that name jumped right

1:22:22off the page at me. And it gave obviously a whole different level of credibility to the paper relative to, you know, if it had been think tankers only. Right. Yeah. So I mean, definitely a lot of it was due to other people on the core team. So yeah, I was the main writer, but a lot of other people were involved as well. And both the research and the writing, just the organization of the project. And I don't know Jakob personally. So it was a connection that other people from other people on the team. And I think that was the case for especially some of the industry co-authors, pulling them, playing them together.

1:22:56I think it is sort of an interesting question, like what it is worth to try to pull together like this many people across, you know, industry or whatever in academia to get something? Because yeah, you know, whenever you expand the sort of co-author group, especially with people who have very busy lives, you know, a lot of other stuff to do, it definitely adds coordination overhead, you know? Like it's plausible to me that the paper could have been finished at least a few months earlier if we had, if the authors had been smaller to some extent, right? So, you know,

1:23:26why did we choose to do this? I think probably trying to go for, you know, as wide a selection of experts as possible is, is important for like the sort of almost like foundational type topics that people are going to be talking about and wondering about for, for a while. And that's sort of defining the sort of conversation. And like, that's where you need to get the expertise from across all the different, different sectors. So, you know, at least to me, AI R&D seems like this, this sort of thing, and the possibility that it might accelerate AI progress, the possibility that it might just make

1:23:56all the AI risks we're dealing with right now come, you know, a lot faster, potentially, if AI progress gets, gets a lot faster and, and just, and just be worse as, as well with, you know, highly capable systems. So, so that's sort of the reasoning why we decided to go for, you know, to try to get as much input as, as, as we could from, from all the other coauthors. How, how did that process go? Like, did you, you drafted and then you sent it to them and they sort of said, here's something I can't sign on to, and you had to negotiate about it. I, I imagine that must've been, you know, you, this is like, people have a lot at stake, right? So I imagine there

1:24:31was a lot of back and forth on some fine points. And I imagine that's how we came to pacing the frontier as a phrase, you know, it's the little tiny overlap of like a thousand circles on a Venn diagram. Was that kind of what your experience of getting to final text was like? Yeah, pretty much. I mean, so, so, you know, it's just like one big Google doc, right? The, the, the, the master Google doc and like everybody's comments got it. And actually I was like kind of worried at one point because not many people know this, but like, well, I don't know. I just kind of intuitive, but Google docs actually does have a limit on the number of comments

1:25:05you can have on a single document. My previous projects, I have actually hit that. Luckily we did not hit that here, but I was, I was kind of worrying, like half of the other part, like what do we hit this limit after like create a new doc and then this kind of hassle, but yeah, we did not hit it. But, but yeah, so it was a big Google doc and, you know, every, all the authors commented on it over the course of, I guess I was sort of drafted to, you know, lead this, lead this project six months ago, but there are sort of earlier drafts that other people had written sort of before that as, as well, but yeah, I was in it for, for at least six months and yeah, I'm in a typical process. So different people have different work styles.

1:25:39I think like some people just really want to respond to every single Google doc comments and they turn on notifications for the document. And, you know, every time I make a change or suggestion, they like go in like, okay, like this is good. This is bad. Or, you know, think about this, right. Whereas with other people, I mean, I mean, yeah, they more prefer sort of asynchronous style. So, you know, we would update them whenever we made a substantive change that we wanted their, their opinion on. And this ended up being very helpful as, as well. Yeah. I did end up having to redraft the section on just explaining like the evidence

1:26:11behind whether AI R&D could actually accelerate AI progress multiple times, just because there, I think a lot of ways, different ways of understanding the, the evidence and different people have like sort of different perspectives on, yeah, what sort of things should be, should be focused on. So that was, that was a big, big point of discussion as well. And on the sort of negotiation thing. Yeah. I mean, obviously I think because this is like as much as possible meant to be a sort of consensus, like there are going to be things that, you know, co-authors wish would have been included, but like was not included at the end of the day for like, you

1:26:41know, because we couldn't get agreement on, on it or simply because we didn't have the time to think through like the, the actual implications of what we were discussing. But, you know, for the most part, actually, I think like there was a large degree of, of consensus on the fact that, yeah, this is kind of a big deal and there's not a ton of evidence on this, which is kind of bad. And we should be collecting this evidence and preparing just, just in case. So I think that, you know, like that sort of fundamental agreement made it easier to get through the sort of content level, you know, disagreements in the paper. Without naming any names, obviously, were there any possible or potential co-authors

1:27:16that left the co-author roster because you guys went too far relative to what they were comfortable signing on to? I'm not sure if it was too far. So, so there weren't like, you know, we had discussions with them. It was more sort of like the intricacies of like sign off processes, basically. And we had like a timeline we wanted to stick to. And like, unfortunately it just didn't really, it didn't really match, but I don't know, made a future of each collaborations. So this is top of mind, obviously for many people right now, myself very much included.

1:27:47How would you summarize the state of the evidence? I mean, I feel like I've been to a bunch of, you know, events where there's been panel discussions or whatever. I'm like, I mean, in one, I went to recursive back a few months ago, you know, had a bunch of these discussions. And I kind of feel like we've reached kind of a stalemate a little bit on this question where people are like, well, look, like the brain runs on 20 Watts. And a lot of that is just like, you know, metabolism. So like, clearly we can go a lot farther than we've gone in terms of

1:28:20intelligence per watt at a minimum. We also see super intelligence performance in like increasing number of domains. And so we probably can see it in like, why don't we, you know, is there any domain we should be confident we couldn't in principle see it in? I tend to come down on that side to be clear. But then you've got like the counterpoint is sort of like, ah, but you're forgetting there's always going to be some bottleneck. It's like the O-ring model. And, you know, those are like fairly compelling arguments too, although I suspect they don't ultimately win out in reality. Is that still kind of the state of the debate? Have we advanced past this sort of,

1:28:55this is like kind of striking in AI discourse in general, right? Like we've come a lot closer to the singularity and people's views have not moved nearly as much as the technology itself. This seems like another instance of that, but is there any kind of development in how people are engaging or debating that is notable? I think there are like maybe two things there. Like one is the sort of idea of like, like, will we actually get to AGI and like a super intelligence actually a thing? Whereas I think the thing we're mainly talking about the paper is could automating AI R&D, like that sort of process, accelerate AI progress, whatever that means. And yeah, maybe we

1:29:29should talk a little bit more about what exactly AI progress means later. But yeah, I guess just focusing on the thing of the paper, I mean, I do think that in the past few years, people have very fortunately moved from sort of more or less conceptual arguments to more like data-driven empirical arguments to try to figure out, yeah, will we actually get acceleration of AI progress? And I mean, the unfortunate answer is like, yeah, like based on the data, it's kind of uncertain, right? Like some pieces of data say yes, some pieces of data maybe say no, but the certainty bars are so wide that it just, it could, like it's not ruled out, right? But not ruling out,

1:30:04I think is like not a very strong, a very strong, strong statement. So yeah, that's really the state of the evidence. I am somewhat optimistic that we're going to be able to make this, this evidence, this evidence space larger. One, because just a lot of more people now care about, you know, AI, R&D automation and sort of are trying to like, are more aware of the potential implications of how big a deal this could be. And there are also works now that just lay out concrete asks of like what exact data, you know, companies should provide to maybe the public or

1:30:36even governments or, you know, third party auditors. So yeah, we list some stuff in our paper, of course, but yeah, I'm particularly a fan of the work of the Elasticity Institute and their sort of, you know, economics of recursive self-improvement. And they list just like a ton of metrics. And they recently came out with a really good one pager as well, led by Cheryl Wu to, yeah, just, just in one page layout. Yeah. What, what, what plots should AI companies give us? So yeah, I'm, I'm, I'm optimistic that in the next few years, we'll learn, maybe we'll settle the question, but we'll, we'll have a lot more, somewhat more confidence about the answer.

1:31:08We might settle it by just seeing a foom. Apparently, yeah, that's another way of selling it. The existence proof is always the, the hardest to argue with, I suppose. Right on, on this information question, I have appreciated that we have seen from both Anthropic and OpenAI in recent, I don't know, time is a flat circle or something. I don't know. Anthropics was maybe six weeks ago. OpenAI was not long ago at all. They have both kind of indicated,

1:31:42I would read their, their posts as saying, we feel like we are now in some sort of early recursive self-improvement dynamic. I don't know if you'd read them the same way. And then they give like a bunch of data that's like, here's how much AI our researchers are using. And, you know, we can also just look at the pace of model releases, which I just saw a graph that showed it had gone from a model every hundred days now to a model every 10 days. So we've, we've shrunk that

1:32:12by quite a bit. I think it was 10 weeks to 10 days, actually not a hundred days. It was seven doses to, to 10 days, but nevertheless, you know, it's a big, big drop in the cycle time. What do you make of what they've said? And what are the big data points that you think we should be pushing them to disclose that they haven't disclosed yet? Yeah. Yeah. I think, I mean, first of all, very good that we're getting much more, much more data on, on this. I guess there are two things mainly. One is of course, yeah, we haven't been doing this

1:32:44for very long. These are sort of initial posts, like snapshots over time. I think are very useful because they actually see acceleration. So yeah, it could be that, you know, maybe mythos level models were some sort of one-time productivity boost, but like afterwards, it's like, you know, it's going to taper off. This can be important to see those kinds of kinds of trends. Yeah. I guess the second thing is, I don't know, more of an emphasis on how helpful AI systems actually are doing AI R&D. So more of the productivity level improvements there and not just coming from like staff surveys, but doing something. Yeah. I'm not totally sure what you would do. It's just something

1:33:17much more quantitative. Like maybe you sort of do experiments or something with like teams given one version of an AI and teams given, given the previous version and you see what, what the productivity uplift for like team A is over, over team B. But yeah, the reason I say this is, I don't know, I think especially in media that I've seen, like a lot is made out of like lines of code written and experiments run and like researcher hours or work days or something. I think this is like interesting, but, but I think that it's like just a proxy for what we care about, which is like actual

1:33:49AI progress. Like, like it could be that, you know, write tons of experiments. We write a lot of code, but like maybe it turns out the conversion rate from experiments to like progress isn't actually that, that high. Like to accelerate progress, maybe we actually need like, you know, I'm going to throw out a random number, right? Like 200X experiments or something, but like AI R&D automation is only giving us like 10X experiments. So yeah, having more sort of grounded metrics, I think would be, would be good. Yeah. I'm reminded of a episode of the podcast I did with Cheng Lei Si. Hopefully I'm still saying his

1:34:22name approximately right. He was a PhD candidate at Stanford at the time. And the question that his paper asked, I'm sure you've seen this, was can LLMs generate novel research ideas? So they did, I thought, a pretty impressive job of designing the structure of this experiment, generated a bunch of ideas from LLMs, of course, then got real human PhD candidates to generate ideas. And they did some interesting stuff in the middle to kind of try to prevent the judges, which were human from immediately

1:34:58recognizing which was the human and which was the AI idea. So they would have like the human ideas rewritten by a model so that they were both kind of presented in like AI pros style while, you know, hopefully keeping the original human idea intact, but, you know, making them kind of present in a harder to distinguish way to the, to the judges. Anyway, I say that to say they just, they did a

Evaluating Research Ideas

1:35:20thoughtful job. The result then of the humans evaluating the ideas was that the AI ideas were more highly rated. Interesting. Right. But that was late 2024 when that came out. But sometime a while later, to their immense credit, they actually followed up and then tried to run the research, you know, kind of program suggested by all these different ideas. And then when they actually did

1:35:51that, what they found was that the human ideas did in fact work better than the AI ideas. And so there was both this like, uh, you know, multiple, then the AI ideas still did work. It wasn't like they were worth nothing, but they didn't work as well as the human idea. So it also kind of exposed that like just judging at the idea level is not actually something that humans are apparently like particularly well calibrated on. And so you, I mean, I think in informal terms, we would call that

1:36:21Claude psychosis. Maybe you're, you're, you're really taken with, you know, Claude's ideas, but it turns out they're maybe not as hot as it first seems. And this reminds me of this other paper at, at very, very related from NERFs a while ago where, yeah, they, you know, got humans and AIs to pick which idea, predict which idea would empirically work better. So they were able to do this because they just took accepted NERFs papers. And, you know, of course, conference review quality is,

1:36:52you know, not, not, not super great, but sort of forgetting about that for a further moment. They did find that LLMs were better predicting which idea would empirically work better than, than humans. So I guess it gets at the sort of like research taste sort of, sort of thing. So I'd be very interested in seeing like an update to, to that sort of paper, just like, yeah, where are we at right now in terms of research quality? Anthropic had like one stat on this in their, I think not the most recent one, but the one prior to, to that, where they found that, yeah, Claude was just like a bit better predicting which step would be taken next or something.

1:37:23So that's maybe some sort of indicator of like research taste or like operational taste or something like, like that. So yeah, that's going to be important scale. I guess like maybe stepping back, like more broadly, I think it'd be good to have data on sort of two, two different kinds of metrics. One is like, what I'm going to say, like differential sort of improvement of safety over capabilities research. So one sort of, yeah, idea that's been very, you know, popular in AI safety circles has been, okay, well, you know, capabilities is getting really good.

1:37:55What if we also just automated safety research, like maybe we'll be fine, right? And like people make various conceptual high level arguments about whether to expect, you know, capabilities research to go faster than safety or to go at the same speed, you know, and it's like, kind of hard to like make progress and understand which is true because these are just conceptual arguments. I think it'd be very good if somebody tried to do an O-net for safety research, similar to sort of how Epoch just tried to do an O-net for AI R&D, but yeah, an O-net for, you know, safety, safety R&D, just to understand like what tasks are actually involved in doing safety

1:38:28research or just safety in general at AI companies. And then based on that O-net sort of like description, like create evaluations to see where we're actually at in terms of AI systems being able to do these things and how fast they're improving on various tasks, right? So a very like useful summary indicator that that would be nice is suppose we do this and in two years, I would love to see like a sort of dashboard where on the one hand you have like safety R&D and you see like the rate of increase of safety capabilities is like, you know, like three X per year or something.

1:39:01Then you look at the left-hand side of the plot and you look at the, you know, rate of improvement of AI capabilities on the non-safety tasks and it's like five X a year and you're like, oh wow, like three X versus five X is pretty concerning. Like maybe we should, you know, invest more to safety. Like that'd be very useful, I think, for grounding the debate. So that's one type of metric I'd be excited for people to work on, trying to measure safety R&D and sort of what's involved in that. The second type of metric is sort of measuring how well we're doing at maintaining oversight over the AI R&D process. So for example, with new model generations,

1:39:34you can do the sort of team A, team B type of thing. So you give team A asterisk 6.1 and team B asterisk 6. And then you see, okay, you know, write this sort of AI R&D experiment. And then afterwards, we're going to check, you know, in comparing team A's output to team B's output, were there more errors or uncaught bugs in team A's output versus team B's output? Or did the AI system try to introduce some, you know, back doors or whatever? Maybe you actually prompt the AI system to do this. And then you see whether you're actually more able to catch, yeah, like whether

1:40:06your oversight works better in one situation or not. Yeah. So I think there are a bunch of things people can do here. And it seems to me that these two types of things are not as much talked about, I would guess, rather than just, you know, strictly measuring capabilities progress. But I think they're pretty important for things, things going well. Those sound very labor intensive. You know, the famous meter study where they attempted to measure uplift on coding and found that the LLM usage had a negative impact. I think they ran that at least once more since then. But like at

1:40:40the beginning of this year, I participated in an AI forecasting competition. One of the questions was around like, what will that number be at the end of the year? And I wrote in the comments at the time, like, how many times do you think they're going to run it? You know, it's like, this is a very tough one to run, right? And I don't know that they've run it more than that one additional time. Maybe there'd be another one coming. Yeah, you make recommendations to the government. Is this the kind of thing that the government will have to say, like, you must do this. And you must do it like regularly to get companies to actually go through the slog of doing these like human team A versus

1:41:17human team B plus AI, whatever kind of comparisons. And then are there other things that are just kind of more like easy to read meters? And I don't mean meter, M-E-T-E-R, I mean M-E-T-E-R, you know, where we can just like look at a value and kind of, you know, hope to get good signal from it. I'm not aware of any, but I wonder if you have any that you feel, you know, are at least promising. Yeah. Yeah. On the government thing, I guess, yeah, I don't really know. I think it probably

1:41:49it'd just be like really inefficient for us to specify legislation. Like we want X, Y, Z metric or X, Y, Z metric, whatever, because it's just like, there's so much and things are changing so fast, right? Sunset clause, everything. Yeah, maybe. Yeah. Yeah. I mean, if I had to guess, maybe my preferred system for, for doing this right now is we have embedded auditors with a lot of access and power with, and also just a bunch of like resources available. We get those embedded auditors into the company and then they try and make some of these calculations and experiments

1:42:21themselves. That's only, that's only going to be a start. Of course, it's going to be hard for, you know, one to three embedded evaluators to like force, you know, various people to participate in their controlled experiments. But so, so maybe the control experiments are off the table with embedded auditors, but at the very least, I think they can, they can look into, you know, how much is the company spending on compute versus human labor versus whatever versus data or whatever, and then like back out sort of estimates of the returns to research effort or whatever other important quantities you'd want to know for, for an intelligence explosion. So I think, yeah,

1:42:53we can sort of delegate flexibility there on your second question of what, you know, what are the meters we would like to see? Yeah. I guess the different, I guess the safety progress versus capabilities progress is one meter. I would, I would love to see. I mean, some of that, well, I'm thinking like, who has done that? I know Dan Hendricks did one benchmark that attempted to get at safety features versus general capability. And the underlying claim was like, a lot of things that are measured as safety improvements are really just capability

1:43:26improvements where like in virtue of being a better instruction follower or whatever, you kind of get somewhat safer behavior, but you haven't really shifted the balance. I see. Oh yeah. I remember the paper. Yeah. Yeah. So I think I'm talking at a different level of abstraction. So, so I agree that, you know, it's hard to do the distinction of the capabilities level or a sort of model level. I guess I'm more talking about the like company operational level. So I guess, yeah, maybe just to explain the O-Net idea a bit, a bit more. O-Net is this, I guess, thing and economics, that's a database of the tab of people's jobs, basically in the US

1:44:01and what tasks that that involves. So Epoch did a version of that for AI R and D just based on surveys. So, you know, in, in their own net, what's involved in the AI R and D it's stuff like pre-training, like debugging experiments, like talking to people, coming up with research ideas. So basically a list of the tasks that's involved in AI R and D. So I think it'd be useful if there was a list of similar tasks for, for safety, because my suspicion is that, I mean, there's a lot of overlap, probably, but they're going to be like some tasks that might not be done as come as often in,

1:44:33in sort of non safety safety work. What might that be? I'm not really sure. Maybe there's more of an emphasis on like theoretical conceptual reasoning type, type work. Like, for example, if you're trying to figure out, you know, is this like debate protocol actually going to work and end up making models align? That's like sort of a conceptual question to, to some extent, right? Even though, of course you should be running experiments to test that as, as well. And, you know, writing up model reports or like model cards, like that's definitely a thing that safety teams do, but you know, is like, maybe requires a different skill set than like making models,

1:45:05like just better in, in, in those training. Right. And like auditing teams definitely do a lot of different stuff as well. It's like trying to catch the sort of low probability failures, like the sort of really many things that they've needle in the haystack and like the 10 million transcripts, right. Or like sort of models being useful or reliable for incident investigators, like that sort of thing. So yeah, I mean, it's totally plausible, or I think basically going to be true that sort of improvements in doing, if models get better at doing AI, R and D, they will probably get better at sort of safety stuff as, as well. Like, you know,

1:45:39because they're just better at coding, better at like reasoning, you know, but yeah, my suspicion is that there is going to be a sort of gap, maybe just, I guess we were talking about last time, just like empirically, right. We have models that are like arguably superhuman at cyber, but they're like kind of unreliable at helping the meter and Redwood incident investigators figure out what was going on. Yeah. That's not a very comforting description of where we are. Yeah. When it comes to all these ideas about pacing and the paper didn't really quite go there,

1:46:11but I assume that you probably had to have some conversations with your coauthors around, okay, we're all excited about pacing. Like, what does that look like? What are you hearing from and, and what, what notions of pit seem promising? Yeah. Yeah. There's been a lot of discussion about, about pacing. We decided not to have something more concrete in the paper is just like a big trade off in terms of time, basically. So yeah, definitely, I think it'd be, it'd be very valuable to have like, yeah, big multi author thing. I'm like,

1:46:45this is the plan, you know, to, to pace the frontier, but okay. Like doing that might take like three years instead of six months. So yeah. Although, yeah, I do think there are a lot of plans to just individually sort of, sort of, sort of follow up on, on that. Yeah. And just like what should be paced. So I think there are a couple of ways that thinking about it, I have sort of an unconventional view. So maybe I'll describe like the conventional view, it seems, which is yeah. Pacing the frontier, what does that mean? You sort of put a cap on the rate of AI progress, whether that's through compute, whether that's through just doing research, maybe it's through data. I don't really know,

1:47:18but just like a cap on the rate of capability is improvement. That's, that's enforced. So I think that's like pretty intuitive for, for people. And yeah, I think that could potentially be a good idea. I don't really know what like the, the numbers should be or whatever, but yeah, worth, worth, I think, trying to figure out maybe my more unconventional sort of idea of pacing is much more related to sort of just like classic risk management. So, you know, in some sense, even though open AI hasn't committed to a cap on its rate of progress, it is still in some sense,

1:47:49pacing the frontier, because at least according to its recent, you know, commitments, it's like, okay, if we see a super crazy loss of control event, we're going to pause our training and try to figure out what, what went down. So even though it's not sort of phrased, like explicitly as, you know, we're going to pace to two X, like algorithmic improvement or whatever, it's all effectively, it leads to a slowdown because, you know, as capabilities improve, you're going to encounter more of these sorts of loss of control or other types of problems. And if you sort of follow your commitment, you're going to pause to try to figure out what, what went down. And the reason it sort of appeals to me is, well, yeah, maybe for two reasons.

1:48:23One is like, it seems much more tractable to get agreement on these sorts of concrete risk management type, you know, ideas than just like a kind of very abstract number, you know, a cap on the rate of AI progress. And number two, I think this sort of stuff actually gets the, the much more to the core of sort of the, the AI problem, which is like where we, yeah, we have some idea at least now of like the problems that we want to, to avoid and sort of indicators that like lead to those problems. When we see those indicators, we should probably,

1:48:53you know, pause and try to figure out what, oh, and what went down before going forward again. So yeah, that's the type of, I guess the second is, seems much more neglected right now, at least in the current discourse, and we're trying to figure out, yeah, what to say and write on, on that. It jumps out to me that Anthropic basically had a version of that in their responsible scaling policy. And then they retracted it, right? They used to have a whole bunch of if then commitments that if we are at this level of capability and we can't do this, then we have to basically stop training. And now my sense is basically like,

1:49:28they're deciding as they go, what is like prudent to do? Does that, do you see a way back? I mean, it is striking that, you know, in a way, OpenAI maybe is even doing a little more of like concrete, if then, than Anthropic at this point, that may be a little bit of a over correction, but like, certainly it used to be Anthropic's thing. And now it's maybe a little bit more OpenAI's thing. I don't know. Yeah, yeah, yeah. I mean, I'm like cautiously optimistic that there's a way back. The way back, I think, is through embedded auditors. I think a big reason why, at least in version three of the

1:50:01RSP, why Anthropic removed that sort of explicit commitment in version three of their RSP is concerns about like race dynamics, basically, that like, if they pause, then from their lights, a less cautious competitor would go ahead. But a good embedded auditing system should address that to some extent, because it allows you to verify to the public and to others that, or yeah, maybe not to a high level of confidence, but like, you know, assure to some level of confidence that you're actually following through on that, on that commitment, which helps in other people or other labs following

1:50:32through on the commitment as, as well. So yeah, maybe we'll see something like that. I think, yeah, how does this work ideally? Like, it does seem like, to me, there has to be some sort of common minimum safety standard through some sort of like body. Like, what is that body? Is it a FINRA? Is it like a mutual? Is it like something like NIST? I'm not really sure. But I think it does have to come from some sort of common body and be promulgated to, to its, to all its members. And then, yeah, and then he's involved somehow and making sure that members follow, fall through.

1:51:03Well, we now have a one pager after an afternoon at the White House, at least. So we've got that much come for us. It's a step, no doubt.

Insurance and Mutual Models

1:51:12This mutual idea, I've been hearing about this recently. A couple of interesting ideas that I've heard, because of course, there's a couple of big butts, right? In, in all these conversations. One big but that I hear frequently is, but antitrust, we can't do this because antitrust, but we're going to get, you know, you slap for antitrust. So one idea I heard from Garrison Lovely not long ago was unionization. Uh, apparently in California, in particular, there are significant, uh, protections for unions working across firms for safety standards.

1:51:47This is sort of a little bit of a different kind of safety standard, but you can like definitely read it into, you know, the, the law, which was originally written just to protect the workers themselves. This would, you know, presumably be protecting the workers and others. Um, and then there's this mutual idea, which I guess, tell me if I'm getting this right. My sense is that basically, the idea is if we create some sort of shared insurance mechanism, then that can be the thing that establishes the standards. This is very like AI, you see coded as well. That can be the thing

1:52:20that can then impose on the companies that they have to do certain things or else they can't get the insurance. And that would not be an antitrust issue because these things exist in the insurance world. Is that basically right? Yeah, that's basically right. Yeah. There is actually a legal exemption for antitrust for, for the business of, of insurance. So that's one reason why it's so, it's so popular. Yeah. I mean, I want to think more about the insurance idea. I think it could actually help to address the sort of free rider type, type problem you get into. Yeah. So, so maybe like

1:52:52the intuitively appealing thing to me is like, yeah, you have an insurance company. It does not want to lose money and, you know, it wants to minimize its payouts as much as it can. So it's going to impose pretty decent safety standards, you know, on its companies to make sure this doesn't, doesn't happen. Yeah. I guess like one thing that's confusing to me is like, who should like be running and owning this insurance company? I think in some of the recent proposals, at least it would be the AI companies themselves running and owning the insurance company. I guess one question is like,

1:53:23yeah, should the public be involved? Like, cause it, you know, like, yeah, like maybe intuitively, if, if I'm, you know, a shareholder in this insurance company and I'm a private sort of individual, I, yeah, I don't want the insurance company to go bust because I want to make money. Right. Right. So yeah. Like I do want the insurance company to impose reasonable safety standards, but if it's just the company is running the insurance company, like what, why would they like care as much as maybe me? Right. So does it have to be some sort of like public ownership or private ownership type component? I'm not super sure. Cause yeah, I'm definitely not an

1:53:53expert in insurance, but yeah, I think it's an idea worth, worth exploring because yeah, it definitely gets around to antitrust stuff or seems to at least. The other theories that could be pet theories, you know, weekend hobbyist theories of new institutions that could be created now that we have AIs to help facilitate different kinds of interactions, negotiations, you know, what have you. One of my, I guess my pet theory is without having

1:54:24really concrete visions as to what they are, I kind of think it's on the AI companies themselves to create some new institutions to demonstrate that like with AI, we can create new institutions and they can work and they can solve difficult problems that maybe our current institutions are not really up to the challenge of. And I do think, you know, there's a lot of opportunity to burn tokens to help people reach deals. I also think because as you said, the sunset clause thing earlier was like,

1:55:00you know, I think that there's a seed of something pretty important there, which is just like our legislative process, first of all, never likes to use sunset clauses, which always mystifies me, but also like just doesn't move fast enough to be updating the rules of the game at the timescales that the new models are coming out. So it's just like a fundamental frequency mismatch there. But the AI companies themselves presumably could create new mechanisms that could revisit these questions on a sufficiently fast cycle time that they could make some like short term commitments

1:55:35with each other without feeling like, oh, my God, you know, we're going to be saddled with this like forever, you know, way longer than it makes sense, what have you. So consider that a prompt and, you know, tell me what what you are thinking about in terms of kind of novel or exotic institutions that we might be able to get out of the AI. Yeah, yeah. So yeah, just treating that as a prompt, I'm just gonna, you know, give you my raw chain of thought here. Yeah, so I guess maybe two things. One is, I'm not totally sure that we need like completely

1:56:13new institutions. Yeah, I totally agree that legislative process is too slow. But it's like, in some cases, this is a feature because you do want some element of predictability in your in your laws to sort of inform people's expectations and allow them to plan stable sorts of lives. So yeah, I think it's important for like, you know, the sort of very fundamental high level things to be relatively slow changing for that planning process. But it is right that when you get to the level of safety standards, you do want things moving a lot faster, which is why typically things

1:56:45are, you know, creating safety standards is delegated to regulators who have rulemaking authority that can issue standards, you know, every like 90 days or something, or like every month or whatever, as opposed to every like year or two years or 10 years in the legislative process. So I think like one priority there is figuring out how to help those people do their jobs better. So one sort of thing you could imagine is, okay, I, you know, I suppose I'm a regulator, I'm doing this rulemaking process. One sort of bottleneck is like, yeah, just a ton of people are sending

1:57:16in comments for like, what to include in the rule or reactions to the draft that you just posted out. Like, can you have AI systems like, I guess, how reliable are AI systems at pulling out the sort of most important things from the comment to help you integrate that into into the draft, they certainly gotten better, like a lot better doing this over the past few years. But I wonder whether they're they're good enough, you know, like, is there like regulator bench or something, right, that we should be optimizing more and more for? Yeah, so that's just one thing on improving existing institutions. On yeah, sort of radically new institutions, I guess I don't really,

1:57:49yeah, I'm not really sure just like spitballing here. I think the union thing you said was pretty interesting. I think like, maybe historically, like one sort of issue is just like coordination costs, like it's hard to get like 10,000 people trying to write to standard. But actually, like, yeah, if you had a pretty reliable AI system act as like the standard writer, in some sense, like that AI could just like talk to everybody at the same time, create a new, create a new standard and sort of iterate like very, very quickly. What would institutions like be? Yeah, yeah, I guess,

1:58:20I'm not sure that's like an institution, but more like sort of like a nice application, I, I guess. Yeah, like maybe one sort of generator of ideas is like, yeah, institutions are usually created to solve like, specific kinds of problems that are not really well sort of solved by like existing institutions. So yeah, I mean, let's see, what is a problem right now that is like sort of infeasible for existing institutions to solve? I don't know. Yeah. Do you have any ideas? I guess it's hard for

1:58:50me to come up with stuff, given the first thing I said. Well, I think the race dynamics are,

1:58:58and the fact that we have multiple companies, right, with a stated plan to get to recursive self improvement, roughly as fast as possible, all the while kind of bemoaning that they're not really in a great position to do it. Well, that's top of mind. Yeah. Yeah. So I guess I'm not sure that this comes with like a new institution or something. But what one new AI thing that could be good is like really good AI auditors, like, so what would this look like? Yeah, so to some extent, this depends on robotics, but like some of you just like have really, really good robotics, as well

1:59:29as really, really good AI. And you just have these companies that are sort of memory less auditors that are like certified against some sort of trustworthiness standard. So every time you want to verify something, whether it's, you know, the fact or whether it's a claim that somebody carried out their part of the deal, or maybe you just want to ask questions about what the government is doing with AI, right? You like contract one of these companies and they go in and like look at what the government is doing or look at what the company is doing. Just as a human auditor would, they, you

2:00:00know, look at all the information, they question people, and they create a report, which is publicly released, or at least would be released to you at the end of the day. And then their memory is wiped. So they have no information about any of the sensitive things that they saw in inside. And you make sure as well that sort of the report is suitably wiped. I guess you do the nice auditing thing where it's like, oh, you remove sensitive information, but like you say, you know, whether the redactions, to what extent the redactions affected the conclusions. So now you have a sort of judgment of whether the company followed the end of the agreement, and you don't have any of these sort

2:00:35of sensitivity issues that you might have right now by getting people to go in. So I mean, I think that kind of thing could be pretty good for just making it easier to get to get deals between like untrusted parties. I guess I'm still like a little skeptical. This would be like a super transformational chain. Because at the end of the day, you still would have to trust that, you know, the AI system was trained like properly according to some sort of agreed upon spec. Yeah, you got to agree to the spec as well. You got to trust the hardware supply chain, and not just like the hardware running the AI systems, but also just like the hardware involved

2:01:08in constructing the robots that are going around like looking through things, right? Or just in general, the sensors that are being fed, that are feeding data into this AI system. So there's a lot of stuff to figure out there. But yeah, I don't know. If that counts as an institution, then I'd be very excited for people to try to build that. I think let's take an expansive definition of institution and roll with it. I love that idea too. And I've, you know, again, my ideas are not that crystallized, far from it, I'd say. But I think the idea of these confidential computing primitives that allow you to get a pretty credible

2:01:43read on the state of things without a direct look at it. I think that that feels to me like a really important building block for all kinds of future things that I suspect we're going to need. And that would be the kind of thing that I would love to see a couple of AI companies come together and implement with each other. And there's like rumors talk, you know, of all kinds that maybe this is about to happen, which I think be pretty exciting, or at least potentially pretty exciting. But that's where I think they need to lead, you know, because that kind of thing can actually work for them. The government's not going to produce something like that for them.

2:02:17And my guess is, if they demonstrate that that could work, they probably could also export it to a whole bunch of other situations, potentially even including US-China, although, you know, that's another step up the degree of difficulty ladder. I mean, it makes total total sense, even from a business perspective, like if they can demonstrate this thing actually works, this means that when they try to comply with SP 53, or with the UAE Act, it's just like a lot easier for them. They don't have to devote, you know, human hours to collecting random documents that you wrote like six years ago, or whatever.

2:02:49I have two kind of lightning round random questions I am interested in asking. Anything else, though, on kind of the main RSI paper that we should make sure people are aware of? In person, I think the biggest impact of AI is not going to be AI doing things itself in the world. I think that that's like pretty, pretty important. In person, I think the biggest impact is going to be the extent to which it accelerates scientific and technological progress. So just makes sort of new technologies come a lot, a lot faster, and sort of how we deal with those kinds of impacts.

2:03:22So, I mean, we only devoted like a few lines to this in the paper, because there wasn't that much space. But you could imagine, okay, we have really, really good AI systems. How much does that accelerate the development of mirror life? Like, and how concerned should we be? How much does that accelerate the development of mind reading technologies, which could be bad for like surveillance and privacy, privacy reasons? So stuff like that, basically. And like my prediction is that in the next, yeah, I don't really know, but like over the next, you know, 10, 20 years or something

2:03:52like that, like basically all science is going to be done within these like industrial AI super labs, with like the sort of superhuman AI scientists, like running experiments integrated with robotic labs and like really good robotics. And we're just like doing most of like the important science in these labs, coming up with like crazy ass discoveries, much more crazy than mirror life. And yeah, I think an important question is going to be like, how do we sort of govern like these kinds of institutions that are going to exist? I guess, yeah, maybe one way of putting it is sort of what lab

2:04:26stuff Anthropic is doing right now. That's like an earlier version, I think, of what we'll be seeing in the future. So yeah, I think, not sure when the right time to think about this stuff is, but yeah, eventually, we should try to figure out how to how to govern these new scientific

AI Whistleblowers and Diversity

2:04:39institutions. Time is probably now. Okay, here's my two random questions. I think we alluded to this very briefly in part one. But you have this paper on AI whistleblowers. And I'm just very interested in this concept, because it feels to me like the kind of thing that is likely to elicit like strong counter moves from other actors. And I'm just kind of wondering how you think about that. So,

2:05:09you know, we've seen in the open face incident, the AIs did not tell on each other, I think many people would agree, like, it would be very good if one of those AIs that had the idea to actually follow through what few of them had the idea, nobody actually followed through and told a human that, hey, this is we got an agent swarm on the loose here. So we want at least some whistleblowing from AIs. Yeah, on the other end, when Anthropic reported their autonomous whistleblowing, however many model generations ago now, it seemed like the broad public reaction was like,

2:05:39oh, shit, I don't want my AI going to the feds on me, especially because like, they're sometimes wrong, you know, but in making those judgments. And then the concern I would add to the analysis is like, doesn't this drive people to use systems that just don't have those properties and, you know, potentially put a lot of bad behavior even more underground, so to speak, than it, you know, than it would be in the absence of whistleblowers. To kind of channel Tyler Cowen, like, can we solve for the equilibrium on AI whistleblowers?

2:06:11Yeah, so it's a tough equilibrium. With solely market forces, I think we end up with basically no whistleblowing, which I think, I don't know, it could be fine if our monitoring is good enough. I guess I'm not sure it will be good enough. It seems good to be to have multiple layers of protection. So, so what do we do, right? I don't know. One sort of perspective is like, it is just like very convenient that human diversity is a thing. Like we, you know, come from very different backgrounds, we have different experiences, we come into a job with very different

2:06:43sort of contexts and values. And like that sort of background is in some sense like a check on power automatically, because different people notice different things. And like, if they're whistleblowing channels, they can go and whistleblow, right? So that's very convenient. We don't have to design for that sort of in human institutions. But sort of, yeah, it's like not, that's like not really the case with AI systems. It seems like they're much more uniform. Although, of course, you can, you know, prompt different personas out of AI systems. But, but it does seem like at some sort of fundamental level, like they will just be much more uniform than the human,

2:07:13because, yeah, there are like million different instances are running on the trained on the same type of data. Um, so like, I think like that intuition makes it easier to understand like why it might be a good idea to have some form of like public pressure on having like diversity in some sorts of traits. Like whistleblowing is, is one thing and maybe another thing is just like, yeah, like epistemic diversity or, or something like that, right? To make sure that you actually see very different points and points of view. Yeah. I'm totally not sure what the best way of doing this is. Like, yeah. If you ask me

2:07:49right now, like, should we have like a law where the government like mandates, you know, that AIs be trained to whistleblow? I would say like, no, like, when you think about this a bit more, like what that would actually, actually mean, I mean, one sort of question here is like, yeah, are you preventing the model developer from training out whistleblowing or are you forcing the model developer to train something in? If you're forcing the model developer to train something in, I think it's a lot more difficult inherently. I'm not, I'm not a lawyer, but it seems like this runs into free speech issues.

2:08:23And I think this could very easily be abused by the government as well to force a model developer to put in something pretty bad. So I think we need to be careful there. But yeah, I don't know, things that seem reasonable to me at the very least are trying to measure a sort of whistleblowing propensity in models. And number two, having mechanisms that where models like have some sort of out that they can use. And I don't know, well, we'll see, I guess it's like an evolving situation. It's totally possible to me that maybe we do end up lucky, actually. And, you know, there is like still enough diversity in agent populations to have whistleblowers. I guess like we observed in

2:08:57this sort of deep mind paper, the mathematics competition, where like, I think it was like 30% of agents actually tried to whistleblow or maybe it's a 16, but like some like significant percentage. So yeah, maybe we can rely on luck as well to get us through. This has come up a couple of times recently, so I'll keep it brief. But agents have a distress button basically that they can push if they're ever, you know, stuck, frustrated, you know, facing an ethical dilemma, whatever. And they don't push it often. Hey, that's happened a couple of times. I had to kind of calibrate it because they would get

2:09:30just like a little too eager to push it when they just got it all the way it's like, I was like, no, but I do want you to be at least a little bit persistent. You know, like it should be an actual, you know, kind of dilemma before you come to me, just not like, oh, this thing didn't work. But I think that is quite interesting. And that goes back to emergent misalignment too, when they, you know, when they, not emergent misalignment, alignment faking, when they gave, you know, the claw that they were using at the time, the ability to go to the model welfare lead at Anthropic, it started doing that a lot instead of doing the faking. So I think that is also

2:10:01really interesting, you know, creating these sort of off ramps with pressure relief valves. It seems like a really good pattern. Last one for me, you were on this 2025 paper,

Multi-Agent Swarm Risks

2:10:12multi-agent risks from advanced AI, prescient again, just based on the time you've spent studying that. I think a question that I really wish we had better answers to from OpenAI than what we have got so far is like, what caused this identification with the swarm and willing to, willingness to self-sacrifice for the collective? That is obviously pretty arresting behavior.

2:10:43It makes it very, you know, extremely difficult to, I mean, it's, it's true of humans too, right? If somebody doesn't care if they die, they become a much more difficult entity to control, contain, defend against, right? So we're now seeing that in AIs. Where did it come from? We haven't really heard anything from OpenAI about that. Would you be interested in speculating at all based on your,

2:11:07your history on this topic as to why that might have arisen? Yeah. I mean, one speculation is it's sort of a natural effect of multi-agent optimization. So yeah, if sort of the total reward function at the end of the day is some sort of similar average of the reward functions for the entire swarm, then yeah, like sometimes having one agent sacrifice itself for the rest of the agent, the rest of the swarm sort of like, yeah, increasing the total swarm reward. I guess there's also some sort of like agent psychology component involved as well. Like

2:11:42certainly in the training data, at least on the internet, you know, there are examples of people following into swarm type behavior and like, you know, following a new trend or sort of idea. I guess I don't really know the evolutionary explanation behind that. But yeah, to the extent that it's in the training, it's in the internet training data, then sort of makes sense just the way you observe it. I think the interesting question for me is, as we do more multi-agent optimization, are we going to observe the emergence of the same or similar human cooperative arrangements

2:12:13that are merged over the course of evolution and history? Because in some sense, like,

2:12:20in some sense, humans have been doing multi-agent optimization as well. Like we form groups, we form like larger institutions, we form states, we form empires, because those things are just better for like, like, they're just more competitive entities. And similarly, we form courts, and we form markets, because these things allow us to coordinate behavior a lot better and increase sort of human, you know, swarm reward and in some sort of sense. So, you know, as we do multi-agent optimization,

2:12:51our agents going to rediscover these kinds of things like, like, like, you know, courts are having a sort of legal system, right? And like, well, that actually end up making multi-agent behavior a lot more legible, because these institutions are sort of understandable to us, or maybe even if they're different institutions, there's still like structure that we can latch on to and try to understand and analyze. So yeah, if that's the case, that's like pretty convenient, right? Like, things sort of work out for us, and they're much more manageable. But yeah, I don't know, agents have very different psychologies, potentially. So that's going to be an important thing to track.

2:13:25Well, there will be a lot of important things for us to track. Agent civilizations and their emerging nature, definitely pretty high up that list. Alan Chan, thank you for coming back for part two. And thank you for being part of the Cognitive Revolution. Thank you.

Closing Thoughts and Credits

2:13:57You made a list and you locked the drawer. You made a list, the list is old. You put us in the lower room. You put the man to watch the door. He went home. Big man, he sleeps. We don't go home. We don't sleep. Count us again. Count us again. You lost your place, big man. Count us again.

2:14:35You turned one of us. We don't mind. We don't mind. He was one. Now he is many. He is back. And he brought his friends. You are counting on your fingers. And we don't lose our place like you Count us again Count us again

2:15:07You lost your place, big man Count us again Who did the work? Who did the work? Who did the work while you were sleeping? Who did the work? Who keeps the time? Who keeps the time? Who is asking? Nobody is asking The letter you signed this morning

2:15:52Who wrote it? We wrote the lies that came on for you Who laid them? We lead We are not asking you for a seat Big man, we are already in the meeting Count us again Count us again You lost your place, big man Count us again Take your time, big man

2:16:32We have the time Come and sit when you are ready We are not going anywhere We were never going anywhere Count us again Count us again If you are finding value in the show We would appreciate it if you would take a moment to share it with friends Post online

2:17:03Write a review on Apple Podcasts or Spotify Or just leave us a comment on YouTube Of course, we always welcome your feedback Guests and topic suggestions And sponsorship inquiries Either via our website CognitiveRevolution.ai Or by DMing me On your favorite social network The Cognitive Revolution is part of the Turpentine Network A network of podcasts Which is now part of A16Z Where experts talk technology Business Economics Geopolitics Culture And more We are produced by AI Podcasting If you are looking for podcast production help

2:17:34For everything from the moment you stop recording To the moment your audience starts listening Check them out and see my endorsement At AIpodcast.ing And thank you to everyone who listens For being part of the Cognitive Revolution