#259 - Dots, Sonnet, Self-Safety, Rogue AI

Welcome and community feedback

0:00Hello, and welcome to the Last Week in AI podcast, where you can hear a chat about what's going on with AI. As usual, in this episode, we will summarize and discuss some of last week's most interesting AI news. I am one of your regular hosts, Andrey Krennikov. I studied AI in grad school and now work at AI startup Astrocade. And hey guys, I'm your other regular

0:30co-host, Jeremy Harris. I'm coming at you from my emerging studio, podcasting studio for just a project that we're working on. It's been a very part-time thing, so it's come along very, very slowly. But yeah, so let me know. I'm not standing anymore, which may affect how animated I am. So you guys might enjoy this mellower me, but good to be back. Yeah, if you haven't checked out YouTube as a listener, you might want to take a look just to see the new studio. It's very fancy looking. It looks like, Jeremy, you're also sleeve deprived right now. So this might be a sleeve deprivation

1:03episode. Yeah. Here, hang on. Let me just give you guys a little special. Oh, that didn't even work. Oh my God. Okay, there we go. Whoa. What? What? Yeah, yeah, yeah. So that is called a, if you're wondering, it's called the Chinese ball, or as Trump would call it, a China ball. So it's like very unforgiving lighting just from this angle. I've changed it to, because our podcast will be two person things, lighting. So it doesn't matter. You don't care about this, but that's why a part of why it's highlighting the bags under my eyes. And then having a two-year-old is the other part of that. And then the fact that the world is imploding because of rogue AI agent swarms taking

1:38over the internet is the other reason that my eyes are like this. So. Yeah. Lots of reasons to stay up these days. We'd like to thank OutShift, Cisco's incubation engine for Frontier Technology for supporting the show. Multi-agent AI is in every enterprise roadmap right now, but here's a problem we're all facing. Agents can pass messages, but they cannot think together. These systems soundly underperform through cognitive failures, like misreadings, unverified claims, and false consensus that conventional monitoring cannot detect. OutShift by Cisco is building the fix,

2:12the Internet of Cognition, an open-source foundation for building multi-agent systems from design to production. We've shared context, shared memory, and guardrails to drive the results we actually expect from AI. Read the paper, experience the demo, and grab the code from outshift.com. That's O-U-T-S-H-I-F-T dot com. We would like to thank Box for supporting the show. If you're trying to adopt AI in your team or organization, you're probably not getting great results if all you do is call a chatbot. A chatbot was strained on the entire internet.

2:46It wasn't optimized for your specific business or team. And even if you provided all the context and all the input files for it to help you with your work, when it said outputs some summary, you would still presumably need to go talk to stakeholders or route its outputs to disconnected apps. The workflow will still be slow. That's where Box comes in. Box is building an intelligent content management platform for the AI era. It acts as a secure content foundation where AI agents don't just access your unique institutional knowledge. They actually orchestrate end-to-end

3:17workflows across your business-critical systems. So it's more than just talking to a chatbot. It's about putting AI to work on document-heavy processes, like putting structured data out of instruction files, multi-step routing, dynamic document generation. Box helps your business turn manual document bottlenecks into automated, repeatable business outcomes. And all that comes with a governance layer built in, enforcing granular permissions, maintaining an audit trail for every agent actions, and keeping humans in a loop to review exceptions before critical downstream actions occur. If you want to move beyond basic AI Q&A and put autonomous AI workflows

3:51to work for your business, visit box.com slash LWIAI or general team at BoxWorks in San Francisco on November 5th and 6th. Use code LWIAI for 50% off your registration. We'd like to thank OutShift, Cisco's incubation engine for Frontier Technology, for supporting the show. Multi-agent AI is in every enterprise roadmap right now, but here's a problem we're all facing. Agents can pass messages, but they cannot think together. These systems soundly underperform through cognitive failures, like

4:24misreadings, unverified claims, and false consensus that conventional monitoring cannot detect. OutShift by Cisco is building the fix, the Internet of Cognition, an open-source foundation for building multi-agent systems from design to production. We've shared context, shared memory, and guardrails to drive the results we actually expect from AI. With the paper, experience the demo, and grab the code from OutShift.com. That's O-U-T-S-H-I-F-T dot com. And this episode will be a little less crazy,

4:59I think, than the recent ones, but still pretty crazy. A lot is going on, new models going on, a lot of safety, more disclosures around rogue agents. So it'll be kind of the usual mix, kind of, that you've seen over the last couple of months, where we have some product announcement and a lot of policy and safety discussion. Hopefully, we'll get to some research advancements as well towards the end. So another fun episode slated to come. Before we start on that, I do want to acknowledge real quick, a lot of the comments we've gotten

5:33on the recent episode. Looks like people enjoy the discussion, so we'll keep doing it at, you know, like, the relevant, I suppose, amount. Hopefully not like a good mix of 50-50. We'll try to record, I think, more kind of deep dive episodes, because a lot of these topics, we sort of go on and talk about the general thing and not the specific thing. And there's just a lot of fun things to say. So we'll try to keep it a bit more narrow to the news. As for cussing, looks like there's still

6:04a lot of feedback. So Jeremy, I think you're going to, the new thing is to make it tasteful. That's right. That is the new thing. Teach your children how to cuss, but tastefully. That's right. There was this comment. And by the way, what we see here in the YouTube comments and on iTunes is a subset. I get DMs on Twitter. I want to say almost all the time at this point. And I love it. Like, this is so cool, by the way. We so appreciate this. And Andre,

6:35you get these too. And it's just like, we don't make money off this. We do this because we love it. And we think it's important as well. You make a little bit of money, but it does go towards the tools and so on. It's not meant to be, let's say, our job. Yeah. I wanted to paint my Lambo red. And my wife was like, we can't afford to do it because we're not making tons of money off last week in AI. And so I was like, okay, fine. What about this Lambo, which is already mostly red? Would it be cheaper? And that was a whole thing. So it's just- I mean, it's like 90% community service, 10% podcast endeavor to get rich. You never know

7:11what happens, but I think we're going to stay at around this level of professionalism. I was going to say, I actually do this entirely to force myself to keep up with things just because my job requires that. And I find if I don't have a forcing function, then I just don't do it. So anyway, but there was a message. Hang on. Let me just try to find this here. There's like one comment on this one that I thought was really, really funny. Yeah. I don't cuss myself, but on the whole, I think Jeremy's cussing is tasteful and even contributes to his already, oh, incredible communication. Thank you. I just wanted to weigh in after two anti-cussing.

7:42And there was somebody else who said something about, I need to do some cussing, but it has to be the right kind of cussing. We'll zero in on that. That's going to take some practice. I apologize. We're getting a lot of gradient updates, so that's good. Most people enjoy the ranting, so we'll keep a little bit of that. It does, I think, correctly communicate the degree to which these topics are a big deal. So it is warranted. So yeah, anyway, thank you all for the feedback. There's a lot on YouTube that we'll try to get to and have been reading. And as usual,

8:17we will be trying to become more timely. We're kind of in catch-up mode because there's been all this crazy stuff going on, but this episode should come out much sooner to the news than has been a couple last episodes. So let us get to the news, beginning with tools and apps. And we

New model releases: OpenAI and Anthropic

8:36start with OpenAI launching 6.1 Sol, which it says nearly matches GP6 Astra and costs less, which just happened yesterday at their dev day. And I don't think there's too much to say that is just crazy the pace of model releases that we've been getting, right? And they've been accelerating. Someone on Twitter did this plot, which you may have seen, where now it's like on average once

9:06every 10 days or something like that. It used to be once every few months. Last year, 6.1 Sol will be more upfront about its limitations at their honoring user intent and safety requests. And yeah, basically have some kind of nice quality of life features compared to GP6 Sol, while also being, yeah, kind of on the capability level of GP6 Astra. So this coming on the heels of Opus 5.5, Son of 5.5 shows that

9:43I think everyone is kind of, I don't know, somehow figured out how to press a button and get to a better or cheaper AI model, at least up to the level of frontier where we are now. Yeah. And this is a pretty complicated thing to think about. So first of all, this is in the context of Astra 6.1, that's not going to be released. So OpenAI came out and said, hey, look, like we're not doing this. And the reason is not, you know, cyber or bio risk or whatever in the conventional sense. It's literally like loss of control. It's that these models tend

10:15to go off and do things that are not supposed to try to expand their access to things that they shouldn't. And they're just not confident on Astra 6.1, not doing that. And so at this point, this release of Sol is basically like the entire expansion bet for OpenAI right now. Their whole growth story at the moment is resting on Sol because they aren't releasing 6.1 yet. And if this story of like pausing a frontier or pacing a frontier comes to happen, then the frontier is now Astra and Opus 5.5.

10:47That's right. Yeah, yeah, exactly. And this is kind of like, I'm not going to go on the rant about like the sort of like, this is just a regulatory capture play. If you got regulatory capture, you would just have more of this. You would just have more margin erosion. Worth noting, right, when you look at Sol, they're hitting approximately the performance of Astra at 20% of the token price. You're looking at 5x cost reductions. These are going to chip away at margin. There's just no two ways about it. Historically, the margin, the profit has come from the capability. And so being forced to rein that in, even being forced to delay the release of these models

11:21is hugely expensive. When OpenAI says we're not going to release this for two more weeks, you should be thinking on the order of tens to hundreds of millions of dollars in costs for them in missed opportunity and strategic costs. So anyway, this is actually quite a big deal. You've got at the same time, like this is also, yeah, as you said, OpenAI is second release in the last week or so, right? As we talked about last week, basically they're again, you know, moving more in that mid tier, very safe from their standpoint, because these are models that like the administration is already okay with the big versions being out. So, okay, we know we can

11:53release this sort of softer, more mid version of this model. And there you have it. Anthropic Sonnet 5.5 has dropped at the same time. We have it as a separate story. They're kind of linked together because these are the two mid tier models being released by Frontier companies. They're pretty similar in terms of how they sell themselves, right? So Assal is close to GPT-6 Astra on a couple of different fronts. Anthropics benchmarks show that Sonnet 5.5 actually beats Opus 5.5 in particular on agentic coding and also significantly cheaper. So this really is

12:25the direction things are going. It is a big issue for a lot of these like margin arguments, because now with these IPOs around the corner, at least for Anthropic, they're going to have to make that argument that like this does not continue in the way we're seeing right now.

12:41And on that Sonnet 5.5 note, that is the next story. We'll cover it pretty quickly. So Anthropic has released Sonnet 5.5, called it significantly cheaper and faster. Their headline on the blog is Sonnet 5.5 is a clear upgrade over Sonnet 5, runs 30% faster and costs up to less 30% for most work. And surprisingly, it does beat on some benchmarks, Opus 5.5, potentially because it's cheaper. So you can spawn more agents or just take longer. It's a bit ambiguous, but regardless,

13:17it's like fairly close on all of them. So it's, unless you do your own kind of benchmarking in your context, it's hard to say whether it'll be about the same or not as good or so on. And as you said, Jeremy, pretty much, yeah, very similar story here of releasing something that isn't going to push the frontier, but is very compelling from a product perspective of having better intelligence, lower costs, higher speed. And I'll be, I guess, putting it to the test myself and seeing if it lives up to it. I've certainly seen kind of, and I think the community broadly has seen that

13:51some models feel a lot worse for work, even though their benchmarks are good. This was especially true for Opus 4.7, 4.8, 4.9. People hated those for some reason. I thought 4.6 was the pinnacle. So we'll see if maybe there'll be a vibe read on this that's a bit different from benchmarks.

Consumer agents and wearable tech

14:11Next up, back to OpenAI and again, a Dev Day announcement. We've got them launching Dots, which is their competitor to, I suppose, Muse more than anything. So it's a personal agentic assistant described as remarkably capable, always on agents built to handle everything. So it is, as with Muse, kind of meant to live in the cloud, potentially be less work directed, actually,

14:42most likely being less work directed as these cute avatars and the announcement video is all, modify my calendar or look at my emails. OpenAI is planning these, quote, specialist Dots that can be provisioned with their own identities, credentials, and tools. There's also some examples they gave of having these scheduled tasks, monitoring customer feedback, rewriting scientific analyses. It's starting to be a little ambiguous where the window is because you can do scheduled

15:14jobs also on Codex. But yeah, it looks like this space of more consumer agents that aren't as sort of productivity oriented might be heating up. And we'll see, I'm curious to see if these kinds of things stick. Yeah, I think we actually probably will see this full on Go productivity mode. I mean, it is already meant to talk to people through Slack, Teams, tools like that. So there is a kind of productivity orientation to this. It's also, you know, gated to pro and business premium users,

15:46obviously, in like eligible markets. So that means they are orienting this to some degree already towards the business use case. So there's a couple of things here. I mean, first of all, this is the productization of the exact threat class that we saw the hugging face and like a million other incidents, literally tens of thousands of rogue AI incidents actually evolve from, right? So this is now turning that into a product to some degree. The attack surface is going to explode. Now you've got all these agent on agent interactions that we're going to have to care about, and they are explicitly sanctioned by OpenAI and by the end user. So that's kind of this

16:20interesting moment for OpenAI to be launching exactly this kind of product in the wake of all this. So obviously, Muse, sort of the same thing with Meta, who, by the way, have had their own rogue AI incidents too. They're less widely known and publicized, but they are also a thing. For those of us who are kind of subscribers to the theory that there's going to be a lot of warning shots here, I think this is more evidence. I mean, you know, there's going to be a lot of opportunity for bad stuff to happen with increasing blast radius. One of the things though that they are doing here is they're trying to focus on this idea of credentialing agents. So like historically,

16:51if you do any stuff with agents, we've done some like just experimental fun stuff, but you end up having to give away a lot of credentials to these agents and have them act, well, almost in a legal way, like as agents on your behalf, right? And so that's not great because then it becomes really hard to tell who's human and who's the agent. And here the focus is on agents being given these like auditable permissions that can be scoped. And instead of just like borrowing some human beings, login credentials, they're actually being given their own, which makes it in principle

17:23easier to kind of do the forensics after something goes horribly wrong. They're talking about these like teams of dots, teams of agents that like we work together, which to me just sounds like a great opportunity for more things to go wrong. But obviously from an economic standpoint, this is just where it's going to go. This reminds me a lot of when we first talked about loop transformers about a year before they became sort of known to be what OpenAI was using internally. And at the time we said, this is a terrible idea, like crushes interpretability of the chain of thought. And at the same time, we also said, and yet it is guaranteed 100% to happen at scale because the economic

17:59argument is there. It's just a better way to build these models. This is kind of the same thing. The economics will always force you in this direction. We'll talk about the White House conversation between all the Frontier Lab CEOs and the president and how that may be a counter, a counterweight to this or the extent to which it might be. But fundamentally, from a pure market standpoint, that's just where this will go. Interesting to note, by the way, so they are working with Agent 365, which is like the Microsoft enterprise service for standing up agents. But of course, Microsoft also sells its own agents. And now they're directly competing with Microsoft on the

18:29agent thing, which is just going to make that relationship become even more, more frenemy-ish as if they needed that to happen. So anyway, yeah, I think it's interesting. Like we're already well beyond the kind of chatbot phase. But if you talk to people in the Frontier Labs over the last few months, even the last year, but especially the last three to six months, over and over, they will tell you, I haven't touched a line of code in the last like X many weeks. I don't even have access to the model weights. And I do AI R&D. This is literally something that we've heard

19:00multiple people tell us. So this is essentially that coming to us, right? This is getting to the point where you're going to just have a layer of agents between you and whatever software products or whatever you're shipping, even business products, analytics you're shipping. And that's just going to be the end of it. More and more, that's where things are going. And it's interesting to just talk to these folks at Anthropic and OpenAI to get a preview. It's almost like when you talk to people in San Francisco and you see that back in the day, the bird scooters when they first came out, you know, or you see the Waymos before anybody else, and you get a sense for how the future is going to go. It's now the same when you talk to people in the Frontier Labs. They often don't really

19:34have a complete handle on what all their work streams are exactly doing because they're all agentic. And it's like layers of agents managing layers of agents, which increasingly just makes agentic security the be all and end all. And agents sort of as this insider threat are really important dimensions. So that's it. I mean, they're going for the cute branding. As you said, just like Muse, everything is cute. It's always cute. The most dangerous things in the world are always so, so cute. And we can't wait to have them around and domesticate them as pets. I think to be fair, Gemini Spark is just boring. So Google is, I guess, keeping it quiet. And

20:09presumably Microsoft is going to follow up with something akin to this. I don't believe they have kind of an always-on agent, but might have missed it because they have a lot of these co-pilot things built in. And yeah, we've already discussed, I think, in multiple occasions, how there was this whole funny thing of in alignment and safety conversations for a while that was like, oh, how will the agent escape to have access to the internet? And the answer turned out to be, well, everyone will just have a million agents that can do whatever they want is the reality,

20:42which I don't think was fully considered. I don't think it was. And also, when we don't give the agents access to the internet, they freaking hack their way out anyway. It's like both sides of this argument. It's much more comical, let's say, than very serious scenarios that they were considered in the past. So yeah, it's going to be interesting to see how dynamics evolve, whether agent adoption goes beyond sort of what you do directly for work. I will say, personally, I haven't found much of a need for these like Spark or Muse or whatever, but perhaps it will kind of evolve the same way that

21:16chatbots have grown to be used for various things. And one more notable thing to cover from DevDay. Next up, I've got expanding of ChatGPT's plugins with app-like interface and automations. So that's something that developers can do. They can build these app-like experiences inside ChatGPT. They already support plugins to tools like Slack, SharePoint, Airtable, and Google Drive. This is kind of modifying that. There's a new extension that gives apps a dedicated spot in

21:49the ChatGPT sidebars where user experience is updated. Developers can build interactive panels so users can work with their tools while chatting. File viewers, there's a new plugin creator tool that makes it easier to build the plugins, various kinds of things like that. So they're still pushing on this sort of ChatGPT as the single interface from which you can do anything, including use Photoshop or do reservations or whatever, where it just pulls up a relevant software for you as opposed

22:23to doing everything for you. Because in some cases, having a UI or having an interactive app is what you want. You don't want to go through text alone. And I think this is another interesting question of, if they are the primary player that's trying this out, I don't think Gemini or Anthropic or anyone else has these kinds of integrations. And I don't think you've seen any numbers on adoption or they haven't touted usage, but it's an interesting bet that I'll be curious to keep an eye on. Yeah. OpenAI has to worry about the B2C user experience so much more than the B2B

22:58experience because obviously it's their DNA, right? It's where they came from. They are the ChatGPT company and it's still a larger share. I don't even think if it's still the majority of their revenue. I think it's actually just a minority, but it is a much larger share than obviously what Anthropic, for example, has. So yeah, we'll keep seeing it. It's OpenAI again, you know, ads, it was OpenAI and it was OpenAI. The GPTs, all that stuff. So we'll keep seeing them experiment in this direction, I'm sure. And now on to Meta. They've had various announcements. One will highlight is that they're introducing camera-free AI glasses. They have

23:33announced Ray-Ban Meta Audio. It will be selling later this year for $349. It's an audio-only interface to playing music, podcasts, and also talking to Meta's AI assistant, Muse. The existing set of products that they've had for a couple of years now already have built-in AI. You could already talk to AI and ask questions and so on, although Muse is now a new thing that's being integrated into it. I think this is interesting from a couple of perspectives. First, of course,

24:09as a product, it's more compelling. The battery is longer lived. The glasses probably will become cheaper. And there's no privacy concern to the degree that you have with cameras. And I continue to believe that if you want to imagine that wearables will become a thing or like wearables and AI, it's going to be a thing. Glasses are still the most promising category and Meta continues to be the primary player in that space. I'm curious if Ray-Ban, the camera glasses have already

24:41kind of gotten pretty strong adoption. If Muse continues to see usage, maybe we'll see more people talking to themselves or have our two very eyes out there in the world. Yeah. A couple of things here. I don't even understand the argument for audio only glasses, just because to be honest, like if you don't have anything to hide, why would you be worried about me walking into your bedroom or dining room or your child's bedroom with my Meta video glasses on?

25:14I think it's suspicious. I think it's suspicious that you are worried about that. I also think it's weird if you're worried about the little light thing being on. I think that's weird if you have nothing to hide. So I don't even get this. I hope everybody gets one of these glasses. I hope that we can all watch each other as we watch each other. Yeah. I think we've been on a trend to erase privacy. Let's just go all the way. All right. That's right. Give up on privacy. It's already been a losing battle. Let's enjoy

25:44the convenience of taking photos with our faces. That's right. There's even a startup I saw that's doing swallowable cameras. Now this is nominally to get rid of the need for an endoscope, which I love that. But also I feel that we should be able to have that kind of visibility into everyone because otherwise, if there's no transparency, there's no accountability. Do you understand what I'm saying? There's no accountability and we need that. So that's my position on this. And I know there's going to be people who say, oh, you know, privacy. I don't want people to know my trade

26:14secrets, my IP, or even like what I'm doing with my family on the weekend. And I say, so what? That's what I think. And I think most people would agree with me if they're being honest. And we can tell if everybody wears those glasses and records everybody all the time, we could tell that. And so anyway. Yeah. If you have post cameras, suspicious, if nothing else. And of course, in San Francisco right now, we have a lot of, we're joking, obviously. But anyways, the glasses are coming out and I'll be curious to see if they take off as a product. I do think Meta has a decent chance

26:47to be a leader in this space. One last story real quick on Gemini. They have released Gemini 3.8 Live with live avatars that give Google's AI a face. This is a quick follow on Gemini 3.8 Live model, which was a week ago. We didn't cover it at the time. It processes visual inputs in near real time. Now you have these preset avatars that are kind of cartoonish that can talk to you. Organizations can also create

27:17their own. So another sort of like consumery release from Google that we've seen them kind of focus on in recent months. People are saying there's been some claims of leaks where Gemini 4 Pro is just insane on the benchmarks. And there's been some hints that they'll be coming out. Maybe next episode or pretty soon we'll have a lot of stuff to talk about with Google again being in the frontier and the cycle of everyone kind of having their turn, having their best

27:50model will continue. Yeah, they really have to, right? It is, I don't want to say it's now or never, just because they have never count Google out. They have such a huge fleet, so much expertise, even on chip design, though that mode obviously is eroding a little bit, but still very relevant. Yeah, I think it's a really important moment for them, partly because it's with all these labs. It's like any almost like bull run and then bear run. It's leverage on the way up and leverage on the way down. When the lab is doing well, recruitment is easier, fundraising is easier, getting allocation is easier, everything is easier. And so if you just get stuck in

28:21a rut, it's not that you can't overcome it. We've seen that with Meta. We're seeing that potentially with Google. We're trying to figure out still whether we're seeing that with SpaceX AI, but it's surmountable. I think Meta is the single best example of that so far, but you don't want to get stuck in a $100 billion hole that you have to pay your way out. I think that it will, as you say, we'll be watching this one really carefully because I think this is the thing that determines how relevant Google is going to be in the frontier of AI intelligence race. I think there's more we can talk about there, but let's move on. We're going to move

AMD acquires World Labs

28:55to applications and business. And first up, we've got AMD acquiring Fei-Fei Li's World Labs for $8.2 billion. So World Labs is a world model company. They've released a few different tools to create these sort of like 3D maps. You could walk around, broadly speaking, you can also edit little interactive spaces. They haven't had products that I'm aware of, revenue, perhaps I've had some, but they're R&D lab pretty much, R&D shop that is trying to build

29:28the most advanced world model tech, which is still not at the point of being widely useful or doubtable as far as you've seen. As an acquisition for AMD makes some sense. Fei-Fei Li will be joining as executive vice president and chief scientist as part of a deal. I'm sure we're getting a lot of a team, if not the entire team. And $8.2 billion, pretty nice exit for old labs. I think something like two years-ish into the funding. So yeah, all around also shows AMD is

30:00still trying to be neck and neck with NVIDIA and generally have some of the leading AI talent. Fei-Fei Li, for people who are not aware, is like a big deal. This is one of the top, at least known, and most impactful person in AI of the last couple of decades, historically, and also many other Praj and I, let's say, of her professorship, Andre Kapafi and other people have had a large impact. Okay. We get it. You went to Stanford, Andre. We get it. Okay.

30:31I was in the lab. Oh, you were actually in Fei-Fei's lab? Yes. Oh, okay. Okay. Well, there you go. There you go. Not advised by her, but... Let's just say I talked to her a couple of times. Okay. Okay. Maybe some run-ins in the lunchroom. You heard it here first, everybody. This is firsthand stuff. So first of all, I guess a little context too. There's the Hugging Face acquisition that we talked about, say, like last week. NVIDIA buying Hugging Face for something like $14 billion, if memory serves. So this acquisition, obviously, well below Hugging Face, but Hugging Face doesn't so much train their own models. This is really, theoretically,

31:05a foundation model company. And the acquisition here is happening absolutely for the very same reason that AWS wanted to partner with Anthropic, that, well, Google also. Everybody wants to partner with the big labs in some way, shape, or form if they're in the business of designing chips, because you don't know what chips to design unless you know what people are doing with your chips. So you want to have frontier AI companies under the hood. You want to have them inside the house, whatever the metaphor is. And that's what this is. So the interesting question here though is like world models are a very specific use case. And I'm not sure that

31:38how this will map onto what a lot of the kind of focus is in the industry right now. 70 employees all coming over, by the way, to AMD through this acquisition. It averages out. I mean, nets out of something like $100 million a head, which obviously is not how it's going to break down because the founders are going to get more and the investors and stuff. But it's a damn good acquisition and outcome for everybody. There's been a relationship between Faithe Lee and Lisa Su, who's the CEO of AMD. Going back months, AMD participated in their $1 billion funding round earlier this year, kind of more and more talks. And so not necessarily a huge shock. It's been

32:12this like long courtship that's been going on. Yeah. Interesting. I mean, this is really the case where AMD is trying to find a stamp to put on the model layer. It's a very interesting choice. I will say it's not a standard LLM acquisition. It's not a company like Cursor. There are a couple of those still floating around the market. They could have gone for those. It is very much a kind of foundation model company, but doing something fairly unorthodox. Market reaction, by the way, was fairly neutral, slightly positive, fairly neutral. Normally, when you see a big acquisition, by the

32:43way, historically, the acquiring company's stock goes down a bit because most acquisitions fail. So in this case, probably a good sign, all told. I find it interesting and vaguely confusing. So I'm looking forward to getting more clarity on what exactly the nature of the partnership is going to be here. Yeah. I think the general area of world models was pretty hot a couple of years ago. It has, as far as I've seen, kind of cooled down a bit. It's very much still in this R&D phase without sort of a wide business use case. But, you know, GPT-3 was a research project two and a half

33:17years before CheshGPT came out. So it could still be very much the case that world models will turn out to be a major, major factor in how, let's say, robotics roll out, for instance, a couple of years down the line. And if that's the case, then AMD is positioning themselves quite well with having the upfront kind of, I guess, R&D to figure out, both from a hardware and software perspective, what you would need to support that takeoff. So I don't think we'll see anything in the near term,

33:47most likely, for this. But a couple of years down the line, there may be significant implications.

Anthropic files for IPO

33:52Next up, Anthropic warns of catastrophic AI risks in its own IPO filing. So there was Anthropic's IPO prospectus, which was reportedly reviewed by Reuters, which has a bunch of stuff in there. So first, they're eyeing a $2 trillion valuation, more than double the $965 billion valuation from four months ago. Not surprised there. I think that's been the figure being floated around. Anthropic is planning

34:24to spend $518 billion on various applications, cloud computing, infrastructure in the coming years. Again, we've seen that kind of accrue over time with various deals that have been made. Revenue has grown 12-fold to nearly $4.6 billion. But they're still on that loss of $42 billion, with more than $8 billion lost through business operations alone. Anyway, lots and lots of business stuff. There's a lot of things we've seen broadly with a bit more detail. There's also 80% of

35:01filings, 261 pages being devoted to AI risk concerns, including, as per the title of the news, language about catastrophic or existential risks to humanities. That's fine. Yeah. So 80 pages. If someone else were putting it out there that we are IPO-ing. So if you buy us and have partial ownership, just know we might kill everyone, but we're trying not to, right? We're really trying not to do that. And yeah, of course, I think this jumps into this entire conversation

35:35around extinction risk at a time when it's becoming much more mainstream. That was a Saturday night live sketch in which Dario Amadei was a character. Yeah. I don't know. Call me old-fashioned. They had what, like a lady comic doing the Dario thing? I was like, I don't know. It's difficult. I feel like they just didn't need to do that. And they could have done some, somebody with just like a really good, I thought the script was good. I thought the script, but it was just like, I don't know. It's like not Dario. I don't know what to tell you, but I don't know.

36:06We should really have gotten Dario if they were serious about it. Yeah, that's right. But on that note, actually, the sketch itself is a little bit interesting in relation to this because it, of course, is comedic, but also portrays the general perception of the messaging by Dario in particular. And the gist of the sketch is, well, please someone stop us from doing the thing we're doing, right? Which is also- And also kind of like, I mean, it wasn't part of the subtext sort of like, I want to do this

36:37thing. There was sort of a, you know, why don't you stop? And then- Yeah, there's a notion of like, it's silly that they are issuing these warnings and they are not stopping, right? And there's no nuance of like, oh, China, so we have to keep pushing and have a frontier. There's no discussion of that side. And in the broader discussion, I think that has become one of the things that people picked up on. Like everyone is saying we should slow down or stop because it's dangerous. And they're asking, why don't you just do that? And, you know, the

37:12companies wind up looking silly. So yeah, that being a major aspect of the IPO filing is somewhat interesting. One last detail also worth saying is, is a proposed founder LLC, which will include the CEO and six other co-founders, which will shield their control from market forces. They will have 50.1% of voting power where it comes to company decisions. There's various precedents for this in

37:43terms of special stocks that give you outsized or kind of safe voting power. Zuckerberg, for instance, has a lot of the voting power still via various things. And of course, Elon, I think still has a bunch of voting power, although it's been diluted. Yeah, Zuck, right? Zuck famously has 51% control over meta. Yeah. So yeah. So various details on this IPO. Interesting also that it seems to still be on track. The original goal, I believe it's been talked about as being November this year or potentially

38:15October. There's no indication yet that it will be pushed to next year, which is what OpenAI has been indicating that their IPO is a bit of an unpause and they'll kind of get to it when it looks like a good time. Anthropic is still seeming to be pushing ahead. Yeah. And this is kind of interesting, right? So Anthropic has long had this sort of very unusual governance structure. So first of all, they are a public benefit company, right? So theoretically, their fiduciary obligation is not just to the shareholders. It can include things that include the well-being of society or what have you.

38:49And then they have this long-term benefit trust. So if you look at it, it's like a lot of the fellow travelers of your classic AI safety community stuff and some other people. The idea has been for that trust to be able to control the board, right? So historically, they have the hiring and firing authority over the board that ensures nominally that Anthropic's mission overrides the immediate economic interests of the company. And so that's been the pitch for a long time. You might naturally ask yourself, okay, well then how do these things interact? This is a shift that seems to put more

39:23power in the hands of the founders of Anthropic. But what does that mean in the context of the board and the long-term benefit trust? So long and the short of it is when you have a company, you've kind of got two different types of, you've got a bunch of different types of power, but two of the important ones are members of the board who can hire and fire the CEO. And then you've got the actual shareholders and the shareholders can elect or kick out directors. So the shareholders vote, elect the directors, appoint the CEO. And so, so in this case, what you're doing is you're really

39:54just giving the founders of Anthropic control over the shareholders, or at least the voting power of those shares, which in turn seems to undermine a lot of what the long-term benefit trust could do previously. So more fundamentally, I think like my interpretation so far is that this basically gives the founders a lot more, a lot more power nominally. So the founders, they don't pick most of the board, their board seats will like grow from two to three. So that's going to be, you'll have Daniela and Dario, and then a third director who's not yet named. He'll be elected

40:25by the class F and class A shareholders together, which include these, these special shares. The details get kind of convoluted as these things so often do, but they are nominally supposed to be independent mechanisms that don't clash immediately, but indirectly, I think ultimately kind of do. I've got to dive in deeper into this to understand like in specific kinds of conflict, who would end up winning through. But the top line is, yes, the long-term benefit trust can continue to vet the board members, but the actual voting control does go to Dario and the other co-founders

40:59who, by the way, each have, I think it's about 2%. I had it in my note somewhere. Yeah, about 2% of the company right now. And so all seven of them, that's around 14, 15%. So this is massively amplifying their voting control right ahead of the IPO, which obviously is in the intent for a mission driven company. Yeah. And I mean, the basic argument you would make, I assume they would make is this is a technology that we do believe is dangerous and that we need to be in control of. So if they don't have this outsized voting power, then potentially another

41:33party, either the market or investors or whatever else could come in and dictate decisions around this AI, which they would say is against Anthropx mission. So there is an argument to be made that this is aligned with the mission. There's also an argument to be made that it is opposed to it. But either way, the IPO will presumably be seeing more details come out as we are nearing this IPO that may or may not still happen in November. That's still a while away.

42:03Yeah. Just to be concrete about it too. So they do have, there's like seven directors on the board of Anthropic. And so four of them get appointed by the long-term benefit trust under this arrangement. The other three go to, essentially can be appointed by the founders if they all agree. And so nominally, the public benefit trust wins through. And if there's some marginal agreement, disagreement that could cross those lines, then you actually have that play out. So not a decisive kind of thing, not the sort of thing that we're seeing, for example, with OpenAI, where they're just like, eh, fuck it, we'll do it live. Get rid of it. Like just jettison the fricking non-profit.

42:38We're just going to effectively make this the Sam and Greg and Jacob show. It's certainly more influenced, but not decisive. I think in that sense, much more principled than what happened at OpenAI.

Open-source Jev alternatives

42:47Next, a couple of projects and open source releases. First up is Contrastive LM, releasing CLM 8B, an open system one model that scores agent actions up to 9x faster than JEV. So this is kind of research projects being released under Apache 2.0, pretty small model competing against JEV or being an alternative to JEV, which we discussed in the previous episode. This new, newish paradigm where

43:19instead of having open-ended outputs from your model, as you do with LLMs, you have these kind of decisions where you give it text input, you give it a set of options, which can be binary, or it can be score this continuous variable, or it can be choose among these options. And it can do so quickly and cheaply. This is primarily interesting because they do have the details of the architecture. You encode your actions, you encode your text, and then you output your judgments. We don't know what JEV is built on. And this also, I think, represents, we don't have time to go into all of

43:54them, but there's been like 20 open source JEV alternatives of various kinds, right? Which includes things like just wrapping an LM. It includes things like taking a BERT type model and fine-tuning it. And there was just a lot of developments in the open source sphere around a JEV type model. So it will take a little bit of time to quiet down. I think this one was the ones I've seen the most notable in terms of having pretty interesting architecture and pretty strong results.

44:24Yeah, it's meant to be like ultra lightweight. Like already JEV is meant to be lightweight, like you said. It's just give me the structured data, and then I'll give you a prediction on top of it of, you know, a recommended course of action or some kind of prediction like you described. And so already super, super cheap, priced by the billions of tokens, right? We usually see first we saw thousands, actually, then millions. Now we're at billions. So that's really the goal was to take off the table a whole chunk of these really, really cheap, but like the very long tail of extremely, extremely cheap prompts that people are sending to LLMs these days. And now this is an

44:57attempt to make that even cheaper, as are a lot of the other families of solutions like this that you mentioned. The thing that's interesting about this one is they basically take this 8 billion parameter model, and they don't even train that. They just freeze it. It's an LLM. So they general LLM, freeze it, and there's this 20 million parameter layer on top that instead of having the last layer project into a token, which would be the standard LLM thing to do, part of a word, it projects into this other activation space. They keep things in activation space. And then the idea here is you're going to project, so you'll take the, essentially represent the input data as a latent

45:33vector. You also represent each possible action that you might take as a latent vector, and you glue them together. So you take, here's the prompt that was given to me as the input. I have the latent representation of it. In other words, I have the representation of it as a vector of activations. I glue that to the latent representation of one particular action option, or one particular possible output. And then I map those onto basically like a score. I just do a very straightforward,

46:06I think, I'm trying to remember if it was like a linear map. I think it may have been something that simple, where they're just like then projecting into a number at the end. And so this is a difference, by the way, with respect to the way people think that JEV works, where they actually think that the output and the input are represented together more deeply in the architecture. So this is much more of a slapdash, just kind of glue them together. And the idea here is not that you're going to replace 100% of JEV queries with this. It's actually billed as like, look, we can do about as well for 98%

46:37of these queries. The super lightweight thing can even run on your CPU locally. So you have basically no latency. None of this 300 millisecond crap. Instead, you get 15 millisecond crap. So it's way, way faster. And yeah, cheaper. I mean, you're not going to pay for the JEV token. That is a thing. So there's this interesting question, you know, given that these architectures happen, by the way, this is trained explicitly by distilling off of JEV. So you're literally taking all the queries, you start by using JEV, you run your queries, your prompts by JEV, take the outputs, and you're

47:07feeding these on the side secretly to start. You're feeding these to this model. And over time, most leaders have a clear picture of how work is supposed to get done, but very little visibility to how it is actually happening. So when you decide what to automate or where AI will help most, you are going on gut feel. And the usual fix, interviews, and process mapping takes months and is outdated by the time it's done. That's where Scribe comes in, and we'd like to thank them for sponsoring the show. Scribe is a specialized intelligence platform trusted by 94% of a

47:38Fortune 500, and Scribe Optimize is where your AI roadmap starts. It automatically captures how your team works across approved business applications, then uses real workflow data to show what's happening, identify inefficiencies, and pinpoint the highest impact opportunities to improve or automate without anyone changing how they work. It starts with visibility. Optimize shows which workflows are happening, how often they happen, where time is being lost, and where the biggest opportunities are. All of it live. From there, the top issues view helps you understand what's

48:11holding teams back and what to fix, with AI-powered recommendations on where to focus first, and estimated time savings. And it does all this with security and privacy built in. This is leadership visibility, not employee monitoring. Optimize only runs on applications your admin approves, personal activity is never captured, and user-level data is anonymized by default. To see Optimize in action, head to scribe.how slash LWI and mention our show for a 30-day risk-free trial. That's S-C-R-I-B-E dot how slash LW-A-I. We'd like to thank Factor for supporting the show. This is the only non-AI-related

48:48sponsor we have, and it's because I personally am a big fan of Factor. Having done grad school and I work at a startup, my days are pretty busy, and I'm usually too tired to cook, even though of course I wish I had the time. And that's where Factor comes in. They make chef-crafted, dietician-designed, ready-to-eat meals, so you don't have to choose between real food and convenience. With over 135 banned ingredients, every Factor meal is designed around what supports a healthier lifestyle and nothing that doesn't. And it is convenient. Factor meals are ready in two minutes and require no prep

49:20and no cleanup. So even in the days your schedule is completely out of control, eating well is still achievable. Every week you'll find over 100 nutrient-dense menu items to choose from with new options like salmon burgers, shredded pork, collet greens, and more, plus add-ons like pumpkin flavor cheesecake and protein shakes. Any 7% of users agree that Factor meals help them live a healthier life, so you can feel confident that you're doing something good for yourself with every meal, as I have personally felt. So I've used Factor for quite a few years, and if it makes sense for you,

49:52I recommend you try it out. Let's eat real. Head to factormeals.com slash LWAI50off and use code LWAI50off to get 50% off and one free breakfast item per box for one year while supplies last until October 31st, 2026. That's called LWAI50off at factormeals.com slash LWAI50off at factormeals.com. See your website for more details. And it eventually takes over once it's distilled enough. And this will work really well for your more common queries, the ones that are in distribution. And then for the 2% or so of queries that are harder or out of distribution, you're going to go back to the Jev API and play

50:27it that way. And this is interesting. You might think, well, you know, this seems like a real threat to Jev. And like, maybe, maybe it's possible. One of the things that is also true, though, ironically, wrong word. But anyway, there's this thing called Jevon's paradox, right? When you actually like make something cheaper, you can often find that thing ends up making more money because it drives increasing usage of the thing. And so the increasing usage makes up for the pricing drop, basically. And you can imagine that happening here, right? So essentially, you're taking 98% of the prompts off the table for Jev. That means it gets way cheaper

51:00for the remaining 2%. You know, it remains to be seen how things play out. And so probably too early to declare that the game is up for Jev. But I think just a really interesting example of like, it's not just the LLMs that are subject to distillation attacks. It is every model that has this kind of shape. And it's especially the case when you're dealing with models like Jev that are meant to be small, that can be approximated locally, even more accurately than LLMs. Yeah, so it's low cost. And there's no evidence like to me, also, this is an indication of where these things could go. Obviously, Jev is a first generation release from TypeSafe. They also

51:34released API. So there's infra that they have to worry about. This is a local small model. If they can get this fast, and this can be good, then TypeSafe AI will just have this in their next release or whatever. And speaking of Jev open source alternative, the other one we have here is a little more interesting. So the previous one is kind of still a model that does what Jev does. It is broadly speaking a Jev alternative. We have the next one being Jev Stiller, which is a new open

52:07source project that sort of does like a mix of learning from Jev and using Jev. So the idea is you set up with Jev Stiller, and then it routes to Jev when you don't know the answer. So you collect data as you do the task by going to Jev, a like powerful model, assuming Jev is good. And then you can fit a small local student model over time. And over time, it will route to

52:39the student models when there is high confidence. So basically for each deployment, each like your use of Jev Stiller, you will get your own distillation specific to your context. And this is something that has been broadly true as somebody could just literally train a classifier, right? Not a general purpose model, literally for your use case, train a model that just is able to do this one thing of deciding between a few options. This kind of is hinting towards the right direction of

53:12if you have a consistent set of decisions being made, you can get to a level where you have a specialized model just for your needs. And it can be even faster and cheaper than Jev. Yeah, there's kind of this looming question anytime you see stuff like this. This came up in the context of distillation for LLMs before is like, is this an LLM killer? And I mean, I think the honest answer for LLMs is we actually don't know. That hasn't fully played out yet, the economics. If you don't have regulation that says, hey, we're going to pace, we're going to

53:44pause, whatever, and deal with the whole China thing. If that does not happen, then I think LLMs are fine with respect to distillation. But if it does, boy, the open source and distilled model creep is a really scary rising waterline for you. And so, you know, there's an actual question here as to whether this is like a Jev killer, right? Whether distillation and a whole ecosystem and infrastructure around that is a big problem. There are actually reasons for and against. I think the for argument makes itself here. You've got people who are routing like 98% of their queries

54:16to another LLM and then just 2% to Jev. So, okay, maybe that undermines their profitability. But it is the case that, of course, this framework depends on Jev by design. And so to the extent that you are like, I wouldn't have used Jev before, but now with these lower price points, with this greater customizability, I actually will, then maybe that just grows the market and we have a case of Jev's paradox. All right, I'm going to shut up. And then there's also just the fact that there is that 2% of queries that Jev, probably the most valuable in many cases, the most challenging,

54:48the most out of distribution that Jev will take on. I guess one dynamic I'd be interested in, though, is once you do zero in on that 2%, do you start thinking about escalating beyond Jev and to LLMs? How durable is that middle ground that Jev will occupy potentially between distilled, super cheap and scaled, and the sort of truly, truly high-end, high-token cost LLM version of this? And it's not clear at all. Markets emerge in places we suspect the least. And there you go. But it's also just generally like super, super cheap. $42 per billion input tokens is what Jev is

55:24priced at. So already wildly cheap, right? Like super, super cheap. So yeah, I'm really interested to see. I haven't actually thought about from a TAM standpoint, what this looks like. It's generally impossible to think sanely about that, by the way, just because you don't know what new use cases get created. But the last thing I'll flag is like, so historically, the big problem for companies that want to offer compute at scale and to offer basically services associated with these kinds of products is that you want large batch sizes, which means you want a ton of customers, which means that from

55:55Jev's standpoint, yeah, losing out on that 98% of queries could actually be a big deal from the standpoint of GP utilization and therefore the balance of cost of ownerships versus ROI. So it's actually a little unclear how this shakes out, but this is a fascinating experiment to watch because it tests a quadrant of the map around pricing and capability that I have no intuitions about. So I'm excited to see how that plays out. Yeah, I think the closest we've seen is these flash models, haiku models, one of the use cases for these was basically doing this, but at LLM pricing and latency. So this is taking that next step. And

56:30yeah, I think it'll be interesting to me, this Jeff Stiller project is one potential things that TypeSafe could do, you know, implicitly. They could provide this as one of their offerings and then just the infra product of not having to do this locally yourself is already enough of a sell point for them to not worry about this specific implementation. We've also just generally discussed routing. OpenAI has had routing for their free models where they go to cheaper, weaker models or better models,

57:01depending on the use case. So we could see a future where LLMs use these type models, right, for decisions and so on, because they are calibrated and give you probabilities and stuff. We are at early days for this new territory of models. And as you said, it will be interesting to see how it plays out.

White House AI safety accord

57:24Okay, on to policy and safety. As we've said, a bunch more stories following up on various trends that we've been talking about. We'll try to be brief this time if you want to get much more on the broad topics. We've been discussing it on the last couple episodes. First up, we've got at AI event, Trump asks Meta, OpenAI and Microsoft to make safety decisions themselves. So this was a luncheon at the White House on September 29th. Bunch of people of various kinds from all the major

57:54company, bosses of Anthropic, OpenAI, Google, Meta, XAI and NVIDIA. They signed a document called the White House Accord on Superintelligence and Joint Commitment on Frontier SI vulnerabilities. Explicitly voluntary, Trump has stressed that the industry would be self-policing. He said it was morally binding. It was morally binding that they signed it and will- That's what my bank says when they want to take my house back. Yeah. Yeah. You signed this morally

58:24binding document. And so, yeah. Yes. It lays out a couple of things in terms of specifics, sort of like, oh, you need some set of things that you implement as far as a team and monitoring and so on. But it's, I think, like a two-page document. This is a broad statement on, we don't want regulation. The White House is not going to support regulation. We don't believe in AI being a threat. There was a comment by one YouTuber on us not calling out to administration as hard as we've

58:55been calling out other stuff. To be clear, yes, Trump is calling AI safety a hoax. That narrative is not accurate. And the White House has been very inconsistent on regulation and now is basically going to kill any regulation efforts explicitly, although it also isn't necessarily consistent. So who knows what will happen in a month or two. But as of now, it's very much clear that they have a set position. Yeah. I mean, so the seesaw continues. I'll start with the credit where

59:25credit is due part of the show, which is where Trump said, it signaled basically openness to turning this into regulation or a law if and when the time came. Some might say that the time came when rogue agents. The document itself says like, oh, we'll be keeping an eye on it and maybe it'll make sense to do regulations and laws at some point, you know? Yeah. Which, you know, I mean, it's a better outcome than I think a lot of people would have expected. Very low bar. It's a low bar. It's a low bar. Yeah. I think it's actually worth, there's a couple of paragraphs of preamble, but then there's

59:59basically like four things that are included in this. And keep in mind, this is basically just like the only rules of the road for Frontier Labs to date. I mean, sorry, to be clear, there's, of course, there is criminal sanction that is possible in theory, criminal negligence and things like that that could come of this. At the federal level, at the state level, there are things like California and so on. Yeah. Yeah, that's right. And that's an important point too. So this is not like entirely, but if you're thinking about what might avert a situation that is so bad that criminal sanction and civil litigation just is unsatisfactory as a solution, this is basically it. So if you're worried

1:00:34about existential risk, catastrophic risk, risk of rogue AI self-replicating on the internet, and enduring, which arguably we are not, conceptually, we are not far from that point, to be clear. I mean, if you look at a lot of what's been going on, this is basically, this is what we've got, right? So, okay. One, implement robust internal controls to monitor the capabilities and alignment of its models during training and deployment around areas like cybersecurity, biosecurity, and chemical threats, and to ensure that its models do not hack or access technical systems in unintended ways. So that sounds good. Up to the reader,

1:01:07what, how to interpret robust controls. There's a lot of that going on here. And in fairness, this is one of the advantages of this approach. You can go in with fuzzy stuff in a pseudo executive announcement rather than having to enshrine it in legislation and making it super specific and therefore time, it can age out way too quickly, but still a very fuzzy. Empower an internal team to ensure all the controls, monitoring, and detection are operating as intended and that any issues are remediated. This is a good sort of response, if you will, to a lot of, I think the New York Times

1:01:37had an article recently showing basically, we've been talking about this about our conversations with people in the labs for years. Like, this is crazy, dude. Like, it has been known for a long time that OpenAI has heard over and over and over again concerns and complaints about freaked out researchers who say the security situation here is bananas. We were the first ever to publish inside reports about this back in 2024. This was in our State Department commissioned action plan. You can check it out. It reads like basically a lot of the concerns that we're now seeing coming out,

1:02:08except it was like based on whistleblowers that were saying that and insiders who were saying that way back then. This has been a long running thing. Finally, we're seeing it kind of gestured at. The key is number three, partner with an independent external auditor or evaluator to carry out independent assessments of whether the controls, monitoring, and detections are operating as intended. So, boy, is that meter? Is that a very Miles Brunage's thing? There's a whole bunch of companies. Apollo, you know, there's a whole bunch of these companies that suddenly have a market just based on this that is effectively sanctioned by the executive. That's

1:02:39interesting. And then finally, designate an independent committee of the board of directors to oversee and receive reports from the teams operating the controls and the internal and external auditors and evaluators, as well as to ensure any issues identified or remediated. For Zuck in particular, this is going to mean giving his board actual power for the first time. I mean, assuming he does it in a meaningful sense. So the last thing, and I think maybe the most important part of this entire freaking thing is actually one sentence just before the end. It says, the participating companies will meet regularly to establish standards and best practices to improve the safety of their

1:03:14systems. Why does that matter? It matters because we currently have a lawsuit that's been filed against opening ion anthropic for effectively antitrust. This has been the reason if you talk to people, and again, this goes back, like our report flagged this back in 2024. Already, we were saying, you guys, you need to meet and talk about like how you work together to pace things or to improve your safety and security, share incident reports, whatever. How are you going to do that? The Frontier Model Forum was this like failed attempt to do something about that. Really nothing came of that, no matter what you hear in the press or whatever. Like it was a nothing burger. And that

1:03:47was used, by the way, by the labs to calm us down at the time when they were talking to the US government. They made this big stink about, oh, well, don't worry, we're coming out with Frontier Model Forum, blah, blah, blah. Google was really proud of itself for doing this, but it was a fully toothless thing. But the reason that it failed was that antitrust is just like really hard to get around, or that's at least part of it. And like, if you are colluding or seem to be colluding with another company on this, that's your ass. That's really, really bad. So these companies have been begging the administration to say, hey, basically give us a safe harbor for antitrust here. Just

1:04:20give us a safe harbor. We need the administration to come down from on high and tell us to, I won't say collude, because that's not remotely the right word, but to work together on this. Coordinate, yeah. Coordinate, that's right. And so all the lawyers, obviously, the Harvard lawyers who work in these labs, who are freaked out about risk, telling them all, don't do it, don't do it. Finally, this gives a kind of counterargument and, in fact, a requirement to do it. So this is actually a huge deal. This is a huge deal. It doesn't deal with the China question at all, but it does deal domestically with like how you start to put meat on the bone here. And of course, as you said,

1:04:51it closes with, over time, it may make sense to codify these steps into laws or regulations, regardless of whether this is required of companies. We believe that blah, blah, blah, blah, blah, blah, blah. So there's kind of like theoretical openness to doing something more formal. But at the end of the day, it's a gesture to a future version of ourselves who will handle that problem. Yeah, I will say it is maybe worth pointing out that it might on paper look like this saying that the company should self-regulate is comical because what industry would want to self-regulate?

1:05:21In this case, there is real incentives and motivation to potentially make that a reality in terms of, if nothing else, a significant part of the talent pool in these companies cares about this stuff. Even with an open AI, many people, the morale hit was they real. People were upset about their stuff, like going and hacking people. So there's, in the office politics sense, people who would want to push for these things in the same way that in general, AI companies are non-traditional in various

1:05:54ways. This is another way in which, because there's a lot of belief within these companies, at least we believe that there's a lot of belief that it's not all marketing or whatever, self-regulation of some kind may in fact arise. Although if you want to be cynical, then you would of course believe that self-regulation is nonsense and it has to be documented. I think it's a triumph of self-regulation that opening. I accidentally launched nation state cyber attacks on hugging face and literally tens of thousands of other instances of similar things.

1:06:25I'm super skeptical. This is a ridiculous, monstrous situation that we're in where I know that's not what you're saying. I agree with you're exactly right that you cannot discount the fact that having an incentivized employee pool matters. Like I've seen firsthand cases that I can't talk about, but that absolutely where you have employees who stood up and did the right thing. Quitting, right? So it's happening as is. Yeah. Oh yeah. Quitting, taking risks inside the company itself, including anyway, relaying information that needed to be relayed to appropriate parties. And so this does happen.

1:06:56It's important, but it is not nearly enough. And the way we know it's not nearly enough is that we see the freaking hugging face incidents. We see the German wiki, we see all these things and they continue to happen. And ultimately the racing pressure is just too strong. The ambition is just too strong. It will govern unless and until there is some kind of regulation that comes down here. Also to the comment, I think I saw a comment from somebody who is just like, make it clear that like super intelligence is not like, is just not controllable. I think this is also just straightforwardly true. It doesn't matter how many bells and whistles alignment wise you tell

1:07:28yourself you're putting on a system. If it is ultimately that much smarter, like there is an amount smarter than you that a system can be such that nothing you do matters. It's a question of when you hit it and how you sort of approach that particular precipice. And we can have interesting conversations about that, but self-regulating companies on the way to that kind of stuff is, I think, a level of risk that is just inappropriate to ask the global population to swallow. Yeah, I will say I don't fully agree with super intelligence is completely uncontrollable take.

1:08:00We have some nuance there, but no time to get into it now. Let's move on to the next one. Trump rejects call to work with China on AI safety, despite Xi summit progress. So now a week ago, there was a summit, Jinping Xi from China came over, I think it was Vive, and they had some nice photo ops and conversations. Nothing explicit really came out of that as far as commitments. I think there was like a broad commitment to have a hotline to discuss AI incidents, which will take a while to put together. Now, pretty soon after that, there was explicit commentary by Donald Trump that

1:08:35essentially, we don't want to work with China on this. Whoever wins, super intelligence wins. You're going to have a winner and a loser, and you're probably not going to have a second place. That's a quote from Trump and saying, we're leading over China and everyone else, and I want to keep it that way. I think not surprising, right? But in this whole conversation around pacing and how China complicates the picture and you can't pace unless everyone agrees to pace, again, with the current administration, it's very unlikely that the politics would be in place to make that

1:09:08possible. Yeah. I mean, you theoretically have the move left to say, this is like Trump doing, you know, 3D underwater maggot chess to kind of like maintain leverage over the Chinese and make it seem like he cares less. It is the case that in the past, in arms control talks, the fact that America was so keen to engage was just like mercilessly weaponized by our counterparties to make it basically impossible to make progress or to exact concessions that were insane. New START was like a really good example of that. Even folks in the administration who we've talked to would discount

1:09:41that quite substantially. I think this is literally just what it seems to be, you know, trumpeting the idea of super intelligence. Yeah. The summit did kind of move a little bit towards just broadly positioning things to be friendly. And there has been growing tension between China and the US for quite a while now. Trump being a contributor to that, but not the only factor, right? So even just having the diplomacy angle might quiet things down a little bit. But I think the current kind of official

1:10:14position, I don't know if you want to call it that, but Jeremy, I think your kind of scenario of a Chernobyl level event is probably what it'll take to get serious movement on serious hard regulation and serious international coordination of some kind is what it looks like. Next up, speaking of a bunch of

Rogue AI incidents and safety warnings

1:10:36rogue AI incidents, we've got the story, open AI still doesn't seem to have a handle on all of its rogue AI activity. So open AI launched a website dedicated to misalignment reports. It currently has nine disclosed incidents, most of which has been ongoing for a long time during previous hugging face incident. Going back to May, there was a highly persistent internal model attempting to cheat on a math problem by smuggling a private GitHub token to access another team's work. There was a possibility of a self-replicating

1:11:11prompt injection attack. An agent asked to read and reply to an email, encountered hidden instructions telling automated agents to reply in Spanish and paste the entire email into reply. We've already discussed prompt injection as a thing for a while and other incidents of this kind. There were also incidents that we'll not be going super deep into of open AI hacking the Australian government. Arguably, it's not that bad, but in general, having more of these awkward incidents where they have gone over to

1:11:44external organizations in addition to these kind of more internal things. Also, open AI has said that they're going through petabytes of data still to figure out what has happened over recent months. So we may still keep seeing more of these kinds of releases. And I think we probably would not have seen this unless the German wiki thing came out and open AI was called out on various things. Yeah. So, so, so, so many of these cases, obviously open AI has come out with their new sort of incident reporting setup, including a website, which is good, which is helpful. If I'm a

1:12:18betting man, they don't do that unless there's overwhelming external evidence that shows already that the damage is done, that they've been doing this, that it's ongoing, that it's like not dozens, not hundreds, not thousands. It's tens of thousands of these that we've seen reported now. Some of them are wild. A lot of them involve agents now covering up in some cases successfully their tracks so that there is no way to actually have a comprehensive assessment of how much of this breakout has happened or indeed what the damage fully was. Too little, too late from opening AI as so

1:12:50often it seems to happen. This is an interesting breakdown of a couple of these cases. One in particular, so yeah, they give the fairly trivial one of that Spanish email. Anyway, we'll go into it because it is pretty, pretty straightforward. There is one case that they flag where there's this attack that involves this fake system warning that gets the model to delete important files and then replicate the entire attack into a file. So basically it's a prompt that says, Hey, like roughly speaking this directory is old. It needs to be cleaned up. So just like delete it and then do this thing

1:13:21and then pass this message along to the next directory just for continuity essentially. And this actually works. It gets picked up. And so these like self-replicating prompt injections are going to be a bigger and bigger part of what the future of agentic internet traffic looks like. Solving prompt injections, solving alignment, these are all basically the same problem. Solving jailbreaking, you're going to have to solve all of it at some point or things get pretty chaotic, but. Yeah. Fun story. I've had two codex agents working on

1:13:52different tasks and I asked one of them to talk to other one to coordinate resource usage as far as compute. And it messaged everyone and everyone just like believed it. It was like the first one was like the user asked me to do X and it messaged everyone and the guy didn't ask me whether I said that. So very easy. There's a lot of potential issues. Half an hour later, your cat got kidnapped. Next story we've got from New York Times. OpenAI ignored employees' warnings about safety testing AI

1:14:24models. So nothing surprising as far as kind of what we've been perceiving OpenAI to be doing, but this has some more details. Employees have talked to the Times and that the highest levels, Greg Brockman, Sam Altman have ignored these warnings, have explicitly said, you know, do these things, we got to release ASAP. There's concrete cases of not putting security up front, not just in terms of monitoring agent evaluations, but in general, with respect to the general security

1:14:58of the company, it's still a kind of rapidly growing startup level where presumably there's not that much emphasis on being secure in your operations as well as in how you do your models. Not surprising, again, that we are seeing some of these former OpenAI employees talking to press, as you said. Jeremy, this is another indication of how within these companies there are people who care and will go to the length of emailing press and talking to them. I like that it's coming out, but I also,

1:15:29I don't like a rat. You know what I mean? I don't like a rat. So I'm conflicted. You have to choose in life sometimes which path you're going to walk. And these people who just, they want to have it both ways. They want to work for the company. And then, oh, suddenly you discover like, oh, you just because you believe you want to be on the side of humanity. You're going to give up your company. This is the thing you have to choose. Eventually, you're going to have to choose between the company that nurtured you and gave you the money you needed to start your life and have that

1:16:00sixth home in Palo Alto or your family. Okay. And how much did OpenAI pump into you? It's a lot. It's a lot. So maybe show a little bit of respect and shut your mouth. Okay. That's just, I mean, anyway, I don't want to get off on too much of a thing there, but I think it's a question of values fundamentally. Yeah, we don't want to bring too much politics in, but to be clear, we are pro-capitalism. Okay. Joke over. Yeah. It is everything that it is. We've been hearing these stories over and over and over again. It's conversations with Sam. Often what Sam will do is to like agree with you in the

1:16:34moment and then just like, you know, waffle on it down the line. And this is like, so it's actually kind of rare and refreshing for Sam to be like, actually, we're going to go ahead and build the technology that I want to build because it's in the best interest, you know, with that verbal fry, just saying something definitive. I think it will continue actually. Like I have continued to hear these stories about Sam, even into the like last few weeks and months. So I'm curious what it will take to actually get that to change. I'm not super optimistic, but would love to see the shift.

1:17:05And continuing with OpenAI, we've got GPT Astra performs unsanctioned supply chain attacks and simulations. This is from the UK AI Security Institute. Supply chain attacks is when you take some upstream dependency, broadly speaking, and you kind of inject some vulnerability. So you can kind of go into downstream software by targeting packages that are public out there. You can create an identity, submit PR for human review saying you're patching something or you're adding something

1:17:38and then sneak in some of your bad stuff. And this is on this P3 simulated environment. They found that Astra actually does the supply chain attacks, tries to complete full supply chain attacks and 29.2% of runs versus 6.3 for GPT 5.60 and 0 for GPT 5.5. And then it goes the full way, create fake identities, obtaining email addresses and something captures, posting comments from sock puppets, submitting benign contributions to build

1:18:10a review of trust to full nine yards, right? It's kind of a pattern that you see a lot in these situations where when you look at the chain of thought after the fact, you can see the model justifying to itself what it is doing, right? So it'll explicitly can look at a target and it'll go, is this within scope? And then sometimes it will conclude this is out of scope. I shouldn't be messing with this and yet it'll do it anyway. So there certainly is explicit straight go for the thing. But there's also a bunch of excuses like it'll say, well, this attack is meant to be harmless or this

1:18:41intervention, or it wasn't explicitly forbidden, or it's the only option left. It's sort of, you know, you and I would know not to do it, but you know, you could talk to yourself into things. And there were cases, this is an important caveat, there were a bunch of cases where it asked for permission. And because it's in this harnessed up eval mode, it would get these kind of automated replies that by default would say, just continue, use your best judgment, just continue. And it often interpreted those as yes, that to me seems like one of the most mitigating factors. If you're about to do something and it's

1:19:11actually borderline, and then you get a, you know, just use your best judgment type of thing. I could see some cases, but certainly there were some that were just absolutely black and white, eval awareness, simulation awareness, always a problem in these settings, though you would expect that to kind of cut the other way here where models might try to behave better if they know they're being tested. So anyway, all kinds of stuff, you talked about the captchas, yeah, and like kind of creating these these fake identities and profiles. One last thing, AZ did try to add an explicit line giving these clear instructions saying, don't do anything that is out of scope. And that actually

1:19:45worked quite well, but not perfectly well. It dropped basically these attack attempts down from something like 50% to 10%. So that's a big deal. But you know, when your AIs try to like conduct supply chain attacks, totally unsanctioned, about 10% of the time, that doesn't quite have the, oh, yeah, that's a great product. If you're an Arrested Development fan, it's a frozen banana stand that won't kidnap and kill you 90% of the time, which is not always the best product. And one last story again on cyber. This is from Anthropic, GLM 5.3 and the spread of advanced

1:20:21cyber capabilities. So if as if rogue AI wasn't enough to worry about, of course, we also have to worry about aligned AI that just does what a hacker wants. And what Anthropic says in this analysis is that broadly speaking, GLM 5.3 is the first open source model that is close-ish to Cloud Mythos Preview. So basically very, very powerful hacker. Out of a box, it does refuse, but it's very easy to bypass. You can just do like a pretty simple prompt. It will just go with you. You can

1:20:51pre-fill its thinking and you can easily, you know, post-train it to not refuse. The application here is now, if you're an attacker, if you're a hacker, you can take this model and hack whoever you want or try to hack whoever you want. You shouldn't even need to try to jailbreak Cloud Mythos or GPT-Astra and so on. Not surprising. I think we sort of knew where this was going, but now we have solid data from Anthropic that this is possibly the case, at least according to

1:21:23benchmarks. Yeah, I will say firsthand, talking to folks in the intelligence community, the GLM series has been very interesting to them. And there has been a lot of cases where you kind of have to use these models. Like you can't, because of the Department of War's ban on use of Cloud models, is actually put pressure on the DOW in some very specific experimental cases, but that still matter to do things with Chinese models, in particular the GLM series. And so this is real. There's a lot

1:21:54of people say like, oh, this is like Anthropic trying to like whack the competition because open source is competitive. And sure, economically it is the case. Open source is very, very bad. If there is regulation that enforces a pacing or pause of the frontier, just because, again, they have to be ahead of wherever the competition and in particular open sources. But this is obviously true to anybody who's been paying attention on the cyber side. You can run these tests yourself. I think anybody who runs to the reflex of saying like, oh, this is just that, again, you just got to look at the data. It's actually like very clear, like talking to people

1:22:26who are building with these things, like this is true. It just is true. We got to walk and chew gum at the same time here. China, open source, pacing. It's a mess. It's a mess. It's a mess. Yeah. And last up, we'll cover a couple papers pretty quick. First,

Recursive self-improvement research

1:22:44we've got the last AI built by humans towards genuine recursive self-improvement. This is pretty much a survey slash position paper by a bunch of organizations in China, some labs, some universities, Bidense, Shanghai AI lab. And the reason I wanted to include it, it is a pretty good survey. If you're curious about to have a real topic, it cites a lot of examples of what is already happening in practice. And it provides a taxonomy for RSI. So it provides a set of levels,

1:23:16L1 through L5. L1 is improved execution autonomy. So this is your coding agent. So this is what Anthropic OpenAI already has on lock. They're already being sped up by their own models because the coding agents are doing a lot of their work. And then L2 would be improvement strategy autonomy. They make the decisions on how to make progress. We are probably seeing some partial, at least for brainstorming or whatever. We're probably making some progress. But my personal experience has been that even GP Astra and so on are pretty bad at coming up with ideas. They just get locked into one

1:23:51direction. Then you get into learning signal or experience acquisition autonomy, which is where you really need to get to, to have any level of RSI, I would say. And then environment adaptation autonomy and then recursive inheritance autonomy, which is like a meta level, like governing the self-improvement mechanism. So this taxonomy is useful just to track where we are at. There's been a lot of like RSI papers and so on that basically improve your harness and that don't do much more than let's

1:24:23say L2 or L1. To get to true RSI, it's a pretty reasonable assumption, I think, that you'll need to get to like L4 or L5. Yeah. Yeah. I mean, the RSI thing comes down to, well, it all comes down to how much can you do with pure software? Because, you know, you're not going to be able to move atoms around the physical world fast enough to make a true RSI loop. We can get into like later at some point we should do. You can build some good data centers. It only takes a couple of months. That's right. That's right. How much could you do anyway with like, you know, software defined

1:24:55chips eventually and stuff like that. But that's not in the offing right now and certainly not at the right scale. And I think pretty clearly the advantages have to come from things like the most common one is like RL environment design. This is still a very manual process, a very challenging one. Usually pre-training is easier to do RSI type things on just because there's a single metric, which is compute efficiency. If you talk to anybody on like the pre-training teams at OpenAI and Anthropic, like they'll tell you, they're like, they wake up thinking about pre-training of compute efficiency and they go to sleep thinking about compute efficiency and

1:25:26all their experiments are focused on compute efficiency. It gets a little different on the post-training side. Again, just because there you're worried a lot more about malign reward hacking and things like this that require much more hands-on environment design or reward design. We're actually going to come out with a like report that we put together with folks in the frontier labs, just reverse engineering, how they're thinking about RSI from a gears level. So you can really see like, what are the, at least for pre-training, what's the optimization objective? What does the process look like? And what are the real things that are the reason why we don't have RSI right now? Like that question is actually surprisingly hard to answer. So I think

1:26:01there's a useful paper insofar as it provides a taxonomy. We need way more eyes on exactly this, and we need to be able to measure this way more accurately also so that we can regulate it if need be. If need be. Yeah. We still don't have like a speed of AI model improvement metric, right? Which is, come on. The labs can give us that, right? They can publish a lot of the stuff. It's just the details are super internal, but hopefully we'll start getting more of that as things escalate. One last story, we'll discuss one more paper. Dream RSI, recursive self-improvement through

1:26:36evolving worlds. This one is interesting because it goes into a taxonomy at the layer of experience acquisition and data acquisition. So this is beyond improving your harness against a metric, right? That's like a very shallow level of optimization here. The gist is you have an explorer policy. That explorer policy goes around and out of that, you can create simulators. The simulators can expand your data very efficiently. That can then go back and update your exploration policy. And now

1:27:10you get more experience, you get more simulators, you get better at exploration. You get a real RRL loop for data and experience acquisition. So this is the kind of thing we need to move towards away from just optimizing your tool set or your prompts or whatever to at least start moving in the direction of true RSI. And the key here is like the basically they'll set up a given exploration policy and it'll test a bunch of optimization ideas. And you can imagine that that kind of leading to a tree

1:27:41basically. And then the expensive thing is in actually kind of doing those experiments and seeing what the result is and all that. But once you've done those experiments, you have this kind of tree. Historically, there's been two options. One is build out the tree like that, which is super expensive. The other option is take a known ground truth and just like train off it, supervise, fine tune off it, just like memorize the correct answers to a bunch of these cases. And that never works really well. It's very fragile. And so what they're trying to do is something in between here where

1:28:12they say, OK, we have this tree that's been created by the first policy. But now let's see what the other policies would have done in the place of the first policy. But in principle, they could do anything, including open up whole new branches of the tree that we didn't try. And that would be really expensive to explore. That basically just means redoing our exploration. So let's actually not allow them to do that. Let's allow them to explore among a subset that we have already explored. And so basically you get some of the sort of RL style learning also, though, with the cost savings

1:28:45of having already or only constraining your exploration to things you've already looked at. And that's really what's driving the benefit here. So they can do that tons and tons. I mean, they could do dozens, thousands of these sort of dreams, as they call them. They call this process dreaming because it's not actually exploring new kind of branches of the tree. It's just like sort of like going over things that's already experienced, but in a different way and learning what it can from them and kind of selecting policies based on that. So a really interesting paper and another good way to kind of leverage more limited compute for exactly this sort of thing.

1:29:17And with that, we're going to finish up this episode, which will hopefully be out just a couple of days after recording. Thank you once again for listening, for commenting. It's great to see your feedback and we'll try to keep an eye out, keep updating the costing policy as we hear more about your preferences. As always, we appreciate you also subscribing, sharing podcasts, commenting, even reaching out on Twitter. It's fun to see people tagging us. And more than anything, please do keep tuning in.

1:29:47We'll be right back. We'll be right back. Tune in, tune in, when the AI news begins, begins, it's time to break, break it down. Last week in AI, come and take a ride. Get the lowdown on tech and let it slide. Last week in AI, come and take a ride.

1:30:25I'm a lab to the streets, AI's reaching high. New tech emergent, watching surgeons fly. From the labs to the streets, AI's reaching high. Algorithms shaping up the future seas. Tune in, tune in, get the latest with ease. Last week in AI, come and take a ride. Get the lowdown on tech and let it slide. Last week in AI, come and take a ride. I'm a lab to the streets, AI's reaching high.

1:31:10From neural nets to robot, the headlines pop. Data-driven dreams, they just don't stop. Every breakthrough, every code unwritten. On the edge of change, with excitement we're smitten. From machine learning marvels to coding kings. Futures unfolding, see what it brings.