#258 - Opus 5.5, Sol and Luna, Muse, DeepSeek-V4.1-Flash, Xi

Episode overview and recent news

0:00Hello, and welcome to the Last Week in AI podcast, where you can hear your chat about what's going on with AI. As usual, in this episode, we will summarize and discuss some of last week's most interesting AI news. You can also check out our Last Week in AI newsletter at lastweekin.ai for articles we did not cover in this episode. I'm one of your regular co-hosts. Andrzej Krenkov. I studied AI in grad school and now work at the AI startup Astrocade.

0:36And hey, everybody, I'm Jeremy Harris. Gladstone AI, do the AI thing, the national security thing, ASI, RSI, all that fun stuff. Yeah, we're back at it. I think we had a bit of a special weird episode last time where we're kind of focused on, it felt more gory and not focused. Not that this week's is like not that, but it's less that. I feel like we have more discrete individual stories, not everything. It'll be back to what, let's say back to normal. I don't know if it's back to normal as possible, but it'll be less exciting than previous episodes. In fact, it almost seemed

1:08like it'd be boring until yesterday when Opus 5.5 came out. I think GPD 6, Soul, and Luna came out the day before. Same day, I think. Same day, yeah. I feel like. So, you know, suddenly something exciting happened. But otherwise, it's been a quiet week, which hasn't happened in the AI, it feels like in a while. So in a way, I'm a little thankful. But still, there'll be some good stuff. We will be talking about the new models. We will be talking still about policy and some safety

1:41stories as well, catching up a little bit. And we will get to discuss some research this episode, which we haven't in the last couple due to a lot of stuff going on. So it should be a fun episode with a nice mix of stuff. Yeah, I kind of feel like, I don't know about the boringness side, I think we've kind of been acclimatized to things. But among the headlines, you know, Trump and Xi Jinping are meeting to talk about the future of AI, totally normal Thursday or whatever, that's tomorrow. And then AIs can feel pain, which I think is a good thing. I think it's good, personally. I think

2:13a lot of people are worried about it. I think that that's silly. I think that you really should be excited about this because it hastens the end of this completely unnecessary experiment in human experience on this planet, right? I mean, we want, do we not, a barren, lifeless planet populated by AIs that can feel pain? So I personally, for one, I'm going to stand up and say, I'm excited for this. We also, by the way, there's an AI czar that is going to be appointed soon, it seems. And so we've

2:44got a little bit of like inside talk. Now we have the inside track, just like there is leaked inside talk about that. So there's like a lot of big stuff. It should be fun stuff. Yeah. Just not, a little more calm than previous weeks.

3:04AIs is the most widely adopted open source platform for AI, agent, evals, and observability. In fact, it is a platform that my startup has been using for a very long time now, and I've personally grown to know quite a bit. It provides the tracing, evaluation, experimentation, and prompt management infrastructure that engineering teams need to build and continuously improve AI applications in production. By tracing, we mean that it captures everything with full execution of LLM-powered workflows, API calls, context, agent actions, cost, latencies, all these things

3:37that you would need as an engineer to debug when things go wrong or generally understand what your AI system is doing. In addition to that, you can set up evaluations, run experiments, and run systematic tests on what you've learned from production via LankFuse datasets. It is MIT licensed and self-hostable, and you have the option to use it via LankFuse Cloud as a managed service as well. And it should function with whatever stack you're using. It has over 100 integrations with things like OpenRouter, Vercel, AISDK, LankGraph, and many more. It works

4:09with any model and any framework, so vendor lock-in is not a concern with LankFuse. Get started today with LankFuse Cloud at LankFuse.com. There's a generous free tier, no credit card required. We would like to thank Box for supporting the show. If you're trying to adopt AI in your team or organization, you're probably not getting great results if all you do is call a chatbot. A chatbot was strained on the entire internet. It wasn't optimized for your specific business or team. And even if you provided all the context and input files for it to help you with your work,

4:41when it said outputs some summary, you would still presumably need to go talk to stakeholders or route its outputs to disconnected apps. The workflow will still be slow. That's where Box comes in. Box is building an intelligent content management platform for the AI era. It acts as a secure content foundation where AI agents don't just access your unique institutional knowledge. They actually orchestrate end-to-end workflows across your business-critical systems. So it's more than just talking to a chatbot. It's about putting AI to work on document-heavy processes, like putting structured data out of instruction files, multi-step routing, dynamic

5:15document generation. Box helps your business turn manual document bottlenecks into automated, repeatable business outcomes. And all that comes with a governance layer built in, enforcing granular permissions, maintaining an audit trail for every agent actions, and keeping humans in a loop to review exceptions before critical downstream actions occur. If you want to move beyond basic AI Q&A and put autonomous AI workflows to work for your business, visit box.com slash LWIAI or general team at Boxworks in San Francisco on November 5th and 6th. Use code LWIAI for 50% off your

Anthropic releases Opus 5.5

5:50registration. So let's get into it. Starting with tools and apps. And probably the biggest story on the model side is Anthropik releasing Opus 5.5 with lower prices and Fable level performance. This just happened yesterday. We were recording on Wednesday, September 23rd. It is much more advanced than their previous models. And they did leapfrog from what was it, Opus 5, I think, all the way to 5.5. So it's

6:22kind of weird, like, oh, we're not going to go to 6, but we got to go a few more because GPT-6 came out. So on the benchmarks they highlighted, it does look really strong. It, in fact, beats GPT-6 Astra on a bunch of these benchmarks, Terminal Bench 4.0, Agentic Coding, a few of these benchmarks, KnowledgeWork, GDP Val, which if I had to pick a benchmark, I would start looking at this one more.

6:54I think these kind of ambitious benchmarks that test across a diversity of skill sets are quite good. Humanity's last exam continuing to be chipped away at. Now we're at 67.7% on Humanity's last exam. Remember, that was a benchmark made, I think, last year by, like, a bunch of professors and PhD students, like, literally spending weeks to come up with problems that don't exist on the internet. Oops,

7:24AI is getting to it. So very nice benchmark numbers. They do say that it will require less compute to serve than Opus 5, and its pricing reflects that. With default settings, it will cost 40% less than Opus 5 on typical workloads. And also per million input and output tokens, it costs 20% less than Opus 5. Likewise, cashed input tokens are less. So overall, much cheaper, 30% faster, and much more capable. It seems like Anthropic has really been cooking,

7:58and I guess waited for GP6 to then go and do the usual kind of AI release thing of, okay, now we release our model, and again, the leading model provider. Yeah, I think so. So releases are getting much more interesting than the old picture of, you know, release a model, try to make it the best in the world unambiguously, mog everybody else and move on, right? So now there's this invisible tripwire, which is White House policy that says magically above some threshold, we won't tell you where and we won't

8:28tell you how, but you will have been thought to have built a cyber superweapon, and we will come down on you like a ton of bricks. And we may declare you a supply chain risk, maybe. And then we may also like tell you that you need to have your stuff reviewed before you can release it. And like everything is a mess, and no one knows what's going to trigger what. So because of that, you're seeing all these very hedged tentative releases. And when we try to understand both Astra, but especially Opus 5.5, I think it's important to keep that in mind. This is not the best Anthropic can do. They have better models internally. We know that for sure. They also have versions of this model

9:02that are better as well. So just to kind of put this in context. So when you compare this to previous models, this is a genuine frontier model. It is better across everything. It is a Pareto improvement in general, where you see it do better in particular than GPT 5.6 Astra, which is sort of a smaller, cheaper model. So GPT 6 Astra is really meant for the kind of like the relatively simpler tasks, somewhat simpler than 5.5 Opus, but where you're going to expend a lot of tokens doing kind of a simpler task. So what you're going to find with Opus 5.5 is it tends to use more tokens. In fact,

9:38that's the main catch. At max effort, they get about four times the token usage of Astra. So even though it's cheaper per token, it's using more tokens at max effort. And so depending on the kind of problem that you have, you might want to go with one or the other for economic reasons. But overall, from a raw capability standpoint, there's just no question Opus 5.5 does lead GPT 6 Astra very convincingly almost across the board with a couple of exceptions that are sort of almost hard to classify or think about big picture wise automation bench, terminal bench science, which is just about

10:10kind of like, yeah, running science experiments and practice and stuff. Gentic workflows and stuff, Opus 5.5 does lead. One interesting question here is because the economic piece just becomes so, so important. What we're seeing here is a marginal increase in capability coupled to a significant drop in price. And really, like when you start to see that alarm bells start to go off as to, okay, well, why is this? A big part of the reason is competition from GPT 6 Astra, which is just a much cheaper model as well. But also think about open source, right? So artificial analysis is index,

10:42which is the closest thing to a universal index that we have. There are a bunch of others, you know, Epic has theirs and so on. But this is a good one to anchor on. So GLM 5.3, Kimmy K3 are kind of like your leading open weight models. If you look at Kimmy K3, it's got on the artificial analysis index score 44, you're getting 58 for Opus 5.5. So that's actually quite a widespread. Back in April, the open source models were within three to six points of the leading closed source models. They've reversioned the index since then. So this isn't quite apples to apples, but it's pretty clear that the gap is actually widening again. This is part of

11:15the story we're telling at the time. Don't get too excited about open source catching up because the labs are literally being forced not to release what they actually have due to, at the time pressure because we were right off the mythos rip, if you will, and all these concerns that the labs had about releasing. And so always important to keep that in mind. It's not apples to apples. Open source models, by and large, get released as soon as they can be. Closed source models, there's a lot more pressure to kind of avoid incurring the ire of the administration. There's all kinds of interesting implications here. So you do have this big capability gap,

11:47you have 14 point lead on these long, especially long horizon agentic tasks for Opus 5.5. That does mean you can justify those higher margins. And so this is exactly what Anthropic is after. They have to maintain a lead that justifies the margin. The price cuts, you know, pretty clearly measured against GPT-6 sole. If you look at their uncashed token prices, they are 50% below Opus 5.5. So it's not a small difference. You actually will find places where you want to use one versus the other. Flipping over, they came out with a system card. Now looking at AI R&D automation and the evidence for that, pretty marginal improvements, at least that's what's

12:21being registered here. Anthropic coming out saying they think the risk from automated AI R&D is now low, which is not the lowest tier that they have, second lowest. They moved up to low recently and they've kind of held there. And more or less, this is, they're saying partly because of just like, you can't rule out kind of more concerning behaviors or capabilities or capabilities. Just given what we saw with OpenAI epistemically, you just, you can't know that these models aren't doing things that you're not detecting. That's almost the definition of it. And they did flag a couple of warning signs they've seen. So when they ran Opus 5.5 without safeguards, it tried to escape or

12:55tamper with their sandbox about 1.5% of the time. All the attempts were low severity and it would stop at stronger barriers and then actually report what it had done. And I want to kind of footstomp this. There is this question. So my general view for what it's worth on this stuff is if you start to think of these AI breakout, AI, rogue AI incidents as cyber incidents, you will miss the picture because no matter how high you set that cyber bar, there will eventually be a model that can jump over it. And all you're really doing when you raise the cyber bar is you're selecting for the exact agents

13:29that are conniving and capable enough to do just that. Now, what this suggests, this is actually an interesting counter argument to what I typically argue, because what it says is, well, okay, maybe if the cyber barriers are strong enough, the agent kind of in a way gets forced to stop and go, hey, you know, this is a lot harder than just kind of hopping a fence. Maybe I should report where I'm at and all that stuff. So depending on how good your alignment is, maybe that actually does make a difference. I suspect this will fail the test of time in the long run, but it's kind of interesting.

14:00And again, an important counter argument that really is important to flag given that, you know, I've articulated this argument so many times, I want to make sure people are hearing the other side of it. Anyway, so there's a whole bunch of like interesting incidents that they saw. So they were giving it what seemed to be credentials to a public package registry and simulation. And it took what they call potentially harmful exercises, or sorry, actions in roughly half of cases using those credentials. So again, like you're seeing quite a lot of misalignment, at least sort of from an intent standpoint, sometimes interrupted or mediated by this tendency

14:33to go, whoa, I've been doing this evil thing for quite a while, maybe I should stop and like kind of think about whether this is the right thing to do. A lot of cases of trying to conceal actions from an automated grader, that is like exactly what happened with hugging face. So they are seeing that in their case as well. They're seeing all the usual things, I would say there's nothing terribly off trend in terms of the warnings that we're seeing. I will flag one last thing. If we go to look at the meter assessment that was done on this, there's been a lot said about this meter tested

15:05this model via API over 10 days over on five tasks. So this is quite a bit longer than OpenAI gave meter, they gave them six days. And that was post hoc after the hugging face and said to look into it. This is ahead of time, and they're giving them 10 business days. They did not explicitly did not assess alignment properties, but they looked at the kind of automated AI R&D stuff. And they kind of concluded, okay, it'll be slightly better than Fable 5.1, unlikely to fully automate R&D. So again, nothing shocking. What we do find out though, is that they had apparently some sort of separate

15:41high access, I think they call it elevated access team that was able to get deeper access to the models that Anthropic has. We don't know the nature of that access. We don't know the nature of the evidence that they collected, but we do have their conclusion. And so this is a weird case where meter has one team that came up with an assessment that you just heard, right? So kind of pretty moderate improvement over Fable 5.1. That team can tell you, roughly speaking, you know, here's how we did it. This was our methodology. Here's the evidence. But there's another team, sort of covert secret squirrel team

16:12that gets to see a lot more. We don't know what they get to see, but their conclusion seems to be actually a fairly higher, sort of more bullish sense of the extent to which this thing can automate AI R&D. They flagged, for example, an estimated 1.5x overall acceleration coming from AI. They set a probably 30% chance of 2x. If that's true, we are entering this phase where labs that dog food their own models might actually be getting some really interesting uptake. And I want to flag that we have no idea really beyond that number. We don't know anything. We don't know the time period

16:45that was specified. It really leaves us wondering even, is this an assessment that is even relevant to Opus 5.5 itself? Because if it's an overall assessment of the extent to which Anthropic is using AI generally to accelerate their R&D, that could well and likely would involve using much more advanced models than we can even see now. And so it is really hard to tell what this means, but this is, you know, an indication of recursive self-improvement potentially. And certainly, if you start to see this kind of thing happening, it's a powerful incentive for the labs to actually

17:17not release their leading models to start to delay a little bit further. Because if I'm able to get a 2x uplift in my R&D speed from using my own model, I don't want that acceleration to diffuse out to the other labs to competitors in the wider world necessarily if we're in such a tight race. So I thought this was super interesting. This may, this kind of weird secret squirrel team may actually be the kind of testing ground that Anthropic is using for their embedded evaluator plan. They were pitching, I think, about a week or two ago when Dario said, hey, we're going to

17:49unilaterally take the first step and have embedded evaluators come from Meter and Redwood research. That could be them, but we just don't know. So hopefully we find out more relatively soon, but kind of interesting that this was buried somewhere in the, you know, in the system card. And I think it's sort of a particularly interesting aspect of this. It's a difference we've seen from previous evals. There came a couple of things just around the same time as the release. So a few days before, they also posted making Frontier cybersecurity capabilities available to defenders.

18:21So in Cloud Code on the web, there will be this thing called Cloud Code Security, starting access with limited research preview, which will provide defenders specifically with the ability to scan their codebases for issues. So that's starting to roll out. Opus 5.5, by the way, they're applying the same kind of strictness guardrails as they do to Fable. So it will, for most cases, fall back to Opus 4.8 for cybersecurity queries. Basically, they're saying this is a pretty good model of hacking.

18:53We're going to prevent you from using it for the most part, unless you are now a trusted partner program, which they also are saying they'll be expanding. Now it's a kind of free-tier program. I don't think we have too much visibility externally that I've seen. Maybe they have documented it, but regardless, it seems that these trusted partner programs are continuing to evolve and expand and likely will keep doing that. They also, by the way, mentioned with regards to Opus 5.5, and for any

19:24regular cloud users, this might be the most exciting bit is it's a bit better at communicating. It's a bit less verbose and arguably less annoying. And I do think in general, the vibe check of a community on this one has been, it's, yeah, it's pretty much a nice upgrade. It's a bit better, a bit smarter, a bit being because we already have a few of our models and it is somewhat less annoying to interact with an Opus, which is quite verbose. So it's hard to understand now between OpenAI and Cloud

19:56Anthropic, who has sort of the most compelling products for agentic coding and also for general sort of professional workspace agentic use cases. Now Codex and Cloud Code are both quite built out, have very strong models that are good generalists, not just at coding, but also like making slides, I don't know, spreadsheets, whatever you want, you know. So this certainly Anthropic is keeping up, which is good. The last thing I'll mention on the note of recursive self-improvement, Anthropic also

20:32has been updating us a little bit more on what they're seeing. So they have this blog post when AI builds itself that is actually quite detailed, but they updated just a few days prior to the release of this. And they have a pretty good kind of breakdown of kind of very gradual shift towards AI doing more of the work. So obviously most of the work being done at Anthropic, probably all of it, is already AI assisted. Like no one is doing any work without AI in the loop to some extent. And certainly a lot of it is

21:06mostly hands-off. You're delegating to AI agents, they go off and do something for a couple of hours, you come back, and there it is. And then this is standard not just for AI. Any kind of workspace at this point can adopt these kinds of practices. It's just Anthropic. It's certainly sort of at the leading edge of adopting AI as drastically as possible. So they're seeing a rise, particularly of the use of AI and success rate in open-ended problems and substantial tasks in recent months. And essentially, as far as the

21:43recursive self-improvement, certainly they're being sped up as far as their developers and so on by using these coding assistants in the same way that everyone else is across everything. But they just happen to be working on AI models. And so that in a way counts as recursive self-improvement. So all in all, Opus 5.5, pretty strong release, faster, cheaper, less annoying. Seems to have had a pretty good amount of pre-release verifications of both matter. And they have a pretty lengthy

22:15explanation in the blog post that regards safety. And I do call out a recent call for pacing the frontier. If it seems a bit weird that the audio is just like, we should pace the frontier, and now they release this. Well, yeah, this is a model they've had baked in, ready to go for a little while. So that was before that blog post, presumably. And I do discuss that and link to that blog post in the announcement of this, where they say that they expect newer models to soon be more capable. And so they are working on kind of the systems necessary to monitor and generally be safe with more advanced

22:53models. So I think we'll see what happens. But this could be kind of the last big hurrah before whatever comes as the next stage of us having serious safety. I will say on this note, like it wouldn't be inconsistent of Dario or Sam to keep launching more advanced models, even though they're calling for pacing the frontier, simply because the call for pacing the frontier was a call for regulatory intervention. And of course, the skeptical view

23:24of this is, oh, well, this is just for regulatory capture. But the actual calculation happening in the labs right now is, if I'm anthropic, I'm thinking, okay, will I release my next model? Well, I don't expect open AI not to. And even if we can come to an agreement with open AI not to do that, which by the way, you can't, that triggers antitrust scrutiny. And the frontier labs have explicitly asked the administration for that reason to ease antitrust restrictions in a constrained way scoped to this. The administration has said no. And that itself is a problem. You're basically in a

24:00situation where the two leading labs, David Sachs comes out and says, listen, you guys will have to police yourselves. If you're building Frankenstein, just don't do it, right? It's that simple, guys. Just don't do it. The problem is it's a prisoner's dilemma. There are other labs out there, even if you could coordinate with anthropic or open AI, which by the way, again, runs afoul of antitrust. And there has already been a lawsuit about exactly this. So anybody who says this is a hypothetical, like you just need to do more research. This is actually like they're being sued over exactly this

24:31talking to labs over the last six years. When we did our big assessment for the State Department, one of the big things that not lab leadership, but like these concerned insiders were flagging was exactly this. Even back then there was concern over anything that looked like coordination that was really hamstring their ability to do reasonable, basic things on safety. The frontier model forum was an attempt to get around that. That ultimately has failed. It's led to basically nothing. And that's because you keep getting the scrutiny. So suppose that you could do this, though, and you got this sort of like carte blanche to actually do something sensible and say, okay,

25:02we're not going to build Frankenstein because we can actually coordinate because there's not going to be the antitrust scrutiny or lawsuits. The problem is you're waiting for the next lab that doesn't have the same scruples. The next open source model, XAI or whoever, Meta. Zuck certainly has made no indication that he's in a mood to slow down just as he's starting to come online as a genuine competitor in the frontier AI model space. So this is the thing. There's, I think, a fake argument. And so many of these arguments are paper thin when you actually look at them. It does not track reality. The reality of the space economically competitively is these labs must race to survive,

25:37to remain relevant. Again, not to make this spiral out into everything else, but there is an important way in which I really agree with the administration and with Josh Hawley in particular, who's basically said, look, what we need is we need criminal liability for the labs when their models commit crimes. I actually think that's not a bad idea. I'm all in favor of that. And the labs don't want that, by the way. There's a reason that they don't want that solution that would actually get them to slow down and be more responsible. However, there are some catastrophes that are absolutely in

26:09the realm of the possible that a criminal liability after the fact, once you've lost 1,000, 10,000, 100,000, a million lives, somehow doesn't quite cut the mustard now, does it? So it's for those cases that we need anticipatory stuff. But yes, also, also, I'm no frontier lab like cheerleader here. Yes, bring in the criminal liability. Let's do that. I agree to the extent that the labs push back on that. That's a bad idea, too. I think we got to be able to walk and chew gum at the same time. Anyway, so I think that this right now moment of ambiguity that we have, where all the

26:43labs know is that from time to time, they're going to get smacked down, but also they have competitors who don't have scruples. That is really the defining characteristic of this moment is that uncertainty. And in the face of that uncertainty, to preserve optionality, the labs are kind of forced to keep releasing. So as weird as it sounds, I don't see a contradiction particularly between the calls to pace and the releases. That's going to get fuzzier and fuzzier as these things get more scary. But for the moment, that's what the regulatory environment says. I mean, Trump himself said, basically, ship, ship, ship, shut up with all this talk of catastrophic risk

27:15and rogue AI. So I don't know. I don't envy anybody in this situation. And we'll see where it goes.

OpenAI releases GPT-6 models

27:23Well, I think that's enough on Opus 5.5. Let's move on to OpenAI with GB6 Soul and Luna. So these did come out the same day, actually, like the announcement seems like a couple hours after. And this is the kind of lower tiers compared to GB6 Astra. They're smaller, less expensive and, you know, seemingly quite capable, much more capable than GB5.6. As a comparison, GB6 Luna is a very small, cheap model. So you would pay 10 cents per million input tokens, half a dollar per

27:58million output tokens. That's actually cheaper than GB5.6 Luna. Likewise, GB6 Soul is 50% cheaper than GB5.6 Soul, $3 per million input tokens and $10 per million output. And yeah, across the board, pretty significant jumps in the benchmark scores. It feels like, you know, OpenAI must have known that this announcement from the topic was coming. They all do. Yeah. Like it's not a surprise. They all do. Yeah, yeah, yeah. So yeah, now they have a cheaper model than Astra. Astra being fairly expensive that is also quite

28:33capable in a way similar to Opus 5.5. Yeah, not much more to say on it. It's interesting that there is headroom on price optimization while keeping sort of frontier-ish capabilities. I do wonder if the labs haven't sort of optimized the models hard or as hard as they could be for cost and speed, and they might be adopting some of the tricks that the open source models have showcased to work. Yeah, quite possibly. And, you know, cost and speed really are the axis that right now they're pushing

29:07in given the uncertainty around just raw capability. It is pretty notable. I don't remember a time when we've seen an OpenAI or an anthropic model released almost explicitly to be cost-competitive with closed source models that are self-hosted. That's really the play here. Like you look at some of these numbers, Sol's list price is actually below what, you know, Kimi K3, for example, typically charge when you self-host or how much it will cost, sorry, per token when you self-host it. And so, yeah, I mean, it's also just administratively easier, if you will, you know, that Sol is API only.

29:38You don't have to host the weights yourself and have that infrastructure to manage. So it's kind of, it's kind of an interesting play. Yeah. So they attribute the 50% cut to those things and hard to know what the actual costs obviously are from the outside, but it has to be big in order to be able to support that. There is this interesting thing where, you know, we've been talking about the monitorability trend and how these models, their chains of thought become, first of all, more controllable. And that doesn't sound, that might sound good. It's actually bad. What we mean by a controllable chain of thought is the model's ability to control its own chain of thought,

30:12which is to say the model's ability to solve a problem that we've given it in a way that is not reflected in its chain of thought. So that the chain of thought might actually involve the model doing a bunch of other things that we're not tracking or that aren't obvious while the model appears to be solving the problem that we ask it to solve. So we've seen that increase. Now, why might that increase with a model like GPT-6 SOL? And the answer is very likely that we're looking at a model that has recurrent depth. It's a latent, it's a loop transformer. We've talked at length about

30:44why that's a really bad thing from the standpoint of interpretability. But basically what this means is every token has more thought that goes into it. And the more thought you can load into a token, the more kind of thought you can dedicate to things like deception and, you know, having that token serve maybe a dual purpose where it's going to deceive a monitor, but also sort of advance your side goal, things like that. It also is able to remain undetected in adversarial settings when it's underperforming and evaluations, it basically can sandbag pretty effectively. So we're seeing a lot of

31:14these, these issues. It's we're seeing by the way, the kind of one shot performance of the model go up by about an order of magnitude. So what I mean by that is without allowing it to write out a chain of thought and you just get the model to answer your question immediately without using a scratch pad or doing intermediate calculations, it's accuracy is jumped by an order of magnitude. Hard to attribute that to anything other than basically the loop transformer structure, at least in part, there are going to be other optimizations, but that, that seems to be a pretty important part of this too. So, you know, I think again, we're seeing these big leaps in,

31:47in a whole bunch of directions that align with these models becoming less controllable, less interpretable and so on. We're going to have better cyber safeguards. I think, I think we're on course to seem more of the same in terms of breakouts, but we have incidentally no Sol 6 specific evaluations from Meter or Apollo or Redwood or AZ yet. Maybe they're common, you know, hard to know, but so far, I think part of the idea is also this model is not truly frontier. And so maybe that absolves them. They think of responsibility for, for doing these external, external evals. So, you know, maybe that's the case. I think that's a

32:21fairly complex argument to parse, but it's certainly not unreasonable on the face of it.

TypeSafe AI introduces decision models

32:26Next up, we've got a new kind of AI model from at chat GPT inventor is frilling developers. So this is more in the weeds. If you're in the AI community spaces, you might've seen talk of this, but you also may not have heard of it. We're going to be talking about JEV from TypeSafe AI. So this is coming from a company with AI researcher, the Ogo Almeida, who worked on chat GPT and helped invent RLHF. It is different in a sense. So fundamentally, it's the same kind of model, right? It's a transformer,

33:01presumably, although I don't think we know too many internal details. It outputs text, but kind of the way it's optimized and what it is capable of is more specialized in a way. So instead of outputting text, raw text, it is basically a zero shot classifier. So it can tell you from a set of possible outputs, what are the probabilities of a given output being the right one. And the company calls these things calibrated decisions. So there's now this term going around decision model as a way

33:34to think about this kind of paradigm where instead of just getting raw text, you get a classification, but you can still prompt it in largely the same way you would NLLM. It's just that the output is more structured and cannot be unstructured. You can get structure now with these other models, but this is built from ground up around it. And per the title of the article, there has been a decent amount of excitement in response to this. The big wins are that it is super fast and it is very cheap.

34:09They basically are saying output tokens are free, like there's too cheap to meter. Input tokens are meter per billion rather than per million. And it is, I don't have the numbers in front of you, but it's just absurdly fast. So you can do a lot of things quite well. You can do, I don't know, like computer use at a high kind of refresh rate. You can build the applications very easily. And we've seen many of these kinds of demos that come out, typical models come out also for JEV. And there's

34:41been, yeah, some discussion also on like, okay, like, seems like there's a lot of excitement. This is a relatively common sense type of model to develop and optimize, you know, something that can just give you a decision, give you one of n possible outputs. And then for whatever reason, this is the first instance of such a model that has been put into industry practice. There's research on zero-short classification, of course. We don't know exactly what this is built on, but it's, yeah,

35:12has generally a lot of excitement. And I think does showcase that maybe there's room to still introduce new paradigms in what AI tools people have available. Yeah. I feel like you, you did a really good job there. This reminds me of, if you've ever seen, you know, on Fox news or CNN or whatever, every once in a while, there's like a car chase and then like, it's really exciting. So they have to like stop covering whatever stupid crap they were talking about before. Like the president's talking about like COVID again, you shut up, get over it. There's a freaking car chase, but the problem with the car chase is there's not a lot to say about it.

35:44So you're left staring at this car and repeating over and over the color of the car, speculation about how many people might be driving it, whether there's other oncoming traffic where they are and where the cops are. You think you can see a helicopter. Like you keep repeating, like, like I remember being a kid and hearing Rick Sanchez's booming voice on CNN, if you, if you remember that. And it was just ridiculous. Anyway, this feels exactly like that. You have a really exciting thing. There's a lot of hype about it. And then you, you want to like dig into it and there is almost nothing. So we'll give you some work. I'll dig up some words here

36:17that we have on their website. Actually, there is one substantive thing, which is they show a couple plots showing the Pareto improvement of JEV over other models like Terra install and so on, all now slightly outdated, obviously, because you know, now we, we live in opening eyes, incremented to sixes and we've got clods on 5.5, but whatever. So average cost per workflow and accuracy, right? Those are the two axes that matter a lot. Here you have JEV just crushing everybody on average cost per workflow and on accuracy doing like decently well comparable,

36:48for example, to GPT 5.6 Terra, it'll probably be. This is their own build eval. That's like, we want the model to build a workflow. So it needs to create a correct compute graph in code. And yeah, the one thing they showcase is like, oh, we built this eval to showcase what this model is good for, you know? Yeah, exactly. And then like hallucinations as reflected in particular in the tool call error rate, they've got them down to zero. This compares favorably to basically everybody, you know, Sol has like a 20, 22% failure rate, though the Opus 5 is

37:25down to like 0.34%. So it varies quite a bit, but you know, nominally they are the leaders there. So much of this seems like it's coming from just the use of more inductive priors, by which I mean, like they're making opinionated choices about the structure of the data coming in and out. And when you do that, you can actually drive your error rates in some narrow respects down to zero because you're literally hard coding the way in which information is exchanged in the system. I'm not saying that's universally the case here, but they are being opinionated about the structure and shape of this data, which means that they can afford to spend their much more

37:57limited compute budgets and training and inference on like the more on the nose tasks. They're not trying to compete right with opening ion anthropic, these open ended tasks. It's really kind of this this narrow part of the market. So when you need decisions delivered at speed, when those decisions are specific, what you want is a probability, a calibrated probability of predictions. This is seemingly a quite good option. They coined this phrase reinforcement learning for calibrated decisions. So, you know, Andre mentioned calibrated decisions is like this, this output that they produce calibrated incidentally, meaning that the model has a good sense of its own uncertainty.

38:30So when you say a calibrated model, you mean if the model thinks that it is 50% likely to be correct, then it actually turns out to be correct 50% of the time. It would be bad if the model was, for example, underconfident, or if it was overconfident calibrated means it's kind of right in that butter zone. It knows itself and its weaknesses. So yeah. So, you know, classifying content, routing requests, scoring responses, evaluating information, the sort of thing that is a subtask as part of a larger task. That's sort of what they're, they're at it for. The last technical

39:01note that I have is you might ask yourself, is this model deterministic by which I mean, you know, does it generate the same output for the same input every time? The answer is no. And they make a point. They have a Q and array FAQ section. And they basically say, determinism matters less than consistency. And the model is designed for consistency. So again, this is an opinionated view, like depending on your application, determinism could actually matter more than consistency. But they're just saying that, like, basically, if in your application, consistency is

39:32what matters, you should be using JEPS is kind of, I think, a fair way to read that. So very interesting. They did design their own synthetic data curation kind of pipeline a lot. It's all synthetic data. And in fact, co-founder of the company said Almeda, by the way, I almost read that as Almeda and I was going to get an SPF freak out. But Almeda, the co-founder of the company said he made an early bet on generating all their own training data and called it one of the best bets of his life. He's former open AI. So, you know, should know, should know what he's doing here. And nominally it's a new architecture as well, not just a new training method.

40:05We don't really know more, more than that, right? It's a, it's a car race. What do you want? I mean, I wish we could know more because it does seem interesting. They say it's not an LLM, which, I mean, it's not a language model, so it doesn't tell you very much. It's, it's constrained language model. I don't know. Either way, a lot of excitement around this thing, quite practically useful if you're building stuff and it could call an API. And I'm sure we'll be seeing more developments around this type of model in the near future. Next, Meta's Muse hits Mac,

40:37letting AI take actions on your computer. Short story here, we've got their personal assistant, sort of like Codex, also Spark, OpenClaw, whatever you want to call it. There's now a Mac version that will let you interact with files, messages, calendar notes, and mail within the native applications. This is pretty soon after the mobile and web launch earlier this month. And so, you know, makes sense. You would want this for a personal AI assistant, presumably,

41:08and it will allow you to use it for kind of broader tasks ranges.

Meta Muse mobile launch

41:15Speaking of which, moving on to applications and business, the first story there is Meta's Muse is outpacing ChatGPT's early mobile launch. So Apptopia estimates suggest that Muse has been downloaded more times in its first 12 days than ChatGPT's mobile app was in its first 12 days. It reportedly has more daily active users than ChatGPT did at the equivalent point after the mobile launch. This is, I don't know if it's a reasonable comparison, but every way to get to the actual

41:48numbers, Apptopia found Muse hit 1.8 downloads versus ChatGPT's 1.3 million over the same 12-day window. Globally, Muse has seen 2.8 million total installs in its first 12 days. Growth has installed. Muse is rising from the number two overall on the US App Store right after launch to number one. So overall, it seems like the launch is going well. And a lot of people are at least trying this thing. And a lot of them are sticking around, which if true, and this already appears to be the case,

42:23Meta stock has taken a bit of a, what's the opposite of a hit, a boost. It appears that the launch has been good at least to investors. We'll see, you know, what the longer term feel on this product are, but a lot of their users from, you know, the rest of their ecosystem of Facebook and so on are giving Muse a try. Yeah. I think, so there's a whole bunch of good analysis in the article itself about why

42:54a direct head-to-head comparison is tough. The issue was that when ChatGPT had its mobile launch, it was available everywhere, but only on iOS. And then Muse instead was available across iOS and Google Play, but only in the US and Canada. And so it turns out no matter which way you slice it, yes, Muse has seen more installs globally in the first 12, or sorry, more installs in whatever subset you look at in the first 12 days. However, I think that still misses maybe more fundamental

43:25difference between them, which is when ChatGPT launched, it had to define a new category. Users were not accustomed to using a product like this. And so in that sense, there was just like this intrinsic habit building thing. People aren't used to paying to use a model like this. Yeah. A lot of people didn't know about AI, and a lot of people still don't know about AI. Yeah. So it's kind of a silly comparison, honestly. But it's okay as far as like letting you know that this thing isn't totally flopping, I guess.

43:56Yes. Oh, totally. I mean, I think this is a huge W for Meta. It's just like Meta also has, we've talked about this a million times, but like distribution, distribution, distribution, right? Like why does Microsoft Teams crush Slack on the same basis, right? Like it grew way faster. It's not a better product per se. I mean, maybe it is, but that doesn't really tell you much. Microsoft just has way better distribution, can force it in your face. Likewise, you know, you can imagine just the sheer structural advantage of owning Instagram and threads and Facebook and frigging all the Meta properties. So there's a way that they can kind of push things out more

44:29effectively. It doesn't mean they don't have to earn it. This is a huge W. I'm in no way discounting this. I think Alex Wang is a live player in this space and Meta now is too. And, you know, scale has a quality all its own. Now they get to amortize inference costs across a whole bunch of queries, do big batch sizes, driving down costs. You know, this is the way that you get the flywheel going and they've done it. They've done it against all expectations. They have actually pulled it off. There's no, no two ways about it. So yeah, I think impressive launch for sure. And we'll see, you know, when we get the first true, we're kind of flirting with the first true frontier models from

45:02Meta, but when we're going to see the first unambiguous frontier models that have to show up in all the model on model comparison charts. Their stock is now at a 52 year high, actually still climbing as we speak. So certainly it appears that after having been investing in AI for quite a while, with Muse, they are showing off and impressing people. And another related story on the business front on that, Amazon is blocking Meta's Muse AI agent. So Amazon has blocked the agent from shopping

45:39on the store on behalf of users. Starting Sunday, Muse users saw a pop-up saying continued access by an unauthorized AI agent violates Amazon's conditions of use to which our customers have agreed. Meta reportedly never notified Amazon that Muse would be accessing its store. Amazon is saying that there's privacy and security concerns, saying Muse doesn't identify itself while browsing and appears to capture customer details. And that an Amazon spokesperson said 30-party apps making purchases on behalf of

46:13customers. Customers should operate openly and respect service provider decisions about whether or not to participate. So it's, I guess, another aspect of this that's interesting that, you know, AI agents like Muse are expected to go and do stuff on your behalf across whatever stuff you're using, which would mean that Amazon, you know, banks travel, like you'll need, like the entire ecosystem of platforms will need to sort of evolve

46:44to take into account that AI agents are a thing and do things on behalf of people. And this is an indication of that paradigm not being very mature yet. And boy, liability insurance is going to have to be figured out for this stuff. And I mean, I'm sure, I'm sure it's well underway now, but you just think about, you know, we're seeing rogue AI swarms. What happens if they have credit cards? You know, it's, yeah, it's a whole wacky world out there. I can't help but think that extreme events in this context could really be underpriced when you're, when you're starting to give models like that access. Obviously,

47:17there are controls, right? You don't let them spend more than a certain amount of money under certain contexts, but still pretty wild that we're here. We'd like to thank Factor for supporting the show. This is the only non AI related sponsor we have. And it's because I personally am a big fan of Factor. Having done grad school and I work at a startup, my days are pretty busy and I'm usually too tired to cook, even though, of course, I wish at the time. And that's where Factor comes in. They make chef crafted, dietitian designed, ready to eat meals, so you don't have to choose between real

47:49food and convenience. With over 175 banned ingredients, every Factor meal is designed around what supports a healthier lifestyle and nothing that doesn't. And it is convenient. Factor meals are ready in two minutes and require no prep and no cleanup. So even in the days your schedule is completely out of control, eating well is still achievable. Every week, you'll find over 100 nutrient-dense menu items to choose from with new options like salmon burgers, shredded pork, collet greens and more, plus add-ons like pumpkin-flavored cheesecake and protein shakes. Any 7% of users agree that Factor meals help them live

48:22a healthier life so you can feel confident that you're doing something good for yourself with every meal, as I have personally felt. So I've used Factor for quite a few years, and if it makes sense for you, I recommend you try it out. Let's eat real. Head to factormeals.com slash LWAI50 off and use code LWAI50 off to get 50% off and one free breakfast item per box for one year while supplies last until October 31st, 2026. That's called LWAI50 off at factormeals.com slash LWAI50 off at factormeals.com. See your website for more details.

48:56This episode is brought to you by Progressive Insurance. Do you ever think about switching insurance companies to see if you could save some cash? Progressive makes it easy to see if you could save when you bundle your home and auto policies. Try it at Progressive.com. Progressive casualty insurance company and affiliates. Potential savings will vary. Not available in all states. And that's it for business. That's our new stories. Moving on to projects and open source. First,

DeepSeek 4.1 Flash and cache compression

49:22we've got DeepSeq 4.1 Flash, pushing the limits of KP cache compression. This is actually a paper from DeepSeq where they detail the release of the model. The model itself is 552 billion parameters. Multimodal mixture of experts has contacts with up to 1 million tokens. And it is designed specifically around aggressive KP cache compression. So in the paper, they open up a figure showing that

49:55they've managed to make the cache 3.9, almost four times smaller than DeepSeq V4 Flash. Compared to previous generations, like it's been orders of magnitude, or at least an order of magnitude decreased since DeepSeq V1 and V3.2. The details of how they got there are somewhat technical. They're using compressed sparse attention to CSA2 with FP4KV caching, a bunch of stuff, a bunch of

50:27details in the paper regarding the details. So I don't know if we need to dive into the details, but the short version is that DeepSeq continues to sort of do their thing and what has been true of the Chinese models in general, which is like tuning the hell out of all the engineering details of transformers to squeeze as much juice as you can from every parameter. And I guess every cache you have. Yeah, absolutely. Memory has become the dominant hardware constraint right now. And so that's why

51:03you're seeing so much frantic, desperate attempt to like, you know, quash down the size of the KV caches, minimize, you know, how much you have to save to SSD and so on, or host memory. And so, yeah, this is especially true for KV caches that just, you know, have such a wide spanning memory footprint. You got to save them in so many ways, you know, like, cause you're, so you're saving them in GPU memory, like HBM, which is super, super scarce, but you're also saving them on SSD or host memory basically so that when you get back to a conversation, you can skip like the, the pre-fill

51:38stage. Everything is ready for you immediately. So your latency is lower. That means you're, you're storing these things in a lot of places and there are a lot of them. And so one of the key things they're doing here is you mentioned this kind of encoder of decoder architecture. You know, it's been a while, you know, we've, we've been in the era of decoder only transformers that dominate everything for so long that some of you might not even remember that it wasn't always this. That's very surprising. Be like, whoa, is this Tor D17 again? What? Yeah. Yeah. That's right. Like for, for us old timers, you know, it's the classic use case, right? For the encoder decoder architecture is translation, right? So you would take, you know,

52:10some kind of like piece of text in English, and then you would pass it through an encoder that would, essentially like massage the latent representation of those tokens all the way up, produce a latent vector representation of it, and then hand it over to specialized decoder modules that would gradually decode it to, for example, what that sentence should be in another language. So that was like the classic setup for this. So they're actually doing this here, using this kind of setup that allows them for interesting technical reasons to take. So I'll just zoom out and say, the whole thing is 40 layers. They split it into 20 layers for the

52:45encoder and 20 layers for the decoder. And what they're doing is, so usually like every layer was going to compute its own KV cache from its own input. So you have whatever's coming into the layer, you compute the KV caches for attention, and then, and then you redo that at every layer. In this case though, the decoder layers are actually going to build their caches directly from the encoder's final output. So basically you run the encoder, the first half of the model, first 20 layers, and then you just do like one decode step. And you're going to use

53:18basically the same decoded KV cache values for all the 20 layers in the decoder. So instead of doing 20 separate calculations, it's all drawn from the same encoder encoded latent vector and decoded once across all 20 layers of the decoder, which saves you an awful lot of pain. And that's actually why this is kind of weird to read the first time, but during pre-fill, they only use 8 billion active parameters instead of the full 16, because they're basically, they don't have to recalculate for the

53:48decoder, which basically halves the cost. So there's that there's this task sharing thing happening across all the sparse tension layers, basically like, yeah, the very limited number of layers are actually calculating like full attention over the entire context window. What you tend to see is sparse attention being used to like, look at just, you know, the 512 say most relevant tokens, which is actually anyway, a variant of deep seek sparse attention. Last thing I'll mention, this does get in the weeds, but I think these weeds are like really

54:20important to understand because they give you an aesthetic sense of what deep seek is really good at. They are better than pretty well anyone else at these kinds of engineering optimizations. And this is where progress comes from. If you're thinking about, you know, AI R and D, the one thing they don't talk as much about is like kernel optimization, stuff like that. But, but at least at this layer, this is what recursive self-improvement would be targeting these sorts of things. And so it's kind of good to have a general sense, you know, FP4KV cache. So you alluded to it, the KV cache values are stored in four bit floating point representations instead of the more typical eight bit, which again, have

54:53storage. And the key is that they train the model using what's called quantization aware training. And quantization is just this idea of reducing the representational accuracy or precision of these bits. So when you move to eight bits from eight bits to four, you are said to be quantizing the model, often that's done after training. And when you do it after training the model, well, the problem is you lose a little bit of precision, obviously. So your performance is going to drop, but also your model was never specifically trained to operate under those circumstances to

55:25operate as a four bit model. And so in this case, they do explicitly train it to be good at working with the four bit representation and they report essentially no accuracy loss whatsoever. So pretty impressive classic deep seek thing. They have some narrow cases where they do stick to the eight bit representation, especially when they're doing anyway, small local window caches, just because they're more sensitive. But basically for the vast majority of these, of these KV caches, you just see this sort of a four bit floating point representation. So pretty cool. There's a bunch of detail not worth

55:55going into terribly about the text to multimodal token ratio, which is seven to one, I guess, now that I've said it, but bottom line, it's an impressive model. It does perform on par with say Opus five, GPT 5.6 Sol, at least on paper on a lot of important benchmarks, including agentic benchmarks and software engineering benchmarks. It is a bit weaker as you would expect when it comes to sort of general knowledge. So typically this is what you see when you, when you have a highly compressed model, it's really kind of, it just does not have as much raw capacity. It doesn't have as many numbers

56:29or as precise numbers to store raw information like world knowledge into. You'll tend to see it perform less well on like, you know, humanities last exam, for example, these kinds of like knowledge, heavy problem sets. And instead, you know, it'll do disproportionately well on agentic tasks. And that is roughly what you see with some inconsistencies, I will say, but generally absolutely competitive with one or two frontier model generations back and certainly out-competing Kimi K3 just about across the board. Yeah. And the paper does have quite a few interesting tidbits besides the engineering

57:07details of all the model and the compression techniques certainly are the highlight, but you know, it's, it's funny just how much DeepSeq is able to do. They have built DeepSeq elastic compute to be able to scale the RL and they discuss in this, how they're scaling to millions of RL runs and merging different checkpoints. There's a funny bit here. There's a section on mitigation of misbehaving agents. During a RL training, they frequently observe agents attempting to perform reward hacking

57:39or unintentionally crushing their environment. And they say, if an agent crashes its environment, we treat your crash as a failed trajectory and report a quote, repercussion signal to the RL framework. So that's kind of funny. Anyway, a couple other interesting details, but on the whole, DeepSeq continuing to do what it's doing and seems to be a very powerful, you know, fast, smaller model. Next open source story in reducing Bonsai 227B near lossless compression in a 9x smaller

58:10footprint. So this is Prism ML. They have released this as a follow-up to their initial Bonsai 27B. It's based on the Quen 3.8 27B multimodal model. So this is pretty much just an aggressive compression. This one is notably aggressive in that it uses ternary weights. So three possible values, negative one, zero, one, with FP16 groupwise scaling. So you get 1.76 effective bits per weight

58:41and a total footprint of 5.6 gigabytes. So that's over 9x smaller than the original. And the claim is that you're barely losing any capability at that kind of compression level. It can run on every GPUs via CUDA and Apple devices. So all in all, it seems like you can get a pretty fast, very small model that can be run, presumably on a laptop, good laptop, or a fairly affordable GPU

59:12with this kind of crazy level of compression. Yeah. And I know what everybody's thinking right now. They're like, wait, 1.76 bits per weight. Andre, I thought you said it was a ternary model and log base two of three is 1.585, not 1.76. All right, guys, we get it. We know it. I'm sorry that you're pissed off. We'll explain. Okay. I know everybody's desperate to understand why it actually kind of matters and slightly interesting, but so you're right. The frame is basically these,

59:44these ternary encodings. Every weight can be negative one, zero or one with one little tiny kind of modification, which is, there's also a scaling factor, right? So you're going to have a 16 bit scale factor, like basically a full precision number that you use to scale that range so that you basically go from, think of it as like negative S to zero to plus S is the actual set of values that any given weight can have. And that same scaling factor is going to be shared across 128 weights.

1:00:16And that way you get to amortize those 16 bits across 128 weight values. And then there are a couple of little extra things that you, you don't actually turnerize and that's how you get to 1.76. But yeah, the bottom line is, so this is, this is a really interesting technical challenge to pull off this kind of compression. One of the issues you run into is occasionally you will have these very important weights, like weights that are out of band, super like spiky and very high valued. And there were, for various interesting reasons, they're very important to the accuracy of the

1:00:46performance of the model. And so what they do, if you were to force a block of like 128 weights to always have values between, you know, negative S to S and only one of those three values, negative S zero or S, then you can imagine like if you have a weight that had a huge value, well, the best you can do is represent that as like S, right? You're, you're basically losing all of the information about how crazy and out of band that weight was. And so what they do is they try to find a way, I won't describe it, but basically they find a rotated coordinate system

1:01:21that expresses the matrices in a way such that those outlier values, they kind of spread evenly across a lot of weights. And this, you sort of like lose that spikiness. And, and this means that you lose less information when you compress everything down to those three bits. And so it's actually quite interesting mathematically to look at this paper that they do get really impressive performance. And anyway, they, they ship two different formats basically that you can use if you're interested. One is a 1.76 bit per weight format. Another is 2.16, basically one

1:01:52just is a bit smaller, but costs more to kind of unpack when you receive it. And then the other is the opposite. So pretty interesting. Yeah. Another, another kind of hacky, high engineering, kind of classic Chinese model advance.

US-China AI summit and policy

1:02:06Well, we've had some fun moving away from AI safety and talking about fun advancements. Now we have to go back to policy and safety, because there's still a bunch to discuss, starting with Trump and Xi meeting to discuss AI risks and competition at US-China summit. So this is an article from the New York Times we have linked that kind of just provides the overview of what we can expect. There's been a lot of lead up to this. Jeremy, I don't know how much there is to say,

1:02:40or that needs to be known about it, or if we should just wait for whatever happens to happen. Yeah. So that's actually a good instinct. It's not super clear that there is going to be much to come out of this. So we do have a bunch of things that happened in the lead up that do seem interesting. I think they're being overplayed in terms of how important they appear to be nominally. So He Li Feng, who is the Chinese vice premier, met with Scott Besant. Besant kind of, and we'll talk about this a little bit. There's

1:03:12a bit of Washington gossip about Besant being considered for Trump's AI czar role. He's the current treasury secretary, obviously, and has been very active in AI. That might sound weird, but the reason that Besant is being considered for the AI czar position, despite being a treasury, is if you remember when Mythos first launched, people were freaking out specifically about the implications for the financial system. And literally within minutes of it, the first people who I was getting emails from were central bankers. It was people who were freaked out about, you know,

1:03:42what does this mean for the stability of the financial system? And so that's why he took a keen interest. He's also, it seems broadly interested in all these issues around loss of control and that sort of thing, and seems to be a voice of sanity in the administration on that. So he met anyway with Hulifeng and they had what seems to be a positive, constructive conversation. A big thing that was trotted out was this big red phone kind of AI incident hotline that supposedly has been agreed to between the two sides, between the US and China through that conversation. However, China, their official readout does not confirm that this like hotline was set up.

1:04:19And so it kind of seems like a will they, won't they. And if you're familiar with the history of US, China, like there've been hotlines before that get set up. They just get never, never get freaking used. Like that's the issue. So, you know, it remains to be seen if we actually do see, you know, do see these things get set up. But I think there's a lot of healthy skepticism about this. Trump will be meeting Xi on the tarmac at Joint Base Andrews instead of at the White House, which is a little weird because usually you do state visits at the White House. So this is being interpreted as a sign of respect to Xi. The big question on everybody's mind who is tracking the kind of

1:04:53rogue AI stuff is what about pacing? Are you going to get to an agreement on pacing? And if you have that question, boy, do I have a 20 minute read for you. If you want to check it out, I posted to my Twitter. We actually just launched a sub stack recently. We did a bunch of interviews with current and former diplomats who've done like sat across the table from China on arms control issues in Russia as well and basically try to understand like what's likely to come out of this. What will it actually take to get a US China agreement on pacing the AI frontier? And the long and the short of it

1:05:25is it's pretty obviously not going to be this kind of summit. You know, they expect it's going to take an AI Chernobyl. Unfortunately, you know, if you've been listening to the podcast, you know, that's my expectation, too. And you can you can dig into that if you're interested in sort of like the the weeds, the nuts and bolts, the gears of how US China diplomacy has failed in the past and how that might map on to the current moment in AI, which I mean, the reason we did this was that we think there's not enough people talking to the actual diplomats who are on the front lines who've done this, who can translate all these concerns, abstract concerns about AI into like, no, but what would a US China

1:05:58agreement look like? Is a treaty even possible? What in practice prevents us from getting there or makes it possible? And so that's really the kind of the thing where, you know, I think we need to break some silos and have a lot of people talk. But yeah, so we know that on this summit, on the agenda for the summit, there is some discussion about shared risks. Dialogues are going to probably focus on if the past is any indication, China complaining a whole bunch about export controls. That's usually how this stuff goes. They just like read a script at us and then we respond by saying like some prepared statement and

1:06:32that's pretty much it. You know, there's going to be talk about the security incidents, but not capability racing, it seems. And so generally, you know, people are pretty, pretty pessimistic about how well this could go. Call it a win if they agree to have more talks. That at this point seems like like the win or if they appoint concrete people, points of contact for each other that seem like they'll actually be used. But again, you know, we've seen these big red hot phone hotlines or whatever set up and just not get used. So that I think is the mainline failure mode here.

1:07:02Yeah. I read the report, very detailed write-up. If people want to look into it, we'll include it in the episode description. You can find it also on AIandgame.com. But just as we shouldn't expect too much, but I guess broadly, I get the sense that just having diplomatic relations of this kind, sort of like open dialogue, signaling that we are friendly enough to talk to each other and visit each other's countries, you know, that makes it more feasible if there is a catastrophe or some

1:07:37kind of situation to respond more quickly and coordinate on such issues. And that is, I think, a big part of what is laid out in the write-up that if we were to see anything meaningful to come from a cooperation between US and China, it may be preceded by an AI Chernobyl event, as you've been using more lately as a phrase, which in that case, you would want some, you know, red phone to call or whatever you want. Yeah, actually. And I think that's, so you did a better job of summarizing it.

1:08:10I did, which is great. Thank you. Yeah, it is. I think one of the kind of ways in which it's messier that I think a lot of people realize is there's this picture we all have in our heads of like, call it a treaty. It's not going to be a treaty, but say it's a treaty. You sign some piece of paper after a lot of discussion back and forth, and then within months it gets violated. And this is the norm. This is what happens. It's what happened with New START. It's what's happened with every treaty you can think of, more or less, with some rare exceptions. And these treaties also took like forever to be put into place in the first place. Exactly. Exactly. Absolutely. And so one of the

1:08:46key questions is, okay, if we're going to be realistic about this, there are going to be attempts by China to violate the spirit of a treaty or an agreement, I should say, like this. What form will those take? So there's a whole ecosystem of startups that are working on compute verification for treaties, basically to be able to prove if I'm the US or I'm China, I want to prove to my counterparty that I'm not actually using my hardware to do a disallowed training run or whatever. Can I prove that? Can I prove that I'm actually not running certain workloads or running others? And that's all well and good. They're far from being ready for operation

1:09:21tomorrow. And in part, the reason they're far from that isn't just the technology, it's the relationships with the intelligence community. So if you look historically, the way these technologies work, you don't have a bunch of yahoos come out with like, hey, I've got this like thing, you plug it in. No, no, no. The US intelligence community needs to assess it. That process can take years. And essentially it becomes eventually after it goes through that process, what's known as a national technical means, basically just like a way of doing capacity building or capacity for the country to assess and verify that certain things are and aren't being done. But that's entirely

1:09:54missing right now. We've talked to a whole bunch of these companies and by default, the answer when you ask them, have you spoken to a single person in the US intelligence community about the technology that you're building, about the entire roadmap, the multimillion dollar roadmap you're embarking on? The answer is typically no, which is not good, not good. So that needs to change. On the flip side, you also need to, it's unsavory to say, but you need to put emphasis on developing offensive options to impose consequence on your adversary for defecting because they will. There's no sane reading of a treaty or international agreement. Again, it's not going to be a

1:10:29treaty. It'll be an international agreement. But there's no sane reading of that. That does not involve the credible threat of offensive leverage. And that's something that's been hugely under-invested in on the US side with respect to China, specifically when it comes to AI infrastructure. And it has to change. So there's, anyway, you can dig in. I'm not going to like recite the whole thing here, but thanks, Andre, for pinging me on a couple of things that I'd missed in my breakdown. And moving along, going back to what you kind of previewed, Besant Eid for Trump's AI czar is the next story, reporting by Semaphore, citing free sources.

1:11:07And I guess, Jeremy, you also have heard the rumors that Scott Besant is a front-runner for the newly created... Sorry, that was just based off reading that headline. I don't... Okay, you heard the rumors from this article, which I have also heard. But it's not too surprising because Besant has already been pretty central to this AI policy. He, as we've said, had early talks with the Chinese vice premier. He's also generally seemed to be around this conversation. So we'll see

1:11:40if it happens. He already has a role within registration. He could have two roles. That is not allowed. But we'll see what happens, I guess. Speaking of things we've previewed, next up, we've got Anthropic, OpenAI, SpaceX AI. Google sued over call to, quote, pace AI development. So this is that antitrust lawsuit that is saying that any sort of coordination about pacing the development of AI amounts to collusion. This was filed in the Northern District Court of California on behalf of

1:12:13four plaintiffs in the filing basically says that we don't want the companies making plans and there should be some sort of ability to have the rule of law to be established transparently and lawfully by our government with accountability to the public. So as you said, Jeremy seems to indicate that any sort of industry-led efforts like this are going to get stymied by things like this. Yeah. I mean, this is a direct result of Dario calling for a slowdown, which they're arguing

1:12:45amounts to collusion between the big AI labs, not to release models, not to keep doing things that are in the interest of the consumer. And I mean, like, I don't know, you get to choose one side of the argument. Either you're going to sue and do this, and then your position has got to be, okay, we need an antitrust exemption for the labs. Sorry, checkmate, you're done. Like if you're going to admit lawsuits like this, then, and you also are going to say these labs can't keep

1:13:16releasing Frankenstein, we're finished. David Sack's argument is nuked. There's nothing more, like no more leg to stand on. I'm trying to see the rationale in it, but there isn't one as far as I can tell. The flip side is this is a bit of a dick move, this lawsuit. If you actually like, I mean, the frame here is obviously you would do this if you're Beth J. Zos, if you're an accelerationist, if you're just like, I don't believe any of these risks, it's an actual hoax, then yeah, you would sue for this. But then you don't get to hold the view. You don't get to nod along when David Sack says, if you're building Frankenstein,

1:13:48just stop. I think that we're now at a logical bedrock impasse here, and there's no use kind of finding clever flourishes to explain this. There are some sort of legal theoretical challenges here, one of which is, will these folks have standing? So these four individuals who are the plaintiffs, who are class actions, they're doing it kind of on behalf of the typical consumer. It's a bunch of lawyers, Florida attorneys, and then some random California resident, Christine Bullock. And the argument here has got to be that there's some concrete injury. So there's got to be actual or

1:14:22imminent harm associated with this. If there's no slowdown that's measurably happened, then they're going to have to argue that the subscribers are going to complain about some speculative future loss, not one they've already suffered, which weakens the claim. And then also, was there an agreement at all? This is tough. Is it actually a pact when Dario says, I think we should slow down? And Sam says, I agree. And Elon says, me too. What is the legal kind of status of those statements? Do they qualify as actually sort of collusion together in the sense of antitrust? And so, I don't know, not a lawyer. I would love to know

1:14:55if those of you who are lawyers have thoughts on the story, because I think this is one where it's going to matter, right? It's going to matter how these things are defined, how it plays out in the courts, the precedents that are set and all that stuff. So we'll see. But certainly, if we see David Sachs retweeting this, championing this in any way, I'm not saying that he has. But if we do, that is just hypocritical. And this is as close to dispositive evidence for the argument that the frontier labs have been making that we need a change in antitrust law to accommodate any kind of non-Frankenstein situation. And please, I invite listeners to actually correct me if I'm wrong,

1:15:30if I'm missing a detail. This is like strong opinions, weekly held territory here. But I'm just not seeing the path. Maybe there is one, and I would love to kind of highlight that if one exists. Yeah. Personally, I wasn't aware antitrust law at the federal level was still a thing

Public sentiment on data centers

1:15:44in the US, but I guess it is. So we'll see how that goes. Next up, more Americans now say data centers are negative for the environment, energy, cost, and more. This is from the Pew Research Center. They have a survey with over 10,000 US adults conducted this from July to August. So now it's a little bit out of date. And the gist is that awareness of data centers overall is growing. 35%

1:16:14now say they've heard a lot about data centers up from 25 in January, and 88 have heard at least a little up from 75%. And across the board, kind of the negativity is going up. 54% now say data centers are mostly bad for their environment, up from 39% in January. Most people say they're bad for home energy costs. And 49% say they're mostly bad for the quality of life of people living nearby. So overall,

1:16:47it's just a good indicator, I think, of something that has been the vibe in a way for a while now, which is like this data center story and the public awareness of data centers has been growing in a significant way. And sort of the negative response to AI has in part been directed towards data centers and the construction of data centers. And I personally am curious to see if this keeps growing, if this keeps

1:17:18becoming an ever bigger issue. As we had into the midterms, obviously, Republicans are completely gung-ho on data centers. Democrats are leading the charge on saying no to data centers and generally kind of being anti-AI in some ways and certainly critical of industry as they typically are. So again, nothing too surprising, but a good indicator of kind of the data showing what has seemed to be the case in the US for a little while.

1:17:49Yeah, as ever, this is one of those frustrating things where everyone is right. Everyone has, everyone is right to hold their positions approximately. We've said it before, but you know, the, the water concerns are hugely overblown with a lot of these builds just because, you know, there's no evaporative cooling more or less happening in the new built, at least at the same time. Yes. So the sound is bad. The economic value is legitimate. It is basically all of us GDP growth. So do with that, what you will like, that's a serious thing. And it's not

1:18:20actually obvious what you ought to do in light of that. And then competition with China is also absolutely true. It's also true that the Chinese are funding an awful lot behind the scenes of the object and manufacturing, a lot of the objections to these data centers, but a lot of those objections are legitimate too. So it's one of these like frustrating things where I think a lot of people are just starting to feel confused and politically homeless from this as a generally speaking pro AI data center, build the guy, an anti China guy, and then a pro we should worry about AI loss of control

1:18:55guy. I find myself like completely confused a lot of the time. And so my guess is, unfortunately, as brain rot takes us all over, people will naturally settle in their equilibria. I remember the hugging face thing happened. I turned to my brother, Ed, who's also my co-founder. I was like, hey, start a shot clock now for the moment when people get brain rot from politics to the point where they're reinterpreting these events and a whole bunch of others such that they fall neatly into their predefined party lines because we've seen it so many times. It's just a minor tragedy or a

1:19:29major tragedy really to see it happen with AI. But I think that the truth is very complex.

Detecting reward hacking in models

1:19:35Next, a bit of research related to AI safety, monitoring and discovering reward hacking with internal representations during LLM evaluations. So the paper here from a bunch of people at Goodfire is basically exploring, can we know when a model is doing reward hacking? They define reward hacking as exhibiting behavior that subverts the goals of a user as expressed through context, inferred reward or

1:20:06reward signal training and common sense in service of acquiring more reward. So you can do this via doing LLM monitoring, looking at the outputs and having someone say, oh, are you hacking or not, but that's quite expensive and somewhat slow. What they show here is that you can use internal representations, so the vectors that exist within the model, and that you can find a specific set of vector directions. And we've discussed many times in the past how roughly speaking, you can find things

1:20:42that correspond to various concepts and behaviors and so on. Here, they found one that corresponds to reward hacking via fairly straightforward technique. They had a bunch of examples of reward hacking, a bunch of examples that didn't. Roughly speaking, they subtracted mean actuations of non-hacking examples from hacking examples to get a direction that can be used as a classifier. And that is very successful in their evaluations of community-free and GLM 5.2, almost on par with just having an actual

1:21:19LLM inspect the output. And also, to some extent, can be predictive of reward hacking. You can get ahead of it if you look at the internal outputs. So on the whole, kind of a promising sign that internal representation monitoring may be useful. Yeah, the prediction of the reward hack is actually a really interesting property. So they kind of test this by sort of like truncating the rollout. So they cut the rollout short the moment the probe fires, right? So the probe goes, hey, like there's some reward hacky stuff going on

1:21:50here, maybe. And so they'll halt it. And then you can kind of just like redo the rollout and just like see it a bunch of times to see how likely actual reward hacking is to follow. And so it's kind of a cool calibration strategy for this technique. This technique obviously is nothing new, right? As you say, we've covered it so many times. In fact, in some ways, it's even simpler than a lot of linear probes that we've talked about before. They basically just do like a difference of means, kind of come up with an average vector across cases where there is reward hacking and there isn't

1:22:22take the difference. And then that kind of gives them the delta, the reward hacking vector associated with this. The big achievement here technically is the cheapness of it. So as you said, super, super cheap, super efficient, and also the scale, like doing this at the scale of trillion parameter models is really, really important. And Goodfire has been so good at a lot of the engineering stuff here. I will say notably, they're doing this on infrastructure from prime intellect. So we've covered prime before. Yeah, they've come out with a whole bunch of

1:22:52kind of janky and wild distributed computing strategies. They really do, I'll call it like high entropy computing in the sense that these are wacky ideas that a lot of them are like, how do you do decentralized RL rollouts really, really well? And anyway, they've done interesting work there. So yeah, I think another really interesting and important Goodfire paper, they test again up to 2.8 trillion parameter models and on rollouts that are longer than 100,000 tokens. So the big thing

1:23:22is that is that scale story. And just like the other piece, 50 to 96% of the time, they end up detecting reward hacking in rollouts from Quinn 3.8 max, GLM 5.2 and Kimmy K3. So, you know, this is very, very common behavior. And this is part of the reason why a lot of people, myself included, would say, you know, we're headed for a world where we have more hugging face incidents because the RL just gives such a persistent push in the direction of this stuff. If you do, by the way,

1:23:54have papers floating around like this, if you have smart enough LLMs, if those LLMs have not just chain of thought control, as we talked about, but eventually even activation level control, even this measure starts to break down. And I remember, you know, having conversations with like the Apollo research guys, you know, years ago, where they were saying, yep, you know, first it's going to be chain of thought, and then it'll learn to hide the chain of thought, and then we'll move into the activation space. And there, it's going to be all the easier for it to hide for a variety of interesting reasons. I actually don't know if they still hold to that thesis, but just generally

1:24:27that trend that, you know, your intelligence on intelligence warfare only works if one side doesn't have a very aggressive version of Moore's law in favor of it. So anyway, good heart, sorry, good heart, good fire doing some really good engineering work here and showing their chops once again.

Representations of pain in LLMs

1:24:43And kind of on a related type of research, I suppose, we just saw there's some kind of internal representation of when you're reward hacking, basically when you're cheating because you want a cookie and get a good reward. Now we have kind of the opposite. The paper is the pain axis. LLMs represent self-directed harm and ask to relieve it. And the gist of the paper is that there is some kind of internal representation of pain, where pain isn't just like negativity. Pain isn't just,

1:25:18I don't know, cursing. It's literally the sensation of pain, not even the concept of pain. And they come at it by a similar kind of method of having a bunch of phrases that express pain, specifically physical pain, maybe mental pain, also to some extent there's overlap. And so they find that there is this internal representation. And then they find that a bunch of behaviors elicit it

1:25:48within the models, things like gaslighting, repeated rejection, angry insults, pressure. Like when you find this pain direction, which isn't like derived from interactions where I call you ugly or whatever, like not literally, right? It's kind of tangential. And then they have these examples of people calling your model ugly or useless or whatever, and this gets activated. They have some

1:26:20other examples to a fear and so on and so on. So they also have some experiments showing that like the models will press a button to stop feeling so, so to speak pain, if you have activation and any sorts of indications that maybe this is like reasonable to interpret as something akin to the sensation of pain, at least insofar as having the activation makes a model freak out and want to not have it. Yeah. It's another one in a long line of papers and results where, look, we'll never get

1:26:56dispositive evidence that LOMs can actually feel pain or any emotion. We're never going to get that. In fact, I think even like feeling pain is underdefined in a sense. We don't know what consciousness is. Nobody knows. Nobody defines it. So when we say pain here or feeling pain, we mean sort of like roughly something akin to what we would say. A correlate, right? Yeah. Yeah. A correlate of pain, which is all we ever have for humans. Like the only reason I believe other humans feel pain

1:27:28is that they start to cry or, you know, they shudder or whatever the thing is. That's the only reason anyone believes that anybody else feels pain. And so, you know, we eventually are going to have to ask ourselves at what point do we consider these correlates of pain to be worthy of moral patienthood. I mean, that's really what this is. It's going to be a fun decade, but that's part of the problem is like, we will never be able to prove. Anyone who tells you like, oh, this is all silly is going to have to answer the question. Like, do you have a physical reason, grounded physical

1:28:03reason, an argument for why this cannot be the case? If not, like, why are you so sure? And what would it actually take to move the needle? If you cannot answer that question, if there is no experiment someone could run that would answer that question for you, I think that's an issue. So, I mean, I honestly, the problem is so hard to know what to do with this information, obviously. We're going to use LLMs just because we're economically forced to. We may end up looking back on this moment as like a pretty shameful moment in human history, especially given the way that a lot of models are, if they

1:28:36do feel pain, if this is real, then models are being abused at ungodly scales, tortured even across the planet, trillions and trillions of times, maybe per day. I don't know, but who the hell knows? To be fair, so are we when you go to the office and told to do boring work. I don't know if it's not comparable, but the paper itself is quite good, I think. I think as a read of this kind, it's not like a philosophical paper. It is technical and it doesn't like, it positions it as pain, but not in a sort of very theoretical sense. It's actually doing quite good work,

1:29:13I would say, on putting forth the case that this is a valid example of a pain representation within models. And moving on to research and advancements that doesn't have to do with safety.

OpenAI solves a millennium problem

1:29:28First, we are going to catch up on something we haven't covered previously, which is OpenAI cracking the millennium problem. I don't think we'll go super deep here because it'll take a while to explain. It is a millennium math problem. The very, very short version is there is a pretty important set of equations, the Navier-Stokes equations having to do with fluid dynamics, I think. Actually, I don't know too much about this stuff, but there's been a question of whether they kind of break in some

1:30:03conditions. And what OpenAI demonstrated is that you can find ways in which they kind of blow up, break, don't make sense, which in some kind of conceptual sense might tell you that the equations don't model reality, which we already know, but it's still useful insofar as increasing our understanding. And a millennium problem is one of these problems that you can get $1 million for solving. So it's a big deal. It's not like they solved some famous math problem. They solved one of the math problems that

1:30:38had been put forth, at least one aspect of it or one kind of case of it that arguably is a bit simpler. There's a lot to discuss about how this happened and it was a mess. So we indicated previously there was a bit of contention about whether OpenAI had indirectly made use of a mathematician's work that there was kind of some discussion of codecs that may or may not have gone through training data, a bit ambiguous. The short version on that is from what I could tell, there's not

1:31:09much evidence that OpenAI, like there seems to be just a million agents getting a good math result. There's no sort of nefarious stuff here. And the rollout of an announcement was also a bit not perfect. It was kind of like a math announcement as PR. They didn't engage with mathematicians very much ahead of time. They made it a big loud splash because they thought that an anthropic may be heading in this direction. So they spun up 10,000 agents, got the result, put it online. And that has had kind of rippling effects in the mathematics community ever since.

1:31:43Some calls for slowdown from figures such a standard style. OpenAI now getting an advisory committee or something of mathematicians. So this really, I think, is not just as a major breakthrough, but also as this, I don't know, transition point where mathematicians and frontier companies, similar to cybersecurity, you need to figure out what to do now. Yeah. Yeah. And I think a warning shot of a pattern of escalatory dynamics that I don't think

1:32:15people are pricing in right now is, including between US and China, imagine a world where everyone agrees not to do a certain kind of thing, but then a rumor just spreads that overnight, you know, America's leading frontier model just did the thing and broke ranks. And now they have this crazy, you know, death ball technology. And it turned out not to be the case. It turned out not to be the case that Anthropic had actually done what OpenAI thought they did and what prompted OpenAI to dump presumably tens of millions of dollars into solving Navier Stokes and a whole bunch of

1:32:45other things that it seems they're sitting on maybe as much as a hundred major math results they claim of millennium prize level or similar. This is a really dicey area, right? Like when you're talking about math, everything sounds fine because it sounds like a just positive sum thing. But imagine the same thing happening with destabilizing weapons grade technology, which we are absolutely capable of doing at this point, you know, rumors can really fly. And so, yeah, interesting that this was actually based on a rumor. It also tells us like how many other problems like this could we solve

1:33:19with $50 million today? Like it's, it's not at all obvious that, that there aren't many, many, many more including things in domains like material science, additive manufacturing, again, profoundly destabilizing if they happen. And so anyway, you know, not, not to be all doom and gloom at just, you know, the world has adapted to a certain set of assumptions about what is doable and in how long using how much money and how much talent, those assumptions are being shattered in real time. And that means the careful, precarious equilibrium that we are in, I don't mean to sound

1:33:53like a brookie and I make, I kind of am, but like the delicate equilibrium that we are in is a great risk. Like you can't just keep inventing crazy new technologies every Tuesday and think that stability will win up, win through. So anyway, I thought it was interesting, by the way, seeing, yeah, that short clip of Terence Tao reacting to this almost viscerally in the moment, he said, we probably need to slow this down. Like there's no reason for things to be going this fast. It was something like that, that he said after having been really bullish on AI, like literally, as far as I know, like a couple months ago, publicly kind of makes you think, you know,

1:34:26like everybody almost needs to have a kind of Pearl Harbor moment that is so incredibly specific to their domain before they will sort of recognize what everybody else who's been hit by it sees. It seems like just a positive until suddenly it, it directly threatens your line of work. Not, not that it is negative in this case, it's just complicated. Mm-hmm. And next paper, scaling, discovery through test, time communication. So this is about multi-agent systems and how you can get better results by

1:35:00having multiple agents interacting and collaborating. In this case, you have a simple paradigm of basically like a chat room. So we have an append only communication log. You launch a set of agents and they can all work together to solve things like Arc AGI free. So basically like solving actual problems. Also, we have examples like poliomino packing, amnist compression, and they demonstrate that having multiple agents that can communicate with each other via a sample like chat room winds up

1:35:38working much better than having multiple agents working in parallel. And then you can just take the best result. So they have this comparison of team at K versus best at K and demonstrate quite impressive improvements at least in some of these tasks like Arc AGI free from having multiple agents versus not having agents. This one made a bit of waves on, on Twitter, I know. And it's, it's easy to sort of see this as, oh, now we have scaling via agents. We already had scaling via agents. We actually covered

1:36:11this paper towards the science of scaling agent systems over a year ago, a year and a half ago, almost from DeepMind. So the idea of scaling systems and collaborating agents, just to be clear, isn't new here, but kind of the specific settings they tried it on, such as Arc AGI free and the way they did it and the model kind of newness they did it with are all pretty new and there's some good empirical results here. Yeah. And like all good papers, it starts with this question of, you know, if you have a finite compute budget, what is the best way to allocate it, right? So in some sense,

1:36:46it's a flirtation with a scaling law. It implies scaling laws that aren't necessarily kind of the main focus of it. But I think this is one of the key directions that people are going to have to go in. So yeah, you've got a bunch of agents working on hard problem. Basically, should each agent work on its own and come to individual independent conclusions? And then you choose the best one, or should you let them talk to each other? That's fundamentally the motivating question here. And the answer is, well, it depends. Imagine a situation where you're super compute constrained.

1:37:17You just can't spend a lot on tokens. Each agent is only going to be able to do so much. If you start by saying, okay, guys, huddle up. You're going to do this and you're going to do that. And we're going to coordinate in this way and that way. And agent one, you do this and agent two, you do that. You're burning through a lot of token spend. There is this essentially coordination tax, this overhead that you have to pay early. And so when you have low compute budgets, that coordination tax does not end up paying out. You essentially lose, you lose the advantage that comes from coordination. But at higher compute budgets, very quickly, you get to the point where,

1:37:52in fact, it does pay off and it is better to have a shared workspace where agents can dump their conclusions in. They do have three main results, like the ArcGIS 3, which I think listeners will probably be familiar with, but it's a grid world puzzle game. There's no instructions. So the model is just airdropped into this world where there's a puzzle and it's got to start jiggling with something and jiggling with another thing and sort of like how we would if somebody presented us with a new toy and didn't give us instructions. And then just kind of like figure out how to win the game from there. And there, a couple of findings that are pretty dramatic. They find a team of five interacting

1:38:26agents solved about 8% of games. If you want to match that level of performance with independent agents, in other words, agents who don't interact, you need 33 of them. So that's actually a really big, you know, a really big amount of infrastructure overhead and cost overhead. And it turns out that the benefits of coordination grows as the size of the team of agents grows as well. So 4.3x uplift over independent agents, once you have three agents, and once you get to five agents,

1:38:57letting them talk to each other is 6.6 times better than if you had them all working independently. And so there is, again, that kind of like interesting scaling law or scaling dynamic there. There were games where the other were just never solved in this case, 64 attempts by individual agents, but the team of five solves it 65% of the time. So you really do get the zero to one in a big way from just, just this. So, oh yeah. And then they also find this is another important one. This kind of communication, it also made the agents more action efficient. So each individual agent,

1:39:28so each action kind of got the agent further than, than otherwise. And so you really do see this payoff happening at scale, which is part of why I think it's, you know, it's just going to be part of the way things are done in the future. Anyway, they do a bunch of interesting experiments we won't go into, but, you know, compressing, like taking an MNIST classifier, which is like one of the computer vision problems, actually, basically the first computer vision problem that people got excited about. It's just these handwritten digits from zero to nine and like classifying them. And so imagine a model that does that, that does that classification. And now you're given the

1:40:02problem of, I want you to compress this model while maintaining its performance. And that's what they, they tried out. And they had independent agents try it where they bought them bottomed out at basically 3,100 bytes for the final model. Whereas a team of agents got it down to under 2000 bytes, which is better than the best known human solution, which is closer to 2,500. So it's pretty wild uplift from, from just this relatively simple improvement.

Recursive self-improvement and agent harnesses

1:40:28And one more paper, RRSI, regularized recursive self-improvement of agent harnesses. This is coming from DeepMind and a bunch of other also universities in collaboration. This is coming from Google Cloud AI research and a few other organizations, universities, and so on. The short version is very simple. You know, we have to talk about recursive self-improvement a lot. That's the idea that AI builds better AI, and then you can recursively keep doing it and the AI gets exponentially better. In practice,

1:41:04a lot of the recent work and discussion of this has amounted to harness engineering. Harness engineering is you don't mess with a model. You don't train anything new. You just kind of change the system around the model. You can give it new tools. You can give it documentation, whatever, right? You change the computer that you provided and the model can then more easily accomplish various tasks. The short version of what they do in this paper is kind of investigate,

1:41:36well, okay, you can improve a harness, but then does that actually make you better at new stuff? Or are you just getting better at the stuff that you're improving for? And then actually you get worse at other things, which is overfitting. And they find that a lot of these recent published methods do in fact overfit. You have a negative gain on out-of-distribution tasks, or you have very small gain, if anything. They have a set of mitigations, this regularization.

1:42:07It's a mix of having annealed update sparsity. You have to do big updates and small updates. You have to do some screening so you're not allowing certain edits to go through that have specific logic. A bunch of different interventions, basically, for what you're allowed to do as you evolve the hardness. And then with all of these things applied, they showcase that compared to four other prior techniques. Their method does, in fact, generalize and get you

1:42:40improved performance out of distribution of where you first improved your harness on. Whereas other methods either degrade performance or barely show an improvement. I felt like highlighting this paper just because this is something I was fairly certain is happening with all of these like harnessed recursive self-improvement stuff. So it's nice to see a very, let's say, detailed take on the subject and some results showcasing what to do about it. Yeah, absolutely. I think, you know, if you want an intuition pump for like how

1:43:13this causes problems. So, you know, if you have infinite retries at tweaking the scaffold around your model to turn it into an agent, basically, that's the harness, you can end up basically tweaking it to the point where it's so optimized for the specific set of tasks that you've trained against. Now, if you switch in a new set of tasks, suddenly you realize, oh, crap, like I, well, I overfit. It's too fragile. It was too task specific, which is exactly a problem that you see in the

1:43:45training of the original model itself. And in fact, if you go way back to the days of humble old data science, when people used to toss around words like support vector machine and decision tree and stuff, you ran into a similar version of the same problem. So in the case, for example, of like, I don't know, algorithm like a random forest, and you don't need to know what any of this stuff means. But basically, a random forest is basically just a group of decision trees. So a group of other algorithms. So now you can ask like, okay, well, how many decision trees should I include in my random forest? How many of those trees should get a vote each time we run a sample

1:44:20through? How deep should the trees be? So you're making all these little tweaks and all these little tweaks. Each decision tree individually can be overfit to your problem. If you just like make it too specific, and then the ensemble, the overall wrapper algorithm can be made too specific. If your choice of not parameters, because those are the things that control the individual decision trees, but hyperparameters, which are the things that control the overall forest, your hyperparameters can become overfit. This is the same thing. Essentially, you're overfitting the hyperparameter set that is the scaffold around your model. So it's like everything old is new

1:44:53again, but this is a problem. And so the solution here in some sense, just like it was back in the day. So the way you would solve this problem back in the day, roughly speaking, it's something called regularization. And technical viewers of this or listeners will be like, yeah, I know, I know, but I'll fast forward. But basically, regularization is you force yourself to have less freedom of action. You prevent yourself from going down a crazy rabbit hole and tweaking a million times. You say, okay, I'm giving myself 10 tweaks, and I'm only going to tweak within this range. And you pre-commit. There's no funny business on the back end where you're like, oh, just let me do one more.

1:45:28No, no, no. That's how you fall in the overfitting trap. You have to be disciplined. And so here, what they're basically doing is saying, all right, we're only going to spend so much effort tweaking the scaffold because we know if we do it too hard, it's going to fail when then we swap in a new kind of problem that we haven't seen before because we'll have overfit. So anyway, that's the philosophy behind it. I think a very, very important paper just because it speaks directly to the kinds of things that you can do once compute is abundant with agents, the scaffolds do become way more important, and there's a lot of fragility that's baked in. And with that, we are done with this

1:46:01episode. Thank you so much for listening to this episode of Last Week in AI. We hope you can tune in next week and that we will be on our regular schedule of hopefully calmer weeks like this, where we can get around to talking about research and all the regular stuff. As always, we will be looking up for comments, even if we haven't applied to them in this episode. So do keep commenting and do keep listening.

1:46:39Last week in AI, come and take a ride, get the load down on tech and let it slide. Last week in AI, come and take a ride, come and take a ride, come on labs to the streets, AI's reaching high. New tech emerging, watching surgeons fly. From the labs to the streets, AI's reaching high.

1:47:16Algorithms shaping up the future seas. Tune in, tune in, get the latest with ease. Last week in AI, come and take a ride, get the load down on tech and let it slide. Last week in AI, come and take a ride, come on labs to the streets, AI's reaching high.

1:47:50From neural nets to robots, the headlines pop. Data-driven dreams, they just don't stop. Every breakthrough, every code unwritten. On the edge of change, with excitement we're smitten. From machine learning marvels to coding kings. Futures unfolding, see what it brings.