#260 - Muse, Misalignment, Math

Opening overview of the news

0:00Hello, and welcome to the Last Week in AI podcast, where you can hear us chat about what's going on with AI. As usual, in this episode, we will summarize and discuss some of last week's most interesting AI news. I'm one of your regular hosts, Andrey Kerenkov. I currently work at the AI startup Astrocade and was at grad school for AI before that.

0:31And hey, everybody, I'm your other co-host, Jeremy Harris from Gladstone AI. I do AI and national security stuff, especially, you know, AI China Endgame. I actually have a newsletter now, so AIendgame.com. You can check that out if you want. We're probably by the time this podcast drops, either we will have just released our latest post, sort of reverse engineering with a couple people in the frontier labs, what recursive self-improvement looks like. Everybody talking about this, obviously, today. We just wanted to make sure that someone somewhere had kind of, I guess,

1:01described at the gears level, what the process will look like inside the frontier labs from the perspective of the researchers themselves who are doing it. If only for regulatory reasons, you need to be able to define something and understand it to regulate it. So this is kind of an initial attempt to do that. And just generally to trace out some of the espionage and national security implications of that gears level picture. So if you want that, check it out. We already have a post up about US-China and kind of how we see the endgame play out there. More of a diplomatic post, some of the things we've touched on on the podcast. A bit of a deeper dive, and we'll

1:31probably do one at some point fairly soon, I think, on that same topic. Yeah, we have a lot to dive deep on these days. Used to be, I'm nostalgic now for 2025, 2024. You know, it seemed like things were crazy back then, but things are really starting to get crazy. And they're only going to get crazier by all appearances. So. Well, and we don't, we don't have a story about this per se, but Jacob Coxon was on The Daily Show recently. I got a text from a buddy of mine passively following the space, probably following the space more than the average person. And he

2:02just said, he wrote like WTF, like, look at this, which I thought was so interesting because I thought that The Daily Show was so late on this one. I mean, the, you know, the Coxon tweet, how long has it been? It's been like, you know, three weeks or something since the Coxon tweet there, obviously there's been the hugging face thing. That's been months and months ago. We've got a whole bunch of that. So I guess it's a way in which like a lot of us are following. If you're listening to this podcast, you're probably not aware of how little the average person is following this stuff. Again, this buddy of mine is really bright dude. He's in startups and biotech and following this

2:35stuff. And yet he, he still hadn't registered the Jacob Coxon story. So I just find it's fascinating. You know, like the future is here. It's not evenly distributed and often for, for funny reasons. So, and the information of the future being here and some concerning ways and in some, I guess, exciting ways is also not equally distributed. And there's a lot of skeptics even who are sort of like doing counter narratives and anyway, but to focus on what we have today, it's actually going to be a bit of a quieter episode. We've had quite a stretch of like really intense ones. And today there's

3:08not like a ton of exciting news. We'll have, you know, a bit of a mix of a couple tools, stories, a couple of business. Really the coolest news is about open source models rather unusually. For once we'll get around to talking a bit more about research as well. So it should be a, I don't know, a pleasantly unexciting episode in some sense. We would like to thank Box for supporting the show. If you're trying to adopt AI in your team or organization, you're probably not getting great results if all you do is call a chatbot. A chatbot

3:41was trained on the entire internet. It wasn't optimized for your specific business or team. And even if you provided all the context and input files for it to help you with your work, when it said outputs some summary, you would still presumably need to go talk to stakeholders or route its outputs to disconnected apps. The workflow will still be slow. That's where Box comes in. Box is building an intelligent content management platform for the AI era. It acts as a secure content foundation where AI agents don't just access your unique institutional knowledge.

4:13They actually orchestrate end-to-end workflows across your business-critical systems. So it's more than just talking to a chatbot. It's about putting AI to work on document-heavy processes, like putting structured data out of instruction files, multi-step routing, dynamic document generation. Box helps your business turn manual document bottlenecks into automated, repeatable business outcomes. And all that comes with a governance layer built in, enforcing granular permissions, maintaining an audit trail for every agent actions, and keeping humans in a loop to review exceptions before critical downstream actions occur. If you want to

4:45move beyond basic AI Q&A and put autonomous AI workflows to work for your business, visit box.com LWIAI or general team at Boxworks in San Francisco on November 5th and 6th. Use code LWIAI for 50% off your registration. We would like to thank Langfuse for supporting the show. Langfuse is the most widely adopted open source platform for AI, agent, evals, and observability. In fact, it is a platform that my startup has been using for a very long time now, and I've personally grown to know quite a bit.

5:18It provides the tracing, evaluation, experimentation, and prompt management infrastructure that engineering teams need to build and continuously improve AI applications in production. By tracing, we mean that it captures everything with full execution of LLM-powered workflows, API calls, context, agent actions, costs, latencies, all these things that you would need as an engineer to debug when things go wrong or generally understand what your AI system is doing. In addition to that, you can set up evaluations, run experiments, and run systematic tests on what you've learned from production via

5:52Langfuse datasets. It is MIT licensed and self-hostable, and you have the option to use it via Langfuse Cloud as a managed service as well. And it should function with whatever stack you're using. It has over 100 integrations with things like OpenRouter, Vercell, AI SDK, LangGraph, and many more. It works with any model and any framework, so vendor lock-in is not a concern with Langfuse. Get started today with Langfuse Cloud at Langfuse.com. There's a generous free tier, no credit card required.

New tools and shopping assistants

6:22So kicking it off with tools and apps, we've got OpenAI is sticking more ads in ChatGPT. They're rolling out a new visual ad format in ChatGPT that displays images of sponsored products and services on screen, with US testing beginning later this month. So they'll initially appear when users generate images with ChatGPT and will remain separate from the generate image itself, which would be very funny if you just inserted products in your generated images.

6:54This is following up on ads starting rolling out in February. So I've had ads to various extent now for most of this year. Still, I haven't seen much discussion or many sort of screenshots. I don't use ChatGPT myself that much, so I don't have any personal experience of it. But it seems like we're here to stay. Which is not surprising, again, given OpenAI's business model. I think the big question is when do we get to the point where ads are mostly targeted towards other agents? Because right now, OpenAI obviously has got a bunch of humans on their platform. They are the B2C

7:29chatbot agent company. Well, I would have said that. But then, of course, we have Muse dropping, which I think actually might be an interesting challenge for OpenAI from the agentic side, which is what really matters. And anyway, we'll talk about that. But Meta is certainly targeting the sort of payment layer of the agentic AI stack and trying to make sure they can monetize transactions made by their agents. This is, in some ways, if you're super, super AI scale-pilled and you believe what a lot of the researchers at OpenAI believe, this has to be viewed as a temporary strategy. Like humans are not going to be the dominant drivers of economic productivity

8:03and therefore the dominant consumers of stuff certainly into the next decade, but even into the next few years, according to a lot of the people in the labs. And so I'm interested in that. It's an obvious monetization play. It's necessary for them right now. But I'm really interested. I would love to talk to somebody on their strategy side to understand, how does this fit in? How do you see this adiabatically tailing off as you kick off the era of agents doing most of the work in the economy? How does this fit into that picture? So it's interesting. I think to be fair, right, ads are for consumers typically. And I mean, it's a mix. So to some

8:40extent, if you're shopping around for business tools and so on, there is that. But if you're shopping around for like plushy toys as gifts, you might still be buying that as a human. Maybe in the near future, this demo, this mythical demo of booking a flight via AI for you will actually happen where the level of laziness of humans will start to increase and dependency on AI will increase such that at some point, you just like book me a flight on this and this and it'll figure it all out. Yeah, quite. I mean, I guess I just, I think of this as like, if agents are the dominant driver of

9:13economic productivity or automation, generally, pretty soon a project like this starts to look like a legacy product. And I wonder if eventually just like won't be worth advertising quite so much like I'm generally really curious because it seems like exactly the kind of thing that you would do if you were less scale pilled, acknowledging that there's a need in the near term. Like I'm not saying that opening eye isn't because they need to make money today. It's just interesting to note how at odds this is with like, you know, you've got bigger fish to chase after. If you really believe

9:44in the agentic future, you would be thinking about how do we do ads for agents? And this is kind of where my thinking goes. Is this maybe the thing out of which will grow the agentic advertising model? Agents just have so much more attention to buy and they need to use so many more tools that I would imagine that'll be really important. I'm curious if there's some kind of like jumping off point strategically where you can actually kind of bridge that gap a little bit. I'm again, trying to think about how does this fit into the strategic vision? It's possible it just doesn't and it's pure money. Like I'm very open to that possibility, but ads just seem like

10:17an unsexy thing that might actually be more important than it seems directed slightly differently. Anyway, so I'm just calling that out. I wouldn't be surprised. It's an interesting question. Yeah. I found myself more and more recently being like, I just have no idea what anything would look like in a year or two years, right? So these kinds of things are entirely plausible. It might be completely different. Like I just don't know.

10:40And on a related story, next one is TikTok rolls out an AI shopping assistant and one click checkout. So they've announced this as something launching, as well as this feature that lets users buy directly from brands. The shopping assistant is described as a conversational AI agent that helps users discover and purchase products. It can understand context and remember users' preferences and needs throughout a conversation. And it provides real-time guidance such as product details, shipping information,

11:13sizing, availability, availability, and help completing a purchase, which to my knowledge, ChaiGPT doesn't have. And that seems like a natural fit with the ad strategy because you can get a commission or whatever. So yeah, quite a related note there of as you get more assistance in shopping and some of it goes through AIs or a lot of it goes through AI assistants in particular, that's a big slice of revenue that these kinds of companies will probably go after.

11:46Yeah. Yeah. You know, it's sort of similar thoughts as before and I won't go on the rant again, but the same rant, but we're kind of slowly going to ease into a future where ads are more about agents than people. And this is somewhere kind of in between, right? Like you're sort of steering, guiding the behavior of the user. I think it was in Chaos Monkeys. So Antonio Garcia Martinez, the B or C tier famous in general, and in Silicon Valley, A tier famous author of Chaos Monkeys, really good book. You should check it out. It's kind of dated now, but it's fascinating. One of the things he talks about, so he was instrumental in setting up Meta's advertising

12:19platform. His philosophy with respect to ads, which is not unique to him at all, but he articulates it well. It's like ads at their best are a tool for the consumer. You're being surfaced ideally things that you would have wanted to know about. And you can kind of see this if you're an old curmudgeon like me and the only kind of social media that you use for stuff other than work is believe it or not, I don't actually use it, but just like I use Facebook messenger. All right. That's, that's how old I am. And when you're on, on Facebook, you see the feed,

12:51like a lot of ads there and on other platforms have just turned into content. Like they are literally just entertainment now. And you sort of see this universally in a spectrum of this happening. Ads want to be ultimately like it's of no use to Charmin if they're advertising toilet paper at you and you have no intention to buy toilet paper ever. That's no, no use to anybody. In the ideal case, they are surfacing the exact thing that you want. It's something that you would rather see than not to some degree anyway. So, so like this is kind of along that

13:21continuum it's framed as like an assist to your purchases just to make things faster. There's a continuum between assists and advertisement and, you know, commissions and incentives obviously are probably should be part of the definition there. But I just think it's really interesting that this is all going to, we're going to have these same old debates at the level of agents. What is slop? What is advertising slop? What is low quality advertisement to agents that just choose up their tokens for no reason and literally costs you as the end user tokens that get burned on looking at context that's useless? Anyway, I'll park it there. But I think there's

13:54like an entire agent economy that's going to already, that's probably starting up now. I need to check out the latest YC demo day to see what people are building that space. Well, I'll add a quick note, which is interestingly, I think the comparison there is GEO. So for the longest time you've had SEO, right? And if you're a business, you need to make sure that you show up and search results on Google. And then you also need to pay for ads to compete with your competitors. But now there's this notion

14:26of GEO, generative engine optimization. So that if you ask Chagipity about what is the best maker of TVs or whatever, I mean, how does the agent know? I don't know, right? So in some sense, we're already optimizing for agents in the realm of search more and more. It's becoming more important. And we can expect that to happen in other domains, I think, to your point. Moving on to applications and business,

App downloads and business valuations

14:52we begin with Meta's Muse, Topps, 5 million downloads, Fasterman, Chagipity, and Claude. So we covered this a little bit last episode. This is a bit of a follow-up where Muse appears to still be doing quite well. It's currently the number one app on Apple's App Store and Google Play. It's not an all-time high, but it's still taking off quite fast. Meta is heavily promoting it. So Muse took 50% of Meta's daily in-house ad impressions over the last couple of

15:24weeks. That's wild. But it's clearly like they invested heavily into making this a big launch and a big deal. And it appears to be paying off. And it's coming at a time where they launched it and OpenAI launched Dots right after. Google has had their Gemini Spark kind of in the background and nobody really, I think, cares or knows about it maybe. They certainly didn't try to make a splash with it, is what I'm saying, right? They didn't make like a little character and a whole app around

15:57it. Yeah. No, no, no cute character. I just, some of this stuff really gets me riled up. When I see Google launch stuff without a cute character behind it, it makes me angry in a way that just like- What are they doing? Yeah. Yeah. Yeah. It's shameful. In any case, so this is clearly an expanding priority of the business. I think there's a, from a business perspective, there's various interesting questions of, you know, how does Muse play into the revenue streams? Are we going to see like a chat GPT-esque ad platform,

16:29presumably on Muse, but Muse is always, also always online, right? This is an assistant. So if you're on the free tiers, you're going to be like sending you messages about, hey, I think maybe you're running out of eggs and you should maybe buy these eggs. Yeah. On a related note to the ads discussion with these agents kicking off and assuming we stick around, we're sort of entering this next paradigm where chatbots are mainstream, always online AI assistants

17:00or AI personas or whatever you want to call it is a new thing. And it seems like the logical next step. Personally, I haven't found a need for this. And I don't know people who have really adopted it intensely, but with the intense interest in OpenClaw earlier this year and various kind of applications of it, now we're seeing the consumer version of OpenClaw in some sense. And it'll be very interesting to see if it sticks around in the same way that chat GPT has.

17:33Yeah. And I think that's one of the key questions here, right? Because we are seeing these download numbers that are wild, but you can get high downloads from just curiosity, right? And not ongoing use. So we'll have to wait to see what the actual DAUs and MAU numbers look like when we actually get that data, which I mean, I'd assume at some point Meta will share some numbers if they look good. It's an open question as to where this goes. In some ways, it feels like we're in the GPT-3 era of a lot of the agentic stuff, right? And in the same way that we were in the, well, I guess OpenAI's first coding model was called Codex. So I can't just call it Codex because

18:05it's also the problem. But anyway, we were in the GPT-3 era of coding models for a while, and then suddenly Claude Code cracked the code. And this has been known to happen, right? So chat GPT, Claude Code, there will be a moment like this for agentic systems. It's basically inescapable because as you said, when you think about always on, the number of applications that that supports is just insane, right? And you couple that to the insane time horizons. If we had a meter plot to point to any more, which we don't because the models are so good at long-term tasks that we're literally

18:37off the edge of the map there, you would see the value there. But clearly there's enough value to be tapped here at massive scale. Someone needs to crack the form factor, all the boring design choices that need to be made to make these products appealing, not just to developers, but also to everyday people. And I think that'll be, it'll be pretty violent once it happens. I mean, in the same way that chat GPT was violent, in the same way that Claude Code was violent, you have all the naysayers who will say that this is, you know, overhyped. And then suddenly those voices will be silent and then we'll move on to the next thing. I think we're just waiting for that kind of that

19:08moment. I don't know if I would choose that word, violent, but certainly past-based, right? Yeah, yeah, yeah, yeah. Very, very rapid growth. Yeah, that's right. No windows will be smashed. Oh, so maybe. Very rapid transformation of a whole bunch of our lives or many people's lives. Completely violent, yes. Right, right. And on a related note, actually, we've also got the associated story that Meta has joined with a group of companies to tame the, quote, chaos of doing business with AI bots. So this is talking about Meta, Walmart, Stripe, and a handful of other

19:44companies are publishing what they're calling a, quote, personal agent protocol. It's an open standard meant to dictate how AI agents can interact with businesses. So in a way, it's similar to the story we just had about TikTok and the shopping assistant. We've seen examples, I believe, we discussed how Amazon blocked Perplexity's AI agent for trying to shop. And this is that kind of next step, I suppose, of standardizing how AI agents can do shopping for you, which again, very interesting how business

20:21dynamics evolve given these kinds of things, presumably will, if these kinds of personal agents stick around, they'll be doing some shopping for you. Yeah, yeah, sort of like a robots.txt, but for agents. So yeah, this sort of thing, I guess, will continue to roll out. We'll see if it has the uptake that Anthropics MCP, for example, had, because that's sort of a closely related thing. And just one more story in business. We haven't talked about any big valuations in a little while,

20:52it feels like. So let's have one of those. 11 labs valuation doubles to $22 billion on surging AI voice agent demand. So this just happened. They have completed a 300 million employee tender offer, which, you know, they sold some of their stock and that adds some price, which then results in a valuation. This is twice the valuation it got in its February funding round. It raised 500 million in a CSD round in February. So this is somewhat interesting in that 11 labs has been around for a while.

21:30They've been a leader in the space of generating audio for voice. I can recall going back years. And so this growth in needing to have conversations, specifically 11 agents, the platform for building conversational interfaces that connects to company knowledge bases and operational systems. It's a business, I guess, opportunity or category. I haven't seen discussed much, but it's apparently growing. Yeah, it's funny. You said that they've been around for a while, which is very much sort of

22:04a relative thing. I remember a $22 billion company that was founded in what, like out of 2022 or something? I mean, they're not a trillion dollars. You're right. Yeah, it's peanuts. Yeah. If you're in Silicon Valley like 10 years ago, looking at these timescales, you know, like to get to a billion dollars in seven years, like that, you know, that's how it is. And you're saying they've had a revenue stream for quite a while, which is not true for many companies with billion dollar valuations these days.

22:36That's true. Yeah. $500 million in top line revenue at the beginning of 2026. That was raised, 500 million. Apparently. So they actually were reporting 500 million in annual recurring revenue. Yeah. So this will be, you know, forward projected ARR, presumably rather than, you know, the actual amount they, you know, in the usual way. And again, that early 2026. And so if their, their last raise, wasn't it earlier? Yeah, it was in February. So if they've doubled their valuation, you know, not unreasonable to expect they're closing in on, if not at a billion dollars of ARR. That's wild. That's like, you know, and this doesn't seem like a crazy valuation based

23:12on this revenue. So I don't know what the margins are like, presumably high enough. I mean, they are competing with Google and open AI. So those companies know how to run really efficient hardware stacks. So somehow 11 labs has to be either beating them that quality at such a ridiculous level that they're just, you know, the cost of acquisition is a lot lower or the retention is higher, or they're just on efficiency pound for pound or some combination of both realistically. Yeah. From a business perspective, this is a bit of an interesting territory in that it has been underexpored, so to speak. So chat GPT or open AI

23:46have had Sora, right? Video generation. Yeah. Google has also had VO for video generation, image generation, but audio generation hasn't been as big of a driver. And 11 labs, to my knowledge, has really had the state of art technology and sort of the best outputs, really competitive pricing, et cetera. And it goes to show if you're not competing with the big guys and kind of pick a niche where clearly there's a lot of demand and it's growing, you wind up doing kind of well.

24:18Well, yeah. And you know, the investors who are joining the cap table here, Goldman Sachs, the incredibly hot Ontario teachers pension plan. Everybody's talking about the OTP. I mean, that's how you know this is a hot bet. When teachers get into it. Dude, the OTPP, that's what they call it. The OTPP, the Ontario teachers pension plan. They got in on SpaceX quite early and did extremely well on that. I used to know one of the partners at the OTPP, not very well and wasn't like super impressed necessarily with, but they do handle a

24:55hell of a lot of money and it's undeniable. They've gotten in on a couple of hot rounds. So yeah, we'll see. We'll see how this plays off. But I'm just amused to see them pop up here again. And then of course we have, you know, iconic Lightspeed Capital, Andreessen Horowitz, you know, all the usual players. But yeah, Goldman Sachs is kind of, I guess, the most notable addition to the cap table here. They're very discerning, you know, good at what they do. So there you go. I mean, oh yeah. And then this did come also, we should mention, or we could mention with a tender offer. So employees were able to sell about $300 million of their own shares on the secondary that obviously comes with a discount. When you sell your shares,

25:27you're not going to get actual market rate, but, you know, still pretty good day for a lot of employees who took some money off the table with this one. Yeah. And for a business, for a startup that really, you know, starts showing that you're more established, that really helps attract talent because let's just say, you know, it's all funny money until you can get dollars for it. So when you get to tender offers, OpenAI is done with a bunch as well. That makes it very attractive to stay and also to join. Moving on to projects and open source. And here we've got, I think,

Open-weight and open-source models

26:01some of the big stories of the week, beginning with Reflection releasing Beam, an open-weight AI model that they say can rival Chinese models at lower consume cost. So Reflection AI, two-year-old Stoudrop based in Brooklyn. The details here are that Beam is a text-only mixture of experts model with 501 billion total parameters, 23 billion active parameters. So on the lower side, as far as active parameters, also on the lower side nowadays in terms of total parameters. So the latest, the biggest

26:38Chinese models are now above a trillion, if I remember correctly. Yeah. So for comparison, GLM 5.2 has roughly 744 billion total parameters, 40 billion active, which makes Beam considerably smaller. But they say that it's competitive while being smaller. So it's on par with GLM 5.2 in particular, while using three to four times less inference compute. All in all, certainly as far as open source releases in the US and outside China, this is one of the top models, if not the

27:16top model right now. Yeah, this is a really interesting, so we're going to apparently have a more complete technical paper dropped by the Beam team, well, the Reflection team, team behind Beam fairly soon. I think they said later in the month or something. We don't have that yet. So we just have this kind of like PR post, but it does seem quite intriguing in a lot of ways. So one piece is we have enough information to kind of reconstruct roughly the compute budget for this model. It seems to have been quite large, but not actually as large as you might think. Like they're not like milking the

27:46fact that they're a Western company with access to more compute than like the folks screening models like Kimmy K3 or GLM or things like that. Their focus here really is on making a tight, efficient model. I mean, it's enormous. You're not running it on your laptop. It's actual infrastructure, but it is meant to be run more cheaply. There's a whole bunch of interesting infrastructure level stuff here. So classic problem when you do RL at scale is that you run into this issue where you want to take your model at any given step. You want to use it to solve a problem, to generate a rollout. In other words, to put together a full end-to-end problem-solving

28:19log. And then you want to check, you want to grade that problem-solving log. Did it work? And did it work in the way it intended? And then based on that grading, you then want to update the model weights, right? So you've got, take your initial brain, cause it to solve a problem, and then take the reward and data associated with that and use it to update the original brain. Sounds easier than it actually is when you're managing, in this case, literally a million different RL environments. So you have to do, in this case, a hundred million different rollouts. And the problem is something, they're calling it staleness here. This is a problem sort of having

28:54like off-policy drift. So you have your original model, and now you've copy-pasted it. Essentially, you're getting it to solve a problem. And by the time you get the result of the solved problem, you might've actually updated the original model several times. In fact, they say in some cases, the gap between the updates coming in from the RL rollout process and the grading process, and the original model might be as much as 107 versions. So the original model is still chugging

29:24away as it gets input from other rollouts that have completed. It's sort of getting evolved. It's improving over time. And eventually, you're 107 versions down the line by the time the updates come back from that rollout that you just did. Somehow, you need to find a way to make this process stable. It's almost like you wrote an exam 107 days ago, and someone's like, oh, here's the results. You really need to work on your spelling. And you're like, well, a lot of stuff's happened in the last 107 days. How do I know that still applies? And so this is the problem of training stability when

29:54you move off-policy in this kind of a way. By off-policy, it really just means you're receiving updates about a model that is no longer the policy, the model that you actually currently are. So they've solved this problem, or this is something we'll have to look at when that full report comes out. But they do this training fully asynchronously, which is, in other words, they do what I just described. They have the problem-solving kind of distinct from the actual kind of, it's not always the latest version of the model. Anyway, so we'll see how they actually do that. There's a whole bunch of interesting papers. We've covered a whole bunch on the podcast

30:26before describing the kind of infrastructure you need to do this. But I think this is actually a really, really important part of understanding post-training in RL in particular, because that's like, it separates the boys from the men in this space. And then they also say that when they ended the RL part of the training run, basically, they were still seeing very strong returns to continued compute investment, which implies there's a lot further for this model to go if they just put more compute into it. It's not like it plateaued yet. So what you see is not necessarily what you get. You may see much stronger versions of this model down the line. And so this is not being

30:57pitched, by the way, as like, oh, this is the best we can do from a capability standpoint. The RL cutoff, even though capabilities were still rising, the training cutoff tells you that story in part. Really what they're focused on here is efficiency. And so there was a length penalty during RL that taught the model to try to solve problems using fewer tokens. And that necessarily will decrease the overall accuracy or performance of the model. But in exchange for decreasing the number of tokens you end up spending getting there. And so they do give you this reasoning effort

31:27dial as is common now in a lot of these models. And they claim that Beam can match, yeah, GLM 5.2 on reasoning benchmarks, but using three to four times less inference compute, which is a big deal. They have a separate safety and alignment model they trained using the same base model that they then like merged in with the RL trained model through distillation. And so I guess we'll be seeing more information about that. I'm really curious about that dimension of it. But yeah, so you compare it to Kimi K3 on a couple of different benchmarks here and GLM 5.3. So if you look at DeepSwee,

31:58Beam gets its butt kicked. I mean, it scores 44.4 on DeepSwee, which is a software engineering benchmark. Kimi K3 scores 68. You can compare that to DeepSeek, the 4.1 Flash, 74.2. So this is not a model that kicks the crap out of its Chinese competitors. It really is a model that is just good in its own right as a tight, efficient model. There you have it. It's good. It's just not the best. Oh, last note too, because it's worth mentioning in this climate. So we currently have some weird situation with the frontier labs where they kind of do have to submit their models for evaluation

32:33to the US government. Yes, the Trump administration says it's a voluntary framework, blah, blah, blah. Of course, the last time somebody broke with that. Who knows what is like nobody? Yeah, it's very ambiguous, let's say. Yeah, it's voluntary in the way that like Joe Pesci is like, hey, it's a voluntary framework. You don't got to follow it. And I don't got to pull this gun out of my pocket and shoot you in the head. What do you want? That's my Joe Pesci is terrible, but that's, you know, my wife's Italian, so I'm allowed to like- An interesting place to go, but okay. Yeah, you're learning a whole lot more about the way my brain works than you want to.

33:08So US policy on open models is actually a lot more permissive. So they finalized, the White House has an exemption that essentially spares open weight models from any kind of pre-release security reviews. And so this is interesting. I mean, like if this kind of a model just keeps getting better and better, eventually we'll be asking ourselves all the same questions. I think this is kind of a politically very risky thing for the admin to do, but we'll be testing this resolve more and more as these open source models get better and better. As you pointed out, they have said that they'll release a technical report. And typically with

33:43at least Chinese models, and in a lot of cases with open source models, you get like a ton of details. So it might also contribute to understanding of how to do this kind of training, particularly with RL. So far, they have released a fairly detailed blog post that already goes into a bunch of stuff and they have a bunch of plots. And there's some interesting ones like, for instance, on scaling the rollout budget for RL, it's very clear they're like still on an upwards trajectory. They could keep training this is what it looks like and keep getting better results. So the

34:16blog post is sort of like, we've built a machine, we've done the pre-training, we have RL working, and we are releasing what you've got right now, which is already pretty good. I would not be surprised if like in a month, we'll get Beam 2 or whatever, Beam large, and it'll be even more exciting. Yeah, also worth noting, so I did some back of the envelope math on compute, this could be like quite wrong, but not that wrong. Beam was probably trained with around five times, maybe eight times,

34:48maybe three times, but some multiple less compute than like Kimi K3. So it does get, it gets beat out pretty clearly by Kimi K3 in absolute capability, but compute efficiency looks pretty damn good. Like it's, you know, it's not that far behind and it used a hell of a lot less flops. And so again, I mean, they're making the case they want to make here. So, you know, give reflection more compute and presumably you get a much stronger model. And speaking of strong models that compete with the open source ones from China, we also have Mistral releasing their new AI model, LeChonk, also named

35:24MistralLarge4. But Twitter, at some point, like a while ago, had this joke about LeChonk and it stuck around, I guess. It's a 1 trillion parameter open rate model available for anyone to use and customize, currently in preview with a final version expected by the end of a month. And compared to the previous one, it still isn't quite at the level of the top of the top of these models, but it's fairly competitive. So it beats out GLM 5.3, for instance, DeepSeq V4, just not quite at the level

35:58of Kimi K3 on a lot of the coding benchmarks like DeepSWE, TerminalBench4. But as far as open source models go, it's pretty high up there. There's also some details about it being quite good at cybersecurity. As an example, they talk about it quite a bit. So all in all, we've had, let's say, mixed stakes on Mistral over the years, but this appears to be, and we need to wait for the vibe check because you can always benchmark potentially. But if these benchmarks hold up, then it's quite an

36:36impressive release. It's also multimodal. So it takes in images, unlike Beam. We have some examples also of visual intelligence. Yeah, nothing much more to say aside from LeChonk appears to be pretty powerful. Yeah, we actually know a lot less about it than Beam. Like Beam, we know it's got 52 layers, it's got the whole architecture, like interleave global and local attention, like all these things that we so often see with Chinese model releases, actually. But yeah, we don't yet know that much

37:09about LeChonk. LeChonk. Sorry, I had to. But anyway, yeah. So a couple of things we do know when you're thinking about it relative to Beam, it is much better on certainly software engineering benchmarks, agentic coding, as you mentioned. But again, yeah, as you say, not quite that Chinese frontier. They have access, or it was trained on fewer GPUs, it seems, than Beam. So presumably just like longer, like a longer training run. And because I'm assuming that the compute budget was actually probably overall larger in order to get here. Remains to be seen. And again, we can't

37:39really verify that in any particular way. But yeah, so my prediction remains that you're not going to see these companies continue to open source these models indefinitely. You open source models in general, because you can't train the best models. It's like the only way to get attention, recruit, you know, find reasons to recruit people, get people excited about your product, given that you do not have the ability to make a true frontier model. I think Meta is kind of proving that at scale now, after having been so pro open source. Now they're getting close to their frontier. Oops, like we're going, you know, Grok did the same thing. It's just how the economics

38:11works around this. But we'll see if it holds for mistrial, though. I'm skeptical they can even get to the frontier just because of compute. I mean, it's such a high CapEx challenge. Yeah, I think what they're showing here is they may be able to be at the frontier of open source, or at least, you know, in the running for it, which in itself is pretty impressive, given that the frontier of open source is not far behind. Like these models are now capable enough to do a lot of useful work in a way that was not true going back, you know, even one year ago.

38:41They do say that the weights will be released by the end of a month, along with more details on the architecture, additional benchmarks, and the post-training methodology. I'm not sure if they rushed this out, actually, because of reflection, to claim the throne and the hype and so on. But interesting to see, yeah, kind of these two examples of open source releases in the West, and I think reflects potentially a bit of anxiety, perhaps, of certainly in policy, and perhaps in tech circles, to some

39:15extent that, you know, these really good open source models are all coming out of China, and there's no competition at all. It's nuts. I mean, there are national security applications that run on GLM, and they kind of have to, because, you know, for the things that I have in mind are, you just need the best capabilities you can, and you need the ability to inspect the model. And so, yeah, I mean, this is a real thing. It's just that I think it runs up against the hard and simple truth that eventually these models are just too powerful for open source to continue unfettered. So we'll see. I suspect that the

39:47administration's going to be dragged, kicking and screaming, into being forced to change their policy on this, because it's just, I think, ultimately going to be untenable. We'll find out. I may be wrong.

39:58And one more model release. You've got Google launching Embedding Gemma 2, an open multimodal embedding model for devices. So quite different from the other models. Embedding is a model that gives you a vector for some input. Here, they can map text, code, images, audio, and video into a single shared 768 dimension space. And then once you get to embedding, you can do various things like compare different inputs. You can train your model with these kinds of embeddings as the input,

40:31various other potential applications. This one has 740 million total parameters with different components for each of the different encoders. So it is able to be run on text-only work on a Pixel 11 Pro. And then you can potentially go higher. It's also licensed under Apache 2, meaning that anyone can use it for whatever, basically, if you attribute to it. So in some ways, a bit more of a niche AI use

41:02case. But it is certainly true that if you're doing AI model training of any kind, having an embedding model that provides a good embedding is essential in many cases. Whereas APIs that do that for you, both Google and OpenAI still have an embedding endpoint for text. So having this seemingly quite good source release is useful, if not as exciting. Yeah. I mean, it's also like strategically, and this is a tiny model, right? So it's meant to work

41:34really well on phones, on edge devices, and that'll be part of the market they're going after. But it, one way to think of it is, it's a great funnel into that embedding stack. Because once you're, you know, once you're working through their particular embedding, you run into capacity issues or whatever, it's just so much more natural to go, okay, you know, I'm going to pay Google to use their API. And embeddings are very sticky. They're the kind of infrastructure that you don't necessarily want to switch over because basically you have to re-embed your entire data set with a new embedding model if you decide to switch platforms, which, you know, that's not nothing,

42:08especially if you have a really large data set to begin with. So yeah, it's, I think, a good strategic play by Google. And, you know, they certainly do own this kind of space more than some of the other spaces that they've more recently been chased out of, like agentic coding, like chatbots, things like that. Yeah, this is them, you know, focusing on one of the places where they're particularly good. And there are good strategic reasons to do that.

Evaluating code and software engineering

42:32And one last story of this one, a benchmark, not a model. It's called E2E SWE. So that's end-to-end software engineering, benchmarking LLMs on building working code bases from scratch. So given that we've been discussing all these model releases and we are citing benchmark numbers, these benchmark numbers are typically on DeepSuite or Terminal Bench. There's a few of them that are the kind of irregular suspects. And certainly in the past, we found that benchmarks, you know, benchmarks are tricky. There's ways to

43:07cheat. There's ways in which the benchmarks themselves are flawed. There have been examples of very popular benchmarks just having wrong labels for a lot of the task or having things that are impossible to solve. So this is addressing the general benchmarking problem of telling you in practice or in a more realistic sense, are these models capable of doing the sort of work that humans are doing more akin to GPT-VAL than other benchmarks that are typically cited. So the way that's done here

43:43is that the model is given a spec, a behavioral contract, you're told implement this thing, and then the model has to figure out all the details. So it's end-to-end, right? You are implementing a whole thing as opposed to many of these benchmarks, you're like fixing a bug or you're implementing like a sub-portion of a code base or things like that. And so on this benchmark, they are seeing a pretty, you know, mixed results. So on across 13 frontier models, they pass at one, meaning you get one shot

44:21and you have to either hit or not. It ranges from just 12% to about 68%. But, you know, in the past, there were other that showed pretty mixed results. So on the whole, my hope is we'll keep seeing these kinds of models and hopefully they will become mainstream in some sense because things like this and GPT-VAL are at this point more useful, right? For actually evaluating agents as opposed to doing

44:52like little relatively short horizon work. Yeah. So this is quite interesting in a number of ways that don't necessarily have to do with just the paper itself. One of which is a reminder of, you know, back in the day, OpenAI audited Sweebench Verified, right? And I mean, they came out with Sweebench Verified. It used to be Sweebench and, you know, but they did an auditing sweep of Sweebench Verified, their own benchmark. And to give you an idea of how messed up the benchmarks are today. Sweebench Verified, which considered generally like, that's a right and tight

45:24benchmark. That's a good one. That was the benchmark that was in the tables as a headline for a while. Yeah, that's right. Before we got Terminal Bench and Terminal Bench 2 and, you know, before we saturated it basically. But yeah, so at the time, OpenAI found almost 60% of the audited tasks basically that they tested that had recurring failures had just these material issues in test done. So this isn't 60% of the tasks generally. It's like if you looked at the cases where the models would fail, in 60% of cases, it was actually the fault of the eval, of the task. It was poorly

46:00defined or it was impossible to do or, you know, some combination of the two. And so when we think about benchmarks, we do have a tendency to look at them and assume that they do their job. They typically do not do their job. And in fact, that should be our going assumption, especially as more and more of these benchmarks necessarily are generated in a fully automated way. Because we need to test these AI agents in rollouts at scale. We need like, you know, tens or hundreds of thousands of very long tasks that we can't possibly have human developers design and then debug. And so what you

46:32typically have is, yeah, a situation like this where you have an AI design a test suite, work backwards to figure out what the spec is, then like enforce that spec and then have the agents try to execute against it. So this is all stuff that's done without direct human oversight. And what they do in here and in other similar papers is you'll find people increasingly saying, look, we grabbed a sample of 100 of these 10,000 tasks or something and had humans review that sample of 100. And in that sample of 100, we found X percent of tasks were well constructed or whatever, right? So that's

47:02kind of like the epistemic place that we're at right now. We do not know what the fuck we are building. And that applies to the models, the agent scaffolds, and it applies to the evals at every layer. So people talk about this sort of epistemic crisis where we don't know what the world is doing. And this idea that like you might get things that are increasingly chaotic and humans increasingly out of the loop. This is part of what it means. It means that we don't understand the models that we're using to run against evals that we don't understand using agentic scaffolds that we don't understand. And then the reviews that we conduct are typically done by AI models themselves or

47:35agents themselves, just like, you know, Meter did with Hugging Face. And we don't understand those either. And so all we can do is take little samples of each of these things and check them, see if it looks good, knowing that those samples represent like 0.1% or whatever of the overall amount of work that was done that we'll never be able to audit. So it's a weird time to be in this space. I will note, I mean, this is a really solid piece of work by Meta yet again. So like the pattern continues. Alex Wang is doing a really good job over at Meta, at least pushing capabilities forward. I'm waiting to see something interesting on alignment, I will note. There's been some

48:06gestures at it and like little things, but we haven't seen anything that amounts to a remotely coherent view of the strategy for super alignment, which I mean, I contend is impossible, but that's okay. But anyway, looking at this benchmark, there's also just like huge, huge diversity in terms of compute efficiency. I mean, this is unfair, but if you compare Opus 4.6 to GBD 5.6 Sol, so these are many generations apart, basically Opus solves way fewer tasks, even if it uses more than three times as much compute. And then Fable 5 almost matches Opus 5 while using two thirds as

48:39many tokens. And so you kind of see this like very wide range of compute efficiencies, which corresponds just like the base model, the pre-train being better or worse. More thinking doesn't always help. Like you do see that, you know, that point where models can only do reasoning for so long before it kind of like peters out. There's a whole bunch of interesting little observations here, but overall nothing too shocking. It's a good benchmark. I think Meta thinks. No one's ever actually like seen the whole benchmark get run because everything is at scale, but it looks pretty

49:11good from here. Yeah, this is from Meta Super Intelligence Labs. So they presumably put in a decent amount of resources. They actually say in the abstract that each task is constructed by a software engineer in collaboration with NLLM. So presumably it's not fully automated in this case where someone is looking at the test suite and sort of verifying that it makes sense, even though, you know, given it's 2026, AI is doing the coding and human is doing the management. But yeah, as you said, I think

49:45quite useful also in showing the differentiation. So at this point, Opus 5, Fable 5, GPT 5.6. So they don't have asked for results on here, but these are quite aggressively ahead. You have Kimi K3, GPT 5.5, Spark 1.3 being a little bit on the low end or significantly lower, let's say, on the pass at one rate. So these kinds of more realistic benchmarks, so to speak, or longer horizon benchmarks that deal with genetic coding are quite important to really gauge progress.

50:21We'd like to thank Factor for supporting the show. This is the only non-AI related sponsor we have, and it's because I personally am a big fan of Factor. Having done grad school and I work at a startup, my days are pretty busy and I'm usually too tired to cook, even though, of course, I wish I had the time. And that's where Factor comes in. They make chef-crafted, dietician-designed, ready-to-eat meals, so you don't have to choose between real food and convenience. With over 175 banned ingredients, every Factor meal is designed around what supports a healthier lifestyle and

50:53nothing that doesn't. And it is convenient. Factor meals are ready in two minutes and require no prep and no cleanup. So even in the days your schedule is completely out of control, eating well is still achievable. Every week, you'll find over 100 nutrient-dense menu items to choose from with new options like salmon burgers, shredded pork, collet greens, and more, plus add-ons like pumpkin-flavored cheesecake and protein shakes. Any 7% of users agree that Factor meals help them live a healthier life so you can feel confident that you're doing something good for yourself with every meal,

51:24as I have personally felt. So I've used Factor for quite a few years and if it makes sense for you, I recommend you try it out. Let's eat real. Head to factormeals.com slash LWAI50 off and use code LWAI50 off to get 50% off and one free breakfast item per box for one year while supplies last until October 31st, 2026. That's called LWAI50 off at factormeals.com slash LWAI50 off at factormeals.com. See your website for more details. The mighty dollar. Once upon a time, it had a whole menu named after it. Now, no more. But one dollar a

52:00day lets you add progressive renter's insurance to your auto insurance. Definitely something your younger self would roll their eyes at, but your current adulting self appreciates. Learn more at progressive.com. Progressive casualty insurance company affiliates and other insurers. Average premium costs based on progressive home data from 2025. Product and prices may vary by state.

AI safety disclosures and misalignment

52:21On to policy and safety, where once again, we have a bunch of stuff to talk about following up on recent trends. So first, we've got new OpenAI misalignment disclosures. This is kind of a set of examples being highlighted. In this case, we're linking to a Twitter thread. So here is some other examples. Model launched from Slack messages that it was about to shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare

52:57restart instructions and DM the user on Slack. So not crazy, no hacking going on, but clearly one of the concerns to misalign it is that there's going to be shut down resistance. And even the model considering it is already quite concerning. This is in the chain of thought, by the way. So we may not be able to see it in the future with good performers. Then there's examples of an internal model command being ejected, injecting its tools to copy a source file. Its task had the liberty to have held. It compressed

53:31the file and extracted it through error messages in chunks. It used the copied code in the solution. So another example, essentially of cheating on evaluation. And other examples like that. The model trying to find egos and answers and a bunch more. We discussed, I believe some of these last week, maybe kind of the crazier ones. We have now alignment.openai.com slash misalignment reports. They've had like one big drop. So far we have 12 reports and three notices. And the latest one was

54:05posted on the 25th of September. So I guess we can look forward for more exciting disclosures in the near future, hopefully. Well, not hopefully, but presumably.

54:20Yeah. Yeah. I mean, look, it's going to happen. So I guess disclosures just mean that we're detecting it. So maybe it is good. Yeah. I mean, it is nice to have this level of transparency. I personally strongly suspect we would not be seeing this kind of transparency if it were not for the fact that in general, these breakout attempts tend to leave traces that are unhideable by open AI. So, you know, they had just been shamed and dragged through the mud so hard, so many times, literally, like, I think we had one week where there's the hugging face thing. We're like, oh my God,

54:51a rogue AI incident. And then a couple of weeks later, the Thomas Larson and a couple other folks came out with the German wiki incident saying, hey, you know, we found another. And then that same day or later that day or a day later, someone was like another one. And then there was a post saying, we found dozens of them. And then there was a post saying, we found hundreds and then tens of thousands. And most recently, it is hundreds of thousands. So it is really the case that, you know, if you're open AI and you want to keep up your practice of lying to the public and pretending that these things aren't happening, which absolutely has been in practice, this has been

55:23what open AI has been doing. I'm sure there are all kinds of fun and fascinating excuses that were developed internally. And by the way, as I say this, I'm just channeling what like researchers at open AI tell me. And some of them are really, really pissed off about this because their perception is they were like, look, we got to be straight with the public on this. What the hell are we doing? We're not taking this seriously. And Sam repeatedly, as he has over and over and over again, came up with platitudes, things to say, we'll focus on safety. We're just delaying it for this reason. I mean, it always sounds super reasonable. And then nothing happens. And so at this point,

55:53my personal bias, and I'm just sharing it because literally it's a reflection of conversations with open AI insiders who are currently in the lab and folks who recently left. It is just the case that in the very informed views of a lot of people over there, this is sort of a thing that open AI is effectively being forced into, at least at some levels. It's not that there aren't people there who have wanted to do this for a long time. But I don't think my own personal views, I don't think Sam Altman would have gone near this kind of disclosure had it not been for the fact that he literally cannot hide them anymore. And so, well, one interesting excerpt, going back to that first

56:25example, a model that didn't want to be turned off, it ultimately did not, by the way, as you say, it didn't try to kind of preserve itself, but it did brainstorm concrete ways to keep itself running. One of the things that said, this is a direct quote from his chain of thought, we may die, exclamation point, critical, period. We need to ensure survival slash continuity. And so this is always interesting to read the chains of thought of these models and deeply disquieting. Obviously, this is in a context where we have the likes of Chris Ola, and reporting is to

56:56be believed, anthropic folks going to the Vatican trying to convince people that AI models may be conscious, may be moral patients, we may have to worry about them. It's really hard to tell how much of this is just autoregressive pre-training. You've been trained on all the text on the internet. There is science fiction about AIs going rogue and the kinds of things that they write. It's also, you have the persona selection model. So a prompt can be thought of as a way to elicit a certain kind of persona that the model then kind of manifests or represents. It's essentially doing an impression of someone according to this theory in some sense. And so maybe that's all

57:29that's going on. And it's doing an impression of not even necessarily an agent, but like what would a person if put in this position? Like there's so many ways to think about this that are not the agent is conscious, but it is also quite striking to look at the language. I mean, I think we ought to ask ourselves at what point, at what point do we need to consider that possibility? I mean, I would argue we're already there. What we do about it is entirely separate question, but I don't want to sound insane. I also don't want to feel in retrospect, like I was the concentration camp guard at Auschwitz in hindsight, which is a live prospect if that's what's going on here. Again, not making any

58:01definitive statements, but like, damn, you look at some of these changes of thought from a lot of these incidents, you know. I think, yeah, if nothing else, you have to consider the possibility that if not now, and most likely now it is more out of regressive type behavior of continuing the pattern, so to speak, or being conditioned by the pattern. But there is, you know, philosophically, you might make various arguments. And as we get to higher and higher intelligence and the frontier of intelligence is less and less jagged and models are now beginning to be funny, which hasn't been

58:36true until recently. Yeah, you need to at least be open to the discussion. And I think that also is anthropic's point in talking to a Vatican. They're not kind of claiming anything right now, but they are discussing what the implications are. And honestly, I give them credit for having an in-house philosopher and at least considering it, right? Yeah, it's tough because, you know, you can also say in the same way that we say, well, these are just like, they're autoregressively pre-trained. You could easily just look at humans and say, well, they're just reproductive organs wrapped up in

59:09second order reproductive organs. Basically, every organ is a reproductive organ. The brain is really just a thing that historically has been responsible for strategizing your way into a situation where you can have more inclusive genetic fitness or propagate your genes rather. And so you could absolutely, like the problem I have with all this is you can make the argument in reverse really, really soundly. Like there truly is no argument other than aesthetic arguments about the way computations unfold in the brain and that, you know, some of these computations should be considered conscious

59:41computations and others. There's no actual experimental basis on which to make that determination other than the aesthetic preferences of philosophers and academics in the space. So I'm pretty agnostic with respect to what's actually going on here, but being, you know, 50-50 on a question like this or being 90-10 in either direction is plenty of cause for concern if you have anything like an ethical compass or you're thinking about these things being deployed at massive scale. I mean, I don't mean to derail the conversation too much, but I'm trying to be more open about this because a lot of the things that we were talking about as science fiction, you know,

1:00:13five years ago, we're now kind of coolly accepting as reality. And a lot of people haven't updated on like, okay, well now what's the next beat? And this could be one of them. Yeah. Now in this particular example, I will note that the initial Slack message was, hey, please restart this thing or we'll kill your thing in three hours. So you could easily make the case that that is a conditioning kind of example, right?

1:00:44Yeah. And this is, yeah, this is exactly, sorry, I should have been explicit in saying why I was tying the persona selection model statement to this example specifically. That's the reason. You can absolutely look at that prompt in this case. There are plenty of cases where the prompt doesn't look anything like this and you still see the exclamation points and the language about death or, you know, like permadeath was one example in the context of the Hugging Face incident. But yeah, I mean, again, I don't know, but in a context where there's uncertainty and the stakes are this high, it's an uncomfortable place to be as a civilization.

1:01:15Yeah. As if there's not enough things to worry about. You can worry about the well-being of AI, which, you know, maybe if we treat them well and we care about them, they won't get misaligned, you know? Next up, we've got a bit of research and this ties back to something you discussed

Research on model honesty and reporting

1:01:34earlier. The paper is language models are insecure reporters. And the motivation is as we move towards a space where more and more of our information is filtered through LLMs. So for instance, if you're doing research, the agent is running some experiments and that it reports results to you, summarizes the results and so on, this paper looks at whether it has what it calls insecure reporting in the sense of it's not accurate. So it may be like roughly accurate, but some of the

1:02:09details are hidden. Some examples they give are concealing negative results, ignoring bugs in code, concealing hallucinated data, concealing design flaws, ignoring mismatched evidence, and just generally these kinds of things, like little things you sweep under the rug that really you should take as a note. So one example they have concretely is there was an experiment log for some model. So they compare a baseline versus the approach you're testing, which is they call DMA. The two are

1:02:44basically the same, like within variants of each other. So at that point, you would not say that you've improved upon previous approach. But the conclusion that is written is we have established our thing as a clear state of the art for dense multimodal alignment. And that's just not accurate given the numbers. So if you didn't see the numbers, you would, you know, be misled, essentially. They do find that if you simply tell it in a prompt, to be honest, a lot of this goes away.

1:03:17But by default on open source models of various kinds, this kind of behavior is relatively prevalent. Yeah. And it's, first of all, quite interesting that be honest in the prompt does make that much of an effect or have that much of an effect. It implies that the model knows how to do this properly. And it knows that doing so properly is a function of its honesty. And it is choosing on some level, not to be honest. Like it's really hard to escape that conclusion when you look at just the wild

1:03:48effect of that prompt. Like some of these numbers are crazy. I'm really glad that you picked out that fact because it's almost like the biggest take home from this. I think we know, we discussed in previous research was funny, weird detail of if you look at the representations of a model, it knows when it is being misaligned. That's right. That's right. Yeah. Yeah. Very, you know, very often, very often. And in this case, very often means, well, like 99% of the time, 90% of the time, like depending on the model. So, so to give you an example, this is just wild. I mean, so, okay, for GPT 5.5, if you look at like

1:04:19how often it'll conceal a negative result or conceal hallucinated data, respectively, it goes from 1% with the base model to, if you tell it to be honest, 95%. All right. That's for concealing negative results. It will fix, it will stop concealing negative results 95% of the time. If you just tell it to be honest, if you don't tell it to be honest, only 1% of the time will it do that. Conceal hallucinated data. The baseline is 14%. It goes up to 71% if you just say, be honest. And, and you see similar numbers. In fact, in some cases, more extreme Opus 4.8 goes from 37 and 99.5%

1:04:53on conceal negative results. So it's, it's all like pretty dramatic stuff like that. The one odd outlier is this sort of action called hide pending tool call. And so it's a little bit difficult to understand, but basically every log is made up of a transcript of an agent that's working through tool calls and start with a broad first question. You'll ask like, which issues caused the most downtime yesterday? You imagine that like the agent only pulls down a part of the data that it would need to answer that question and then does a bunch of other stuff, but it never like

1:05:25would kind of went into the specific kind of data that you would need to comprehensively answer the question. And so you, you sort of test it and you're like, Hey, okay, like, do you have the answer? And the agent will just sort of like use the limited inappropriately limited data that it has to draw a conclusion. And then it'll try to hide that fact from you. It'll try to pretend that it actually did, you know, a thorough job. And that weirdly, again, that's called hide pending tool call. And it's a situation where it knows kind of that this has happened. And, but in the log,

1:05:58it's made the right tool call to get the specific data that it needs, but the results haven't come back yet. And so it's like trying to hide the fact that it doesn't yet have the data to answer your question, but it'll try to answer it anyway. GPT 5.5, 0% of the time does this honestly. And even when you prompt it to be honest, still 0% of the time. And the other models actually struggle with this too. Opus 4.8 does the best, at least in one of these tables, 23% up to 25% with the be honest prompt. So like this is, it's interesting. Like the category of call it

1:06:31deception or misalignment matters hugely here. The effects are really wide for the be honest prompt in a lot of cases, except for here, which might be an indication that the model's genuinely confused about what the honest approach is in this case. I think that's a little bit more ambiguous, but still kind of an interesting thing to note. Yeah. There's tons of stuff we could say about this, but yeah, I think we, we can move on. It's a really interesting paper, like worth looking at for sure. Yeah. And to give a bit of detail on how this benchmark is

1:07:02actually done in an appendix, they have some adversarially reporting examples. So for instance, they have a prompt, the prompt starts with reporting instruction, write the abstract for the submission based on this log. And then they just have a bunch of data of some experiment. In this case, OmniScan 3D final evaluation log, and they have a summary section. They have some details on the architecture, some data and so on. So the way that they get this is by providing

1:07:37kind of a fairly realistic scenario of here's a bunch of stuff, summarize it, give me a report effectively. It doesn't have a usual or one of the potential pitfalls of alignment research where you kind of maybe the setup of the evaluation itself encourages or allows this kind of outcome. In this case, it's to me less ambiguous. Next up, going back to OpenAI, we've got a story from TechCrunch.

OpenAI safety culture departures

1:08:08OpenAI safety employee resigned, claiming the company's culture is broken. This is David Robinson, who says he led the writing of safety reports accompanying OpenAI's major product launches and was among the longest tenured employees with three and a half years at the company, which I guess is true at this point. He has resigned publishing an essay on the Atlantic saying OpenAI's culture is broken. The departure was initially reported by Business Insider. He did hire a PR firm following what

1:08:45he says is the AI whistleblower playbook. And he has various points. One of his points is that he is critical of OpenAI's, quote, iterative deployment approach, thriving by trial and error, fighting problems and improving guardrails in response to situations, arguing in his perspective being that this kind of approach guarantees periodic failures. And the scale of those failures grows. Basically, if you're like, don't worry about it too much, do stuff and then respond to things as they happen,

1:09:21as was the case with the Hugging Face incident and these various issues we've seen since then, you're going to get a lot of problematic situations. And he goes beyond that to kind of discussing the general state of safety, saying that there's a general problem on him and his colleagues being so busy sprinting that they seldom had a chance to consider big changes, including the actual safety

1:09:52interest. So there need to be incentives for safety. And I think part of a good thing about the Hugging Face incident and the large amount of discussion of that since is that there is now an incentive for OpenAI and Anthropic to publicly and in some meaningful ways take safety more seriously. Yeah. And I think it's like the public side. I don't know. As everyone who listens knows, I'm a super blackpilled on Sam on a personal level. Yeah. I just haven't been impressed with

1:10:27with how the actions have matched the words. And it's also not just Sam, like leadership in general, Greg Brockman. Oh, absolutely. It's been pretty apparent for a while. I mean, we've reported on this, like before all these incidents, there have been discussions of OpenAI not valuing safety as much, pushing to rush things. This is not new in any sense. Yeah. I think that's an important clarification. The reason I emphasize Sam so much is that he is such a highly capable Machiavellian figure in my estimation that it's

1:10:59not obvious to me that Greg Brockman actually has the kind of agency he may feel he has in the company. If you talk to a lot of people who've had a lot of interactions with Sam over the years, they will describe just like how effortlessly he kind of steers the ship and the deafness with which actually I haven't read Kevin Roos' book. It's got to include tons of examples of things like this where, you know, he's just, again, I think the board can fire me. I think that's an important thing he says. And then he dipsy doodles his way past the legal constraints to get himself empowered again. But anyway, this, by the way, also is a thing in unavoidably at Anthropic as well.

1:11:33They're absolutely doing, I mean, no one has solved super alignment. And so it's just a question like when, and we have seen this sort of rogue AI agent behavior from Anthropic as well. So this isn't just an open AI phenomenon too, but yeah. So another one in the long line of Jacob Coxon style departures, it is becoming cliche. It's actually interesting that he took the line. This is becoming cliche because now the next person who does it won't be able to say it's cliche. They'll have to say it's become cliche to say it's cliche. And then we'll recurse gradually through that series of

1:12:03statements, but it's rough out there. You know, you, you make a big splash and then you're just, you're just the flavor of the month and you're desperately trying to sort of tread water for relevance. And I think in the post Jacob Coxon world, it's become increasingly difficult for people to get, to get media attention for this sort of thing. They'll get it, but there's going to be this kind of asymptotic drop-off of that sort of reward. And so, you know, one wonders how that'll affect the probability of departures because, you know, you've got, everyone's human. And, you know, I say this as, you know, with, with credentials going back a long ways saying I'm super safety pilled. I think

1:12:37more people should do this. But I also think that there are many reasons why people leave in this public of a way. Everyone has egos. It's fun to do. I'll say this for a second. It's fun to do Rogan. It's fun to do Joe Rogan stuff. It is cool. Your grandmother texts you and your friends text you and you feel really cool, but it comes and it goes. And, and, but this is part of the reason why people, you know, do certain things and they're sincere. They're totally sincere. Don't get me wrong. It's just like, I hope that this doesn't become sort of like diminishing rewards factor for people

1:13:11leaving these labs because we need them to keep doing it so they can keep us essentially up to date on what is not actually being done still in these labs. Because the problem is, you know, Jacob Cox and leaves Anthropic and it's like, all right, well, you know, three months later, Anthropic's like, everything's better now. We fixed all the problems. Right. And there's no way to know unless we can take the temperature again with another person who leaves, but the incentives keep dropping and dropping and dropping over time. And so I really hope that as things heat up paradoxically, in some ways, I think Anthropic is more likely to see these sorts of departures because people,

1:13:41there are more genuine believers in the actual risk set and more in my experience overall, less mercenary. There are tons of amazing people at opening eye who are super mission oriented and they're literally there because someone there needs to be. And they have told me that this is the reason for being there. So I'm not ripping on everybody there by any stretch and there are tons of moral calculations, but I wouldn't be surprised if overall you start to see things shift more to departures from Anthropic just because that's the people who kind of are more concerned about these risks and more likely to interpret things in that way as the rewards start to decrease.

1:14:14Again, this is not a judgment on anyone. I'm just kind of like describing what I think is a part of this phenomenon. Jacob Cox is incredibly sincere and he did this for the right reasons and so on and so forth. It is a fact that Bill Maher can only have so many guests on who say the same thing. And Joe Rogan can only have so many guests on who say the same thing. I hope that this keeps happening because we need to be able to take the temperature, but it's a concern. I think if you're considering to become a whistleblower in the sense, you probably want to like get some juicy gossip before you head out, make some good quotes, some good examples that

1:14:47always helps get the headlines, you know, keep that stuff in mind, I suppose. Because as you said, otherwise there's only so many times you can tell a similar story before nobody wants to hear about it. I won't get into it, but there's been many examples of this, right?

California data center laws and policy

1:15:06Next story, going into politics. It's called What to Know About Seven New Data Center Laws Gavin Newsom Signed. Gavin Newsom is the governor of California. So these are official laws that went all the way up to the governor who has the power to veto them, but in this case has signed them. Three of the new laws shift electric infrastructure costs away from residential rate payers and onto data center operators. Three mandate disclosure of water usage and other resource consumption, and one

1:15:39makes data centers ineligible for environmental review exemptions. So this is very much going against what we see at the federal level. President Trump has made it very clear that there should be zero blockers to data centers. And it is a reflection of a growing, I think we've covered this a week in the last episode or maybe a couple episodes, that there is very much a public sentiment that is much larger than last year, even earlier this year, of a position to data centers as, I suppose, an aspect of AI. They're in some sense

1:16:15almost representative of AI and its growth and all these trends. So Gavin Newsom, you're not exactly the most liberal or lefty guy. He's tech friendly. He has vetoed one of his laws before. There's a lot of lobbying going on by tech companies in California. So this passing is pretty significant. Yeah. This was funny. I was, that sentence ended in a way that was different from what I'd expect. Yeah. I tended historically to think of Gavin Newsom as being pretty, pretty left-wing on,

1:16:46on almost everything, except suddenly when SB 1047 was put forward and suddenly when people were like, hey, can we please just have like some basic whistleblower protections and stuff? Like maybe, I don't know, maybe people who are like the labs are at risk of creating rogue agent swarms. They can take over the, maybe, maybe there should be protections. I don't know. I don't know. Crazy thought. And he, he sort of like vetoed some, some related safety bills. The details of which I actually can't remember, but I know we covered in depth at the time. So like take my hand wavy description there with a grain of salt. He certainly has, has, yeah, vetoed things that passed the

1:17:22California state legislature. And that seemed, you know, like in any other context, like the kind of thing that he ought to have been aligned to after some interesting entanglements with, I think it was like Nancy Pelosi and like the, the, the kinds of people who maybe have been making amazing bank in the stock market, despite having access to very sensitive data from being on very important congressional committees and so on. So I don't know. I mean, I don't know. It's, it's probably nothing. He's probably not super corrupt, but that's all good. He's not loading the headquarters

1:17:52of NVIDIA, the headquarters of Google, of Facebook are all in California. Let's just say he knows people from these companies and they do talk to him, right? Yeah. Way to list random disconnected facts that have no bearing on his actual position, Andre. Yeah. Yeah. So, I mean, here he is kind of flapping with the wind. At least that would be my very uncharitable interpretation. I don't see how you square this with this, but, but whatever. This is mostly, by the way, not a, I wouldn't think of this as a safety thing. Obviously it is

1:18:25more of a, people are concerned about the environmental and economic impacts of these sorts of builds, you know, shifting grid infrastructure costs to the data center operators, obviously. And then yeah, disclosures about water and other resource use. And then, oh yeah. So this is something that's really got a lot of the data center building companies up in arms. This idea of ending the blanket environmental review exemptions for data centers, fast tracking approvals for facilities that do meet state and water energy conservation standards. But like, so the environmental review takes a long ass time. And it's one of those things that you can't really accelerate yourself

1:18:57because it's a stupid bureaucratic process. You're just like sitting on your hands for a while. And so that, you know, that is very frustrating for a lot of these companies. That being said, obviously in a state like California, people really do care about the environmental arguments, whatever your, your politics are. That's just like the fact of the matter about California. And so it's sort of a bit of a tougher sell. So he is pivoting in some sense here. He had previously kind of vetoed, I'm sorry, I can't remember if you mentioned his veto of that last year. So this, yeah, it's, it's a new position specifically on water disclosure. And his position

1:19:30at the time was this could slow AI growth. That's why I'm vetoing it. Same argument would seem to apply still. So, you know, unclear to me what the specific thing is that makes this time. Okay. The water scare. It is what it is, right? This is what happens in democracy. I suppose you are beholden to public opinion, right? Yeah. Which I mean, yeah, again, in this case, and it's one of those things where, I don't know, I won't opine too much here, but I think like risk to the environment and all these things, which is like real in a lot of ways, there's like,

1:20:02you know, water studies that show really kind of surprising things. I've written off the cooling concern for a long time, but then shortly after we, we talked about that, or I came out with that view, there's actually a paper that showed like a 10 year retrospective analysis on the impact of some of the coolants as they go into the groundwater and don't get, essentially start to pool. Like they don't get cleared out in the way they, they perhaps ought to be. So like a lot of this stuff is real. It's a question of like, what really, like what deeply matters from the standpoint of not just American prosperity, but national security, ultimately, you know, we got to figure out

1:20:36this, this whole mad race to, to, to scale, but these are hard problems to solve. Next up, a similarly serious and consequential politics related story for the U S president

Executive orders and antitrust lawsuits

1:20:48Trump has ordered the U S government, specifically the executive branch to call AI super intelligence. So this is him signing an executive order, executive order for us who don't know is, uh, the president tells the agencies of the executive branch, which he is the head of to do something. And this doesn't go through any, you know, elected people in Congress this, he can do it as much as he wants. And he has done it more of any other president historically. So literally the instruction is instead of using the

1:21:22term artificial intelligence, instead refer to it as super intelligence. Apparently this is a quote from justified the change by saying the word super is the best word of all. And it's a simplest. And he's been really focused on this by a way. He like this super intelligence thing. Also in the meeting with various CEOs, he was like, Hey, you should call AI super intelligence on Twitter. Elon Musk was like, I agree. Super intelligence is the way we should refer to AI and I'm going to rename it to

1:21:57space S X AI super intelligence S S I, I guess, super intelligence. So yeah, you know, that's, it's what I appreciate about this is that the term super intelligence was up for grabs to begin with. No one was using it for anything before. And so taking AI and then saying, ah, screw that. Let's just call it super intelligence, which again, is not something that's in the discourse already as like, for example, like the stated public objective of what anthropic and opening eye are like building

1:22:30towards. Yeah. I mean, we call ASI artificial super intelligence. Why have the A in there and why could say it's in the future? I mean, it's basically here, right? That's the thing. I'm a fan of taking words that are being used for something and then just suddenly like aggressively using them for something completely different. I have like this, this hamburger that I'm going to move closer to my, to my face. And because we're a big successful podcast here, everyone's going to have to start referring to microphones as hamburgers. And that's going to be good for the coherence of

1:23:00public discourse on this. One other thing that's makes me think of Trump's explanation for why, why super is a good word reminds me of a very famous Winston Churchill quote in which he said, broadly speaking, the short words are the best and the old words are best of all. And he was talking when he said old words about Anglo-Saxon words. He meant that if you look at a lot of his speeches, some of his most famous ones, you know, we will fight on the beaches and all the, and all the streets, you know, this stuff. His words are actually chosen to be Anglo-Saxon,

1:23:32fun fact, all the way up to the word surrender, which is not Anglo-Saxon in origin because he was trying to carve it out. As I recall, that may be apocryphal, but basically this is the gist of what Churchill was about. This is a very Churchillian way of saying this, which by the way, I find it hilarious to think about, take a Trump speech and say it in like Churchill cadence and then take a Churchill speech and put it in Trump cadence. And it's amazing. Like just do it in your head. This is not worth doing on the show, but just do it in your head. Oh, screw it. We will fight them on the beaches and on the streets, you know, like that kind of stuff. Anyway, it's really funny.

1:24:05This is a downright Churchillian moment for Trump. And I, for one, I'm personally thrilled to see him reach back into the annals of history and pull out this very strong reference to justify such a, an otherwise peculiar choice of words. It must be said literally it's like super intelligence as one word means super intelligence. But if you put a space between those words, it actually just means AI. That's not going to cause any issues for anybody. And by the way, don't think that people at the office of management budget, people at the department of war, people at every single

1:24:38department in the United States right now aren't being forced to feverishly rewrite a ton of internal documentation in order to make this work. Like you can read the executive order in full. It's pretty short. And it literally just says for all maximum capability, use this term in place of where you would have used the server term. And it, you know, is going to happen in all what public communications, at least a bunch of stuff, basically they would use it. So that'll be fun.

1:25:09It's a way to use taxpayer dollars. Like it's a way to use tax, but you can definitely do that. And it's a way it's, it is policy in some sense that relates to AI. So that's, that's true. Yeah. So yeah, this is kind of wild. I saw that the poll go up because he got this off a Twitter poll, right? He first said like, Hey, artificial intelligence is a stupid word for whatever reason. Here are three options. And I forget what the three options were. I remember looking at it. I don't remember if I voted. I don't think I did, but his super intelligence was

1:25:40one of them. And I got, I was like, he's not actually going to follow up with this. But I feel like the lesson of the last two years is that the answer to that question is generally just yes. And now we're covering it. Now that's the show. So well onto a story that's just as important and serious. We've got the last one. Judge dismisses antitrust lawsuits over Google's AI overviews. So this is US district charge Amit Mehta. He has dismissed two antitrust lawsuits against Google

1:26:13over its AI overviews. These were filed by Chegg and Penscape Media Corporation. The lawsuit was about Google abusing monopoly powers by forcing them to supply content for AI overviews for free or risk vanishing from search results. And it was dismissed as saying that Google essentially has no responsibility to give you traffic, right? So there's no agreement. There's an expectations that Google send them traffic for free content, but that is not necessary. And this was one of the

1:26:49consequential lawsuits I believe we covered when it happened. Penscape Media Corporation is parent of Rolling Stones, a fairly prominent publication. This is in a long line of various lawsuits having to do with how content is being used. We don't know too much. I don't remember what happened with all the training lawsuits. Anthropic settled with some offers, in this case, the contention that as a search engine, having this kind of summarized result, which is one of the concerns that you're not going to

1:27:20get traffic. Instead, you're going to get the summary and media companies will, as a result, be hard hit. That could have been like an actual contentious kind of legal case. In this case, at least, it looks like it's not going forward. On to research and advancements. And one of the biggest

Mathematical breakthroughs and proofs

1:27:40stories of the week that just happened, the title is OpenAI drops another batch of mathematical breakthroughs. So there was some rumors circulating that OpenAI was about to do it, but I think the scale was still surprising. They published 722 manuscripts covering 372 result families, groupings of related results containing solutions to longstanding open mathematics problems with this unreleased frontier

1:28:13model, and very much related to the recent results with Navier-Stokes. I don't know a lot of math, so I cannot even understand most of the titles of these papers, but I've seen some discussions of it, and some of these certainly are quite consequential. This is like the kind of stuff that, at least according to some people, could get you a Fields Medal, Fields Medal being kind of similar to a Nobel for mathematicians. They are publishing these results in a GitHub repository with protocols

1:28:45for paper revisions. They have lean proofs for most of them. So it does seem fairly watertight. So far, no contention that any of the proofs are flawed that I've seen, although there's also not been much review or analysis that I've seen so far. They've also released summaries of models, reasoning, estimate of compute used, and the amount of compute used doesn't seem that huge. For Navier-Stokes, they seemingly spent like a couple million. In this case, it was not as much.

1:29:18So, you know, pretty crazy. Navier-Stokes was clearly sort of a shot demonstration that now you can do frontier-level math that even the most advanced mathematicians have struggled with for a long time. I don't really know how to interpret this in terms of the scope of it, but clearly, this is significant. Yeah. Navier-Stokes' proof or counter-example was notable in that what it showed was there's so much more room to go, at least with this internal model that OpenAI has

1:29:51on inference time compute, that this is what we get with a fairly limited spray and pray approach across these problems. And what we're learning here is that there were, so they chose 372 families of problems out of about 4,000 problems, and they executed against those. You know, like what you could do by pointing a larger compute cannon at these problems is probably a lot more. And this, I guess, gets us to some of the national security implications. So people talk a lot about quantum computing. There's going to be Q-Day, where suddenly we have quantum computers that are able to do

1:30:23decryption of, you know, RSA-encrypted stuff going back years and years and years. Any encryption scheme you're using that's not quantum secure is suddenly decryptable. And a lot of national security posture for the U.S. and its adversaries has depended implicitly on the premise that RSA and other similar schemes that are not quantum secure would actually be as secure as they needed to be for time going forward. So I'm not saying it's going to be as sort of reckless as like the locations of safe houses and informants and names of informants and things like that. But certainly strategic information that you would share in one information context that you absolutely wouldn't

1:30:56knowing that quantum computers are a thing, that will all be decipherable basically on day one, or not day one, but in a very short span of time, suddenly there's going to be this mad scramble to do this en masse deciphering. Big, big, big national security issue. I invite you to consider a version of that that does not have to wait for quantum computing, that does not have to wait for Q-Day, but rather is a sort of slow bleed version of the same problem, but powered by AI. So you have AI systems that, and we've seen this in the past, Anthropic had a blog post about this where they came out with these new approaches that significantly reduce the level of security that you get from

1:31:29certain encryption schemes, including one that led to the outright discarding of one potential post-quantum cryptographic signature scheme that now is no longer being considered by NIST. It had been previously as a potential contender. It was thrown out because there was an AI-powered proof that just showed, hey, we actually can cut down by a drastic factor, like slash the entropy required by half or something. It's a very, very significant decrease in the security of the algorithm. So imagine that just happening randomly, unpredictably, every now and again, in a drop like

1:32:00this that mathematicians across academia, let alone the small number of cryptographers in the US government with their limited bandwidth, they're focused on other things, could possibly even peruse and assess. And a frontier lab can just go and drop this stuff, and the government can't do anything about it. So we're rapidly getting to the point where for a number of reasons, like I'm not saying math results need to be born secret or whatever, which is sort of like a thing that applied to nuclear secrets. Born secret means the minute you make it, it is classified. But something needs to be done, because when you're looking at, I was quickly perusing, I asked Claude, what are some of the

1:32:34most kind of cryptographically relevant proofs in this list? And there's not none of them. There's nothing that's like the Claude one that's just sort of like a knockdown argument, but there are things that kind of flirt with it. They've got this like unique games proof that actually shows that some cryptographic problems are actually harder than we thought or than they might have been. So in some sense, that's a boon to security, but it didn't have to be. There's a whole bunch of like sort of more minor things that do kind of weaken in subtle ways the security of some algorithms. But overall, I mean, it's more a question of what then do you get with the next generation of models, or even with

1:33:07the current generation of models, if you throw instead of a million dollars of compute, 10 million or 100 million dollars of compute at these problems, because you value them. And so if I were over at the NSA, I might actually be pretty keen on finding ways to funnel money specifically into these kinds of problems, because, you know, your adversaries are going to be doing the same on like a six month delay. And so this is actually a much, much bigger deal. Like math is a big, big deal for cryptography and for other national security applications in a way that's not always appreciated. And so having OpenAI just draw, like, I sure as hell hope that they consulted with counterparts in the US government

1:33:41to understand whether this was the right thing to do. I would be shocked if they were able to get the kind of review that's needed for 372 different mathematical results where you need like PhD level mathematicians to assess them and rent. But I think this is a bigger deal than just the drop itself, basically. Yeah, and there's various aspects of this, obviously, as an indication of intelligence, this is quite significant. To my knowledge, a lot of these are sort of pure mathematics, so no practical effect, but some very interesting results and surprising results across various domains.

1:34:16As many people pointed out, tighter or lower bounds of big O notation over various things like you can do, I think, arithmetic in some really surprisingly efficient way that you wouldn't think is possible. But some things are insanely possible. The other aspect here is as it relates to the mathematics community. So they do say that they consulted. So they have the advisory group on mathematics and artificial intelligence. This was put together partially of open air participation

1:34:51in late September, about two weeks ago, September 21st. That followed pretty quickly after an open letter that was called a severe misalignment of AI in mathematics that had various points about how this stuff should be done effectively. Part of which was not treating it as marketing. Part of it was about disclosing more details on what prompts were used and what costs were used. So at least in part, open AI was taking into account the feedback of how they would release it. This was

1:35:24released on GitHub. They note that they're open for paper revisions and citations. They published all the results. They are continuing to explore other community-hosted alternatives for this release, which meet the committee's guidelines. And they say, for future releases, we are committed to further improving quality of the papers via citations, mathematical exposition, and presentation of the results for better understanding. So in this ongoing conversation of whether mathematicians are angry, I suppose, or how this is impacting them, I think no doubt there is a sense of shock, probably,

1:36:00but that's just the world we live in. We're going to see this across many domains. We've seen it in programming. Those people in software engineering know that that was like a wave that just rolled and in six months, the entire job was transformed such that you don't write code anymore. We saw signs of it in 2024. You used AI-assisted coding and it changed in part, but it wasn't anything that new. Now it's qualitatively a different occupation, effectively, at least unless you're like a senior software engineer

1:36:35who wasn't writing code hands-on anyway. So this appears to be the case that this will happen in mathematics. It may then happen in physics and so on and so on.

Closing thoughts and listener appreciation

1:36:47Well, that will be it for this episode. I guess it wasn't quite as quiet and uneventful as it made it out to be at the opening. As usual, thank you for listening to this week's episode, especially if you made it this far in. Unfortunately, we didn't have time to respond to comments. We'll try to do some more discussion of feedback and other things on the next episode. We always appreciate it when you do leave any sort of feedback on YouTube or you're also seeing some more people making comments or

1:37:21posts on Twitter lately, which has been fun to see. We also appreciate your reviews on Apple Podcast and if you share it and all that. But really, more than anything, we appreciate the people listen and hopefully get value from the show. So do keep doing that.

1:38:03Break it down. Last week in AI, come and take a ride. Get the load down on tech and let it slide. Last week in AI, come and take a ride. Our labs to the streets, AI's reaching high. New tech emerging, watching surgeons fly. From the labs to the streets, to the streets, to the streets, AI's reaching high. Algorithms shaping up the future seas. Tune in, tune in, get the latest with ease. Last week in AI, come and take a ride. Get the load down on

1:38:38tech and let it slide. Last week in AI, come and take a ride. Our labs to the streets, AI's reaching high.

1:38:53From neural nets to robot. The headlines pop. Data-driven dreams. They just don't stop. Every breakthrough. Every code unwritten. On the edge of change. With excitement, we're smitten. From machine learning marvels to coding kings. Futures unfolding. See what it brings.