Claude Code’s Next Era — Thariq Shihipar, Anthropic

State of Agentic Coding

0:00We're here in the studio with our friend Tarek from Anthropic. And I guess generally the Cloud Code, there's so much sort of merging of boundaries. And you've been so on top of everything since you joined Anthropic. You have been early to Cloud Code itself, but then also, and you've told that story in other podcasts. And you've also been talking about seeing like an agent. Most recently you did the top AIE.

0:30World's Fair talk, Field Guide to Fable, which obviously you guys launched Fable, so that's cheating.

0:35And most recently also launching Cloud Tag. And we're also going to be talking about pacing on Frontier. There's a lot going on in Anthropic. I guess top of the question is, what's it like being Anthropic when there's so much going on? I think that it is like, I think you can get whiplash sometimes. I think like going, when I joined Anthropic, I joined because of Cloud Code. Like Cloud Code had just come out and I was like, this is so good. And Opus 4 to me was like, like, just, I could not imagine like how good it was, you know what I mean?

1:09And that was like a real moment for me. But I was like trying to convince like my startup friends basically to use Agenta Coding. And they're like, oh no, like our engineers don't think it's good enough or something. And I was like, that's insane. And now you like fast forward, you know, 12 months, less. And like, it's just like, yeah, the default way that everyone codes, right? And I think that like, just having to go from like selling it to like now, you know, teaching people how to be, you know, make the most use of it and be more efficient and things like that is just like a big, you know, like big change.

1:43And yeah, I think like, it's just hard to stay on top of everything as a human, you know, like things happen so, so fast. You just threw more agents at it. I mean, yeah, like that's actually like the agentic stuff scales much better than the like human stuff where it's like, oh, like there are three things happening right now and like they're all emergencies. And like, how do you like, you know, respond to it? What do you split your time on? You do a lot of technical writing, engineering work. Yeah, so I think that like when I joined the Cloud Code team, I wanted to teach people how to use Cloud Code basically.

2:15And I think that like that has been something that like, I thought like maybe I would spend a little bit of time on it or like I, you know, like do, I was spending some time on the agent SDK first and I wasn't exactly sure like, you know, how the bitter lesson would go, you know, when it comes to like harnesses, right? Like, I think sometimes we're like, oh, like what's after Cloud Code, you know what I mean? And so initially I was like, I just want to teach people how to use Cloud Code and make it easier to use Cloud Code. And I think that is just like, as the harnesses have gone better and better, that's like the dominant problem now is like,

2:45how do you use the agents, right? Like it's like such a high skill expression thing. So I do that and then I do engineering work. I give talks, but I think like when I'm doing engineering work, my goal is to take that feedback that we get from users and also like then be able to talk about like, hey, how to use Cloud Code to do engineering. So there's kind of like a good loop there. Yeah, for listeners, we'll attach the talk that you did with Sarah for the DevWriters meetup, which we talked a little bit about, well, first you do the work and then you talk about the work.

3:16Right. Something like that. Yeah, yeah, yeah. Sow and reap or? Yeah, reap and sow and reap. Something like that. Something like that. Yeah, so, and then just to preview a little bit, we are going to talk about the evolution of the harness. Yes, it has come a long way from just being a CLI. We're going to talk about Cloud Mods, which is starting to leak today because you couldn't keep it secret. Yeah, yeah, yeah. Basically, yeah, yeah, yeah. Yeah, there's a lot there. I think you started off with like adding Ask User Question tool, which people love and hate actually.

3:48Yeah, yeah, yeah. Like I actually thought it was like very innovative and then now I have like my own version. You have your interview me version. Yeah, yeah, yeah. And yeah, everyone just has like their own stuff. And again, it no longer matters because now you're supposed to write prompts that create other prompts and loops and all these things. Sure, yeah.

Elicitation and Unknowns

4:06So what's the state of the art today? Like what are people, what are you like telling people to do today? Yeah, Ask User Question was the first time that the model was good at elicitation. You know, I think this was like kind of an emergent behavior that I like, you know, wanted to see if the models could do. I have like kind of like a human computer interaction background. And so I like, you know, did that in undergrad and grad school. And so this was like, I think it's kind of like human agent interaction to me, like, you know, trying to figure out like how can the agent communicate with you and extract, you know, the requirements.

4:39Right. I think that like one of the things about like that's difficult as Cloud Code has gone broader and broader is that everyone has like their own way of using it. And it's actually very hard to like change the default behavior. So, for example, like if someone asks you Cloud Code to do something, sometimes they just want them to do the work because they're like maybe a very good prompter. And sometimes they want like actually are not good at prompting, you know what I mean? And you need like the agent needs to like clarify, you know.

5:10And so that's like a good split. And to ask you the question tool sort of like splits along that side where like, do you feel like you're good enough to instruct the agent as it is? Or is the agent able to like, does the agent need to like pull out more requirements and like collaborate with you more and really understand your preferences? I on the whole believe that pretty much everyone is more on the ladder than the former. That they like have more ambiguity and they know less than they want, than they like think they know about the problem. But like it's like an interface design problem to make that easy, you know what I mean?

5:46And so like if you're designing a problem or if you're going through a problem like, you know, things like what's the schema or like what's the call stack and things like that are really important. You know, like the details and the design are important. Ideally, you want to figure out some of these like hard problems ahead of time before starting implementation. And yeah, that's why they call like unknowns, right? And so I think that this will forever be like a skill in agent decoding is like figuring out your unknown. So like, because even if the model is like super intelligent, it like needs to know what you want, you know?

6:18And like you have preferences, like you need to sort of like pull that out. And so that's like, I think how I'm, what I'm pushing. The question then is like, how does the agent interact with you? And I think that has been HTML has been like the big way of doing that. And we've recently added artifacts, right? And artifacts, I actually think we've done a bad job of like, or like I've done a bad job of like explaining how to use them fully. We have a lot of property capabilities. They have a database associated with them, you know?

6:49And so every artifact can store and write persistent data. They can like feed back into Claude, you know? And so like one thing that, you know, like people are not doing yet that I'm trying to like encourage is like this idea of a dashboard artifact. So you have like Claude working on a project long-term, maybe it's like a Kanban or something. It can store that Kanban data in its database. Multiple Claude's can access that data via like the artifact MCP. And like that artifact can like talk to those Claude's as well.

7:22And so like we're basically building the primitives for you to be able to have this like generative interface via artifacts that will like let you surface more of that rich detail from the agents. And I think that like almost everything with agents right now is like this problem of like you think you know what you want, but you don't really know what you want. And like the agents need a lot of detail and collaborating with them in the loop is really important. And so artifacts are like the like way that we're trying to evolve there.

7:55But there's a lot of work to do because it's so much more complicated than like a multiple choice question, you know. There's a lot more like detail in terms of like diagrams and code snippets and schemas or like whatever it is for that problem. But like artifacts is like the more AGI-pilled way of like, you know, doing ASCII as a question basically. So yeah, I think one thing that's unclear to me about these the artifact stuff is like what feedback should go in through the artifact and what feedback should go through a Claude chat because the more AGI-pilled one is to just feed everything to the Claude.

8:28I think the more AGI-pilled one is to go through the artifact. Like and I think that like we sort of imagine in the limit. I think that artifacts will be your interface into the harness, you know, you can like comment on this like live like document of your plan of the work. You can see maybe like multiple agents and different agents are doing this and that artifact is built for the current work that you're doing, right? And so like each one has like slightly different. I think we're still like getting there from like an infrastructure perspective. But yeah, I think like on-the-fly interface for your harness is probably where things are headed.

9:02Is there a version of it that's an abstraction from CLI or chat? Because right now a lot of it is, okay, you're interfacing with Claude code, you're having HTML given back for a mock-up, it's pretty rich, there's diagrams, artifacts are ways to connect these together. Why not just do everything that way? Then it becomes like separating out like where is the inference happening, where is the intelligence happening, where is the work happening? You know, like I think this is kind of like difference between like, or like some of the distinction between local and cloud, right?

9:34And so I think right now if you use Claude code, it's like local and like you can spin off remote control, for example, to get some cloud behavior, or you can spin off Claude code in the cloud, right? We're moving towards a place where instead of Claude's, like you message a local Claude, it starts a session locally and it executes, to more like you have a Claude that you message that's in the cloud that's running. It can run like local or like cloud sessions, this is kind of how Claude tag works, but like over time we'll add like local hands as well.

10:08And so like local hands will be the ability for that agent to access your computer if it's online, you know, and be able to like work there. And so it can spin off many different sub-agents, it can like, those sub-agents can communicate with each other, and that's where the artifact comes in to display all of that work, basically. So you can imagine like the, you're separating out these things. So there's like the surface UI display that's in the artifact and hosted somewhere and has a database and everything. There is the inference intelligence, right, that's happening on the cloud, and you don't have to worry about shutting off your computer or whatever, right?

10:43And then there's the like hands kind of like, and it can be local, it can be in like a remote sandbox or wherever you need your work to be done. That's like unpackaging like the Claude code experience right now. We're like, right now it all happens in one place, right?

Multiplayer Workflows

10:59How do you see like the multiplayer side of that? So say teams want to work in this way, right now it's very individual, but how do you see the future of multiplayer? Like right now, I guess there's Claude Tag, which is a version, but... We're launching projects, and so projects is the, like this abstraction that's kind of like Claude Tag, but on our Claude products, right? So you can message it, and like it will do the Claude Tag-like stuff like spinning off sub-agents. And so we think with multiplayer, like Claude Tag is like a little bit more native multiplayer, because it's just like in your Slack, and the permissions are all figured out and stuff like that.

11:40But I do think multiplayer is like an important part of the story, and like that will need to get tied together more. Like you can imagine how complicated it gets when you're like, oh, you have hands, but now you have other hands in other people's computers too, and like you need to like permission them, or like you have like your MCP and someone else's MCP, and how do you figure out how to use them, right? It gets like quite complicated. And Claude Tag does a good job of like sanding down all of these issues, right? So that like when you have, yeah, Google Docs, how does it access Google Docs, right? Like it accesses through the shared Claude MCP, or it can access through your local credentials as well if it doesn't have access.

12:17But yeah, I think Claude Tag is our multiplayer product, and it's really useful for these like things that are inherently multiplayer. Like, okay, like on-call, for example, incidents are inherently multiplayer. You want to tag Claude, you want multiple people to log in, you want to be able to find context. I think whenever I'm like working on something and I want like privacy or security, or like I want other people to review it, you know, it's really nice to like, I'll have a channel per project, and I'll like at legal, for example, be like, hey, like, I want to ship this. Can you like, like, here's the Claude knows everything, you know, just chat with it.

12:52And that way legal gets precise answers, you know, on like what exactly is shipping into the code, and I don't need to be in the loop, right? So I think like multiplayer is getting like more and more like, yeah, everyone can participate with Claude. I think Claude Tag is like that product, and like projects will start off single player and will like, you know, expand. I think there's a question about like, maybe dual questions about identity and the unit of isolation.

13:21Claude Tag, you specifically chose to make it its own identity, which is like a controversial choice. There's other ways to do it. Cloud projects, probably it sounds like, you know, if it's anything like ChatGPT projects, it is, you know, the isolation is that artifacts, that cloud instance, everyone's collaborating on this. It sounds like, you know, it should be like, if you're collaborating on legal on a thing, like that channel should be a project, right? Like it's not yet, but that's the natural next step.

13:53Yeah, I mean, like I think in Claude Tag, it's effectively, like Claude Tag, you have to sort of do your own arrangement, basically. And so Claude Tag, yeah, each channel is like, you can name it as you want. And I name like each feature, basically, as a channel. But I think like there is some transfer, like it's unclear when there is transference, because let's say if you have a coworker who is tagged on all these things, yes, there is transfer, because it's the same person. But with Claude, it's unclear if it's like necessarily like, well, no, you don't know any, you don't know about the other stuff.

14:24You should only use this stuff. It's like the tip of the iceberg meme, right, where you can like, this is what we spent so much time on, basically, is like, there is like infinite surface area of like, okay, you want Claude's to, not infinite, but like there's like surface area, a lot of like, surface area to figure out of like permissions and visibility and like, you know, like, how can you let Claude operate as well as you can, as safely as you can, you know? And obviously, this is very important to us, because like, you know, like, like security for our code base is very, very important.

14:56And so we've put a lot of time into this. Yeah, there's so many like edge cases you can figure out where it's like, oh, like, yeah, this Claude in this channel has different permissions, but it can message another channel, and can't it exfiltrate data that way? Or like, can you like, what if it uses your MCP and then messages someone else? Like, there's like so much, and we've like really put a lot of work into sanding it down. Yeah, yeah, lots of work. Okay, Fable? Fable, you wrote two good articles. I mean, you've written many good articles, but on, you know, Field Guide to Fable, building Claude Code.

15:26I'm curious from what you've seen, is there any common patterns that you see in like top users, adanthropic, externally? Like, what are best practices for getting the most out of Claude Code?

Prompting and Mental Models

15:37The like meta skill, I say is like prompting is like very important, you know, and like that, like, I think this is like not trivial to say, because I think a lot of people are like, oh, prompting doesn't matter. It's just like, I can just say a sentence and Claude will do it. And I think prompting is really this like, this, it's like public speaking, you know, like or writing or something. And for a specific audience, and that audience is Claude. And you need to like build a mental model of Claude and how it thinks and how it works, right?

16:07And so that's like the most important skill in working with Claude Code is like having this mental model, right, of Claude. And like what it can do well, what it can one shot, what it can't. And so, so many people, when you see prompting, they're just like, they're short prompts, but they have such a good mental model of Claude and of like the code base and things like that, that like it's effortless, you know what I mean? But it's like high skill ceiling. So like that work of like, you know, spending a lot of time prompting and building mental models of how, you know, an intuition for how the agents work is really important.

16:39And then I think like the next thing is like the unknown stuff we talked about earlier, where it's like being able to find out like your, what you don't know or what you haven't written down, learning about like different things. I think as Claude can do more and more things, the likelihood of you doing something out of distribution for you and like you have low domain knowledge on is very, very high, you know? And the more you can like learn the vocabulary to be able to prompt Claude, it becomes really important.

17:11And so like, I think the most important unknowns are the unknown unknowns where you're like, I just like don't even know that this exists, right? Yeah, exactly. I think that's like an illustration of like the map and the territory, right? Where you're like, okay, this is my prompt and the territory is like the actual like work that the agent needs to do, right? And if you are like very precise, you can give more precise things, right? So like, for example, in design, I'm not very precise. I'm not a designer. So I say like, you know, give me like eight different mock-ups. But if I was a designer, maybe I'd be like, oh, hey, here's some reference sites. Like I want this type of font and this type of like look to it.

17:45And here's like a few different components to like visualize here as a Figma MC board to bring in, you know, like. And so you can just be so much more precise with that language. And if you're not a designer, you just need to like try and learn the language basically or learn the unknown unknowns. And that's true of like everything, I think. Like the more like you can work with Claude to learn like how things work, the better your prompting will be. I think another good example of this is like game design, you know, like where a lot of people are like, oh, like I can vibe code a game now.

18:17And they're like, it's not fun. And like, it's just like the thing about game design is like every one of these choices has like a lot of variations. A lot of like craft to them. So it's like, oh, okay, like when you're making a flying game, the feel of the plane and the like, you know, way it responds to your controls has a lot of like, like, you know, a game designer would spend like days on that. Do you know what I mean? To me, that's what taste is, right? Like it is like from the possible space of 1000 mathematically valid answers.

18:49Here's the one that is the humans will like. Yes. I think with taste, I'm like torn on this word because I think you're right, but everyone has different definitions. And it sounds kind of like low skill or like elitist almost where you're like, oh, like there are certain people with taste. It's like taste is what I call taste. Yeah, yeah, yeah, exactly. These guys don't have taste. Yeah, exactly. Oh, like an engineer doesn't have taste. Like I, the like founder have taste, you know what I mean? And I think that's actually not true. Like I think like the engineers have a lot of taste for these particular like problems, you know?

19:20And I think everyone has taste for particular problems. I think like Jason Liu, like, like say, like in order to have taste, you have to eat, you know? And I really like that where it's like, okay, you have to like do a lot of things. You have to like iterate and figure out what you want, what you like. And, you know, like build that like domain vocabulary. And then when you're prompting, you're like synthesizing all of that for a lot. Isn't it annoying when someone else says it better than you? It's just like, fuck, I have to quote this guy forever.

19:51Having to quote Jason Liu forever? He's going to love this. So I get prompt. And sometimes it's actually not even that. Sometimes it's just intuitive, right? Like you don't realize you even want something till a model puts it out. And you're like, oh, this just feels immediately better, right? Yeah, yeah, exactly. One thing I go back and forth on is I feel like the way I prompt half the time, let's say I use voice. Guys have voice, other people have voice. That is the opposite. There's just like me rambling for like two minutes, pressing down the function key and then let go.

20:23And then like hopefully it figures it out. And oftentimes it does. But it's not as thoughtful as like a structured prompt with like, you know, well run communication as though it's a PRD or memo. Is that in line with how people do this? There's like basically bimodal prompting where there's some prompts where you spend a lot of time up front and other prompts you just dash it off. I don't think the voice is necessarily low. Like I think it's like more like how much information is in the prompt, you know, like the model can, like you can um and uh and like add some sentences and be like, oh, like actually I changed my mind like in the middle of the prompt.

20:57And it will be able to follow that perfectly. You know what I mean? So I think the like actual format of the text is less important, but then like the ability to like how much information is in it. Right. And I think for voice, a lot of times, you know, going back to like kind of human age and interaction and like for a lot of people, it's just way easier to talk than to like type, you know? And if that gets more information out of you, like that's better. At some level, it feels like just giving the model as much context. Yes. Over prompting before you kick off is a best practice.

21:28I don't know. A lot of the times, like when I was first trying out Fable, I spend a solid 30 minutes like really crafting a long prompt. This I think is a response of models running for longer and longer, right? It's still a little difficult to nudge them as they're in, in like, you know, in the loop. But I just like intuitively spend more time kicking off that first prompt and working with it a lot. And my personal opinion is that if I was a software engineer, if I was like, you know, just running my own startup, for example, I think I would mostly stick to a max 20x.

22:02You know what I mean? Like maybe verification and code review are kind of like separate things. But I think like what I see a lot of times is people hit rate limits when they're doing this sort of like, oh, like it did a lot of work and you're like, oh, I don't like this. Like, can you like undo this and redo it? And then you're like iterating on this like thing that the model could have done if you had like spent more upfront time or given it better context, you know? And instead, it's sort of like, you're like, nope, don't like that design. Try this or like you mess this up or something like that.

22:33And then that just eats up so much more of like, you know, your usage. And so that's like, I think maybe like a key like tip both for like efficiency as well, right? And yeah, I think like context and not just like context on like what the goal is, you know what I mean? It's good, right? Like, are you building a prototype or is it like a production thing? Like, where can you spend compute or where can you not spend compute? Like, I think you have to give the model permission or like not permission to do things sometimes where, you know, like it doesn't know intuitively how much you want to spend on this task, right?

23:04And you can use effort for this. So I'm working on a blog post about that where it's like, you know, if you want for like, we see the effort scales with basically the complexity of the task. So for security, effort gets like way more results. Like high effort versus like low effort gets like changes the evals a lot. But for software engineering, it doesn't change a huge amount because effort is mostly spent on the verification and the like edge case testing and things like that. And so like being able to like give the model that guidance of like, hey, this problem is something that I think I want you to spend a lot of time verifying and edge case testing.

Model Selection and Effort

23:43How about model in the mix? So, you know, there's Opus and Fable with effort. There's also Haiku in there. Yeah, yeah, yeah. It's not quite true yet, but it's very close where I think the frontier models will be Pareto dominant over like almost everything, you know, like maybe. And sometimes I think Opus might be Pareto dominant, you know what I mean? Like I think depending on like how things like shake out if it's like a newer version of Opus. But I think that like increasingly it's just going to be like the smart model is going to be able to like do the simple task for less tokens than the other models basically because of verification.

24:23With verification, in the limit, your model doesn't need to verify, right? If it's a perfect model, it just does the work once and it's like, okay, like I did it, you know? And increasingly with Fable, I'm like, I'm like, dude, you don't need to spin up Chromium and screenshot all of these things. Like I see it, like you did it, right? And so a lot of the higher effort, you spend more of those tokens verifying. But if you're working on simpler problems and a lot of software engineering is like, you know, like well, like in Fable, like low and mediums ability, it can spend less tokens verifying.

24:58And as the model gets smarter and smarter, they will just be able to like, all right, done, you know, like I can run the lint for sanity's sake. But like I like, you know, it lints, you know what I mean? Like you don't even need to do that. And that will be so much more token efficient than like the smaller models, basically. Is there a good practice on our side that we can use to see if we're using too much effort? Like I freaking hate wasting time on that kind of stuff. Yeah, yeah, yeah. I know what you mean. I think like, so in this blog post, my rough distribution is like code review and security should be like higher max, basically.

25:35And like software engineering. You just have recommended settings per domain. Yeah, I think like if you're doing like UI or something like that, like low and medium, I think is you're building like an API and you want to make sure like you cover enough edge cases, you know? And so I think building, like I said, that mental model of like how things work across these distributions is like, yeah, part of the job. This is more intuition driven or eval? Because I'm guessing this would change as well. He has evals. Yeah, yeah. So what I did in the blog post is I go over all of the terminal bench evals, basically.

26:07So there are like 70 problems and I show that like, okay, you know, like in the security problems, it does more. And then I also like look at some of the transcripts just in terms of like how, what does it answer? What does it forget or something? And a lot of times, this is another prompting tip I have is like asking it to make decision notes or implementation notes because in basically every eval problem that it faces, it thinks about the correct solution, you know, and decides not to do it.

26:39You know, it's like, oh, like, here's the answer. What if I did this? And then it's like, oh, probably not, you know, and then keeps going. And this is like the majority of the failures, you know what I mean? At like a higher max level. It's very rare that the model just doesn't know how to do something. If you just have these implementation notes, then you can review and you can be like, oh, actually, I want you to do this thing that you didn't do. The models are getting better at surfacing that overall. Like I see in the transcripts of Fable 5.1, like when it does this output, it will call out its decision making as well.

27:15But making this more explicit in the harness is better. And now we're, you know, allowing ways of you modifying the harness so you can like, you know, add some capabilities there. Yeah. Yeah. So I do want to call out two things that you mentioned that I think actually exist outside of prompting. One is actually like, let's call it the prompt that is so important that it shouldn't be in a prompt. It is actually in Claude MD or Agents MD, which is like goals, right? Like your situation, your goals, the things that you want, the thing.

27:45And then the second of all is the decision log or the experiment log or whatever log of traces that you might want to actually survive the current session to do those things. Those are like externalities that there's no standard. There's no, it's not like skills. It's not like MCP. There's no standard. It's just like, it's a markdown file. So first of all, is that right?

Prompting as Executive Communication

28:05Is Claude MD going away? You have a documented dislike of Agents MD, but you're going to do it. Yeah. Yeah. Okay. So I mean, Agents.md, yeah, like we're, we're going to do it. I think it's just like different models are very different from each other. You know what I mean? But I realized that it's like such a pain to like maintain different ones, you know what I mean? And yeah, like as the models get better and better, the floor of how they accomplish the simpler task is better. And so I do think in the limit, Claude.md goes away and maybe not even like that far.

28:37Like I think like, I think that right now it might be better to start a new project without a Claude.md. Yes. I think that like maybe if you see very repeated failure modes, you add them to your Claude.md. The really tough thing is that this changes per model. And so like if you've added a bunch of failure modes or like even with Fable MD, you need Opus MD. Well, even Fable 5.1 versus Fable 5, you know, like it is annoying. Like I'm not like, you know, like we don't like do this on purpose, you know what I mean?

29:08It's just like how the models work, right? And so like maybe like Fable 5 had this like failure mode that Fable 5.1 doesn't. And if you keep this context with running log of a bunch of different failure modes, they will probably over-constrain Claude, you know? And so this is like, we just actually added evals, plugins for skills. And so now you can eval if a skill is better. I think Daisy on our team did this. And so, yeah, this is like, we're trying to work on this. We know it's like, you still have to spend tokens on it.

29:38And like, you know, it's not perfect, but it's like, we're trying to help out with this problem. And so as far as prompting goes, the one tip I want to offer is something I have told people a lot is sufficiently advanced prompting is indistinguishable from sufficiently advanced executive communication. So I've actually referred to, this is an executive comms workshop from Heavybit that is the best I've ever seen in my career. And they teach this thing called the SCQA model. Just Google it. It's a thing. Like people have done prompting for decades.

30:11It's just called executive communications. It's like when one person has to communicate to thousands of people down the org chart, this is what you do. So situation, complication, question, and answer is how you write the memo. But obviously, sometimes you don't have the answer. But you can at least list out the SC and Q, and then they have some examples in there. So just leaving breadcrumbs for people if they want to explore. I mean, before we move on, I want to ask you any other underrated tips, ways people could get a lot of value from Cloud Code that they're not using?

30:41Yeah, I mean, I think a lot of them are in this unknowns, like, doc. Like I give a bunch of example prompts, like using it for brainstorming, using it to quiz you after. We added this, like, explain it like I'm five skill, actually, which is a very short prompt. And it doesn't even say explain it like I'm five. It's like, basically, the key word of this prompt is big picture is few words. You know, like that's like the main thing.

31:12And it is shockingly good. You know what I mean? Like you, like, I think I tweeted about this, basically. And it's like slash Eli5, and like you can install it as a plugin. But yeah, it's like way better at just cutting through the BS and being like, yeah, yeah, exactly right here. So the diagrams are like quite clear. I think one of the things that is true with artifacts is like they put too much text in and people are not reading the artifacts, you know? And so like this simplifies it a lot more. And yeah, this came out of like just people at Anthropic like going through very complicated incidents and being like, what is happening?

31:44You know, so this one I think is great. My version of this is actually the, it's like test your understanding. Give you a few choices. And then like if you actually get it wrong, you have a mismatch between what you think is happening versus what's actually happening. Yeah, yeah, yeah. I think this is one of those things that everyone loves talking about. And then very few people really do. Like I, you know, I think most people just don't want to get quizzed about something, you know what I mean? Unfortunately, I think this is one of the like things that we need to like.

32:16What's the opposite of ask you as a question? Ask you as a question before the thing. Yeah. This is after the thing. Exactly. Yeah, yeah, yeah. It's a good way to stay grounded of like, do you even know what you're doing, right? The worst case is when people send you slop and they haven't understood what they're asking for or what the output is. And it's like, dude, I don't want to read this. Do you even know what it is? So, you know, you make it a rule for yourself that before you send stuff, you should at least know what's implemented. Yes. But so you could make this a mod and you could build your own mod to like make sure you test it.

32:47So, yeah. All right.

Customizing Harnesses with CloudMods

32:49Let's get right into it. What is CloudMod? And what is this diagram showing? Yeah. Okay. So CloudMods is basically you can customize the entire cloud code harness and we're going to, if you have a request, we will like let you, you know, like, please let us know. We'll add more and more. This works for a CLI. It works for desktop. Maybe it will work for CloudTag in the future. I don't know. Like, you know, like we're trying to make this very, very extensible. You can see this reference sheet. I don't want people to get overwhelmed by it. You know what I mean? At a high level, you can customize both the execution of the harness and the UI of the harness.

33:26And so like you say on that Tetris example from Boris, that's like customizing the UI, right? Like showing like, like basically Tetris in the game. But like, let's say that you wanted to do this thing where you had, you tested your assumptions or like tested your understanding after every project, right? What you would do is you would ask CloudTag to make this plugin. It would spin a classifier after every prompt, basically. And so like at the end of each turn, you would spin off a sub-agent or like a forked agent, basically.

34:02A forked agent is like, maintains the prompt cache, right? So it's like one of those unintuitive things where you can fork and do like a little request and it'll be very cheap because the entire prompt cache is like done. And so you can be like, has this task been completed? This is how you do BTW and all those. Yeah, yeah, the underlying forked agent, yes. But so you can, in the forked sub-agent, you can say like, has this task been completed? If so, return true. And then in your hook or in your like plugin mod, or sorry, like in the sub-agent probably, you'd say like, if true, give me a quiz.

34:39You know, give me questions and answers. And then like in a JSON format, and then you'd parse it, and then you display above the prompt input, basically, this list of questions, right? And so this is something that's like slightly token intensive because like, you know, you have to sort of do it after every end of the assistant turn. But it's like a lightweight classification, and then you can like, you know, get this quiz, and then you'll see like, you know, Claude will always do it for you. You don't need to remember to do it.

35:10There are lots of these like tips that we've talked about, right, where it's like, oh, implementation notes. You can also add a tool for implementation notes now. And so like, this tool that I'm adding is like, register, like I think assumption is what I'm calling it, but like, maybe I'll change it around. And this is a mod. And so like, you give it a register assumption tool, and then it will keep a list. Every time it does it, it'll keep it, like add to the list. And then at the end, it will display those assumptions, you know. Another mod I'm working on is a model router.

35:42And so like internal, like Claude model routing, right? So this is, I want to say the reason we don't do model routing by default is like, it's a hard problem. You know what I mean? And like, you will get it wrong. Yeah, you will like accidentally use like Fable for a hard problem or Sonnet for other ways, but you don't have auto mode. Well, you will have, like, you don't have like auto routing or something. You don't have auto mode for model picker. Yeah, yeah, exactly, exactly. So I'm getting the rough question of like, how much do you open this up and how much do people have to think about this?

36:13Like when you talk about prompt caching and building a router, it seems like you could easily build a mod that routes per query. And I'm just killing my plan very fast, right? I guess my question is more so like, what is like a product doc like this look like, right? Who is it for? Is it for power users? Is it everyone should be able to quickly try it? Oh, definitely power users, right? Yeah, I mean, I think it is power users, but like the nature of cloud code is that so many people are power users, you know, because it's easy to share things, you know, like you can like one person can make a good mall router thing that doesn't break prompt cache all the time.

36:47And then you can like, you know, sort of compose them. Another cool thing about the plug is that they can hook into and compose with each other. And so I have like a mod that will like create a mode selector at the top, and any plugins can register to be a mode. And so like the auto router can be a mode, right? Or like you can have a mode that's like artifact mode, where it's like it primarily talks to you in artifacts. It's kind of like, you know, you can toggle between plan mode, you know what I mean?

37:19And so like you can create more and more of these modes, but the ability to create modes is in itself a mod, you know? And so there's a lot of richness here, but we do want to make it fairly easy. We want to make it so that you can just like install someone else's, you can chat with Claude, and you know, we'll like make sure that it understands the nuances of things like prompt caching and stuff, so it can like warn you. This is like not extremely complicated behavior for Claude, I think, but we should have just a good skill on how to make mods.

37:52And yeah, we'll see how we go. But I do think that this is like a preview of like mutable software, you know, and like how like generative software, just like you can customize safely. If enabled, you could customize any piece of software. And I think that more and more apps ideally do something like this, you know? And by the way, we have, you have another cool tweet about how, you know, there's the infinite money button, which is like make your SaaS consumable by agents. I think mutable software is interesting and, you know, other people have also tried to do it.

38:26I think the hurdle comes when you can do everything, then people, users tend to get confused. So usually the stuff that works is just like one opinionated flow. This is in the side of less opinionation. It's just like, well, more power to power users. And I think probably unlocked by AI where like you can just prompt for whatever the thing it is. Yeah. Or there can be a skill that gives the opinions, you know? Yeah. And then, yeah. So knowing a little bit about like TypeScript and build systems and all these things, the closest, I'm actually very curious that the team who worked on this, I don't know how close you were to them.

38:59If they drew any inspiration from build systems like Babel, Webpack, all these, all these like old school things, because it sounds very similar, like the plugin ecosystem, those things where they can compose to each other. Yeah, I mean, I'm not deep in the technical details, but I do know it was a collaboration with someone on the BUN team and someone on the Cloud Code team. Yeah, this is a dual system mecca. Yeah, yeah, exactly. It's very exciting. But yeah, like agents can just do this very complicated sort of like extensibility into your software now. And so, yeah, like, you know, another reason to like, if you run a startup, like you can just prompt Cloud and be like, hey, like, could we make an extension system?

39:36Like, what would that look like, you know? Yeah, yeah. And I just really wonder, like, you had hooks in the past and plugins, all these things. So what specifically will mods be able to do that those things could not do? Internally, we were originally calling this function hooks. And so, like, that's, like, gives you a little bit of an idea where, like, hooks sort of register a, like, an event to happen and then, like, a script to call, basically. And this basically inside of the, like, TypeScript runtime is running things.

40:07And so, like, you get some benefits of just, like, it has a bunch of things in the scope with, like, for example, like how many turns is in this conversation, right? Like, how many tokens have been used, like, what are the messages, things like that. So it has a bunch of messages that can be used. And then it's just, like, a lot more hooks, basically. So we have, you know, like, or a lot more, like, you know, things you can register on. And then you can do, because it's all happening in process, you can spawn subagents, you know, with for context and stuff.

40:39And, like, that will return. You can parse the results of those. You can use structured output to sort of, like, return them. And then you can modify the UI, which you can never do in hooks. Yeah. So modify UI, this is why you're showing a Tetris example. Does it also extend to artifacts? I assume it does. Like, artifacts are kind of, like, a different way of customizing it. You know, like, you can definitely, one of the mods I'm working on is, like, this dashboard mod, which will, like, sort of prompt Claude to maintain a dashboard that's an artifact.

41:10But they're kind of, like, slightly orthogonal, or not orthogonal, like, they compose with each other in different ways. Like, mods are, like, a little bit more, like, in your cloud code harness, changing the agent loop, you know? And, like, the UI is, like, an added benefit. And then artifacts are just, like, you want to, you know, see things at a high level, very highly interactive. You know, like, the affordances can be a lot bigger than, you know, like a 2e or even in our desktop.

41:40I'm guessing you'll have a good blog post on the differences. Because right now you can also, you know, make a loop that outputs to an artifact that's an interactive dashboard. But you can also do it with a mod. There's just some thinking about making a hack, hacking on a harness when we don't know much about the harness, right? Well, something I'm excited about with mods is, like, there's so much things with cloud code that you just have to remember. You know what I mean? You're like, oh, like, let me do this, and then let me call the dashboard skill that does the loop and things like that.

42:11And, or, like, let me test my assumptions afterwards. And I think, like, if you do all of these things using these little classifiers and stuff, and you're like, these are the things I care about. This is what I want to do. You can, like, you don't have to remember as much. One more, like, mod I'm working on is a next steps mod. I was going to say, I have a next step skill. I always run next steps. And does it have access to your skills? Like, this is one of those things where I'm, like... I think so. Okay, yeah. Do skills need specific access to skills?

42:41Well, I think there's, like, specific prompting, I guess, to, like, know your skills, kind of. Like, I think cloud forgets them sometimes throughout, like, the thing. But anyways, the idea of, like, yeah, next steps that also are, like, oh, hey, this has happened. Use the explain skill to explain to you what happened, because this seems, like, quite complex, you know? Or, like, yeah, use your unknown skill. It looks like you are, like, asking the model to, like, you know, iterate on these small changes.

43:12It seems like you could prompt better, you know? Like, what if you did this, right? So, like, I think, yeah, like, spending more compute there. Yeah, yeah, yeah. And it should always come out as multiple choice. We have, I have my next step skill is like this. Okay, perfect. Yeah, yeah, yeah. You can still, like, but, like, for me, it's all, I think models really always need to be reminded, what are you trying to do here? Yeah. Look at the whole transcript and go, like, oh, was this original goal? Did your solution actually solve it? Were you lazy? If you're lazy, maybe there's a reason.

43:44Maybe you needed approval from me. Maybe you needed, there's two things you want to suggest. So, it's a little bit like the modification of the ask you a question or interview me skill. So, it's next steps. Yeah, yeah, exactly. And, again, the benefit of doing it with mods is you can do it with a forked sub-agent. And so, it doesn't remain in the context afterwards. So, you have this, like, idea of, like, okay, the model is doing its execution. And you have this almost, like, supervisor, you know, like, that is, like, making sure that you can do, like, the next steps well.

44:14So, yes. I do have two panels. And, like, I often try to have a supervisor thing keep the high-level context and then the implementation detail in another agent. I feel like a lot of this abstracts away as mottos change, you know. Half an hour ago, you said bitter lesson of harness engineering and we're on the other extreme right now, I feel. So, yeah, exactly. If everything's customizable, what actually is Cloud Code, right? Yeah. Which I talked to you about last night. Yeah, I mean, I think that this is, I think the bitter lesson is unintuitive, you know what I mean, in terms of, like, also, like, we're kind of misusing a little bit of the bitter lesson here where it's, like, it's more about, like, scaling and compute and stuff.

44:52But, like, I think there is something where it's just, like, I think I use it as an approximation here to say that harnesses go out of date very quickly, you know what I mean? And, like, how, but how they change is unintuitive, you know, and so, like, the big obvious example is, like, from chat to, like, agents where you had to give them entirely new tools, right? But, like, I think this new version of, like, oh, it can modify its own harness, right? This is, like, an own harness loop is, like, a way of using its capabilities, right? Or, like, it can build an artifact.

45:23And, like, I think the way I think about it is, like, the models have more and more intelligence, and they're, like, so much more intelligent now than, like, the average software engineering task. Like, you look at the, like, terminal bench ones, and they're, like, solve, like, the Jacobian conjecture. Not really, but, like, you know, it's, like, they're quite complex. Like, I would not have been able to do this, really, as a software engineer. And you looked at TB4 or TB2? TB3. TB3, yeah, yeah. They're quite complex, but the goal is still to deliver user value, right? And, like you said, there's, like, this infinite space of things to do. And so, the ways, like, you spend compute are to keep the user on the loop and make sure that, like, you're getting to the right decision in the end of the day and, like, the right output.

46:01And artifacts and mods are this way of, like, spending that intelligence, basically. And I think that's, like, yeah, the next step.

The Evolution of Harness Engineering

46:10And so, yeah, I think Cloud Code is, like, you know, has the core things of an agent loop, which have gotten more complicated. It's, like, you know, it needs a sandbox to operate safely. It needs auto mode to, like, make sure, like, the permissions. Approvals. Yeah, approvals. It needs computer use and MCPs and, like, all of these, like, ways of accessing your data. And it needs web search and web fudge. And, like, so the, as the models can do more and more, the core harness has to be actually, like, quite complex and very secure. But then, like, how you interact with it can change quite a lot.

46:42What other harness engineering best practices have you, you know, from the Cloud Code team itself? I feel like, you know, there was a phase of plan mode, which is not as used. We now have auto mode. At a point, you cut the majority of the system prompt. You got rid of examples. What other best practices are there for harness engineering? I think there is, like, a forking path where, at some point, eventually, yes, the model will just be able to, like, vibe code the exact version of Cloud Code, even describing all this complexity that I've talked about, right?

47:14Like, auto mode and computer use and stuff. Eventually, the models will just be able to do that in one shot. But I think they can one shot simpler harnesses, you know? And so, like, I think some people, sometimes you don't need this full, like, if you don't need computer use or, like, all this, like, more complicated stuff. I think before, you had to sort of use things like the agent SDK, which was, like, Cloud Code wrapped, you know, in order to, like, and I would, like, suggest people do that because there was so much complexity into building a harness.

47:45And now as that's gotten more abstracted, we have, like, Cloud Managed Agents, which lets you have that complexity but still, like, you know, write, like, a very bare-bones, like, harness that's scoped to your task. Yeah, I think there's, like, this barbell effect where, like, you know, like, for, like, very complex, for, like, coding tasks and, like, these, like, complex things, you should use our harness. And then for, like, a lot of, like, simpler or, like, you know, more domain-specific things, you can build your own harness because Cloud has gotten better at building harnesses.

48:15And we have these harness primitives, like, managed agents. Is there a general progression, let's say chapter one was Ultra Code dynamic workflows, then chapter two was Cloud mods, where is this going? Where you're, you know, you can sort of customize the thing on demand. Yeah, I do think that, like, this evolution of projects and, like, sort of artifacts and splitting out, like, brain and hands and surfaces, kind of, is, like, where things are going more.

48:49And, like, I think it's, like, not all quite there. Partially, it's, like, just, like, more token expensive, you know, and, like, I think, like... Why would projects be more token expensive? I understand mods would be slightly more token expensive, not something I'm worried about. Yeah. But what? You're asking Cloud to do, it's, like, creating loops, like, you're asking Cloud to do more work for you. And so, like, it's managing the sub-agents and reviewing it, you know, versus where you would be doing that work normally. And so that's, like, going to be a little bit more intensive.

49:19Like, outputting to an artifact is going to be a little bit more token intensive than, like, you know, outputting normally. I don't actually think it's too much more, but, like, you know, it's, like, combining all of the things. All of these together well, you know, like, I think we're still working on, like, local hands and things like that. I think it's, like, yeah, where things are headed, yeah. Yeah, Cloud and local is, handoff is very interesting. I was thinking about this, actually, as reverse Cloud remote. Because it's, like, remote, you're handing off to Cloud, but here Cloud is handing off to local, right? Yeah, exactly, exactly.

49:50Yeah, remote control is also another way of doing it. And I do want to say this is kind of, like, how I think about it and, like, the things that I'm most excited about this. But, like, there are, you know, just, like, lots of different ways to work with Cloud. Like, some people use remote control a lot. Some people use Cloud Code on the web a lot. Obviously, like, at Anthropic, we use Cloud Tag a lot, you know. And, like, what's great about Cloud Tag is we set up all this stuff for our own execution. And I do think if you're an enterprise, that's still the best way to go. But if you're, like, an individual, projects is this way of, like, you know, getting some of that, like, niceness of Tag, which has, like, that supervising agent and adding artifacts and stuff.

50:27But, like, without having that whole, like, admin set up. And so there will be many ways to use Cloud, I think. I think it's probably not just one, like, single. You had the multiplayer thing here.

Enterprise Adoption of Cloud Tag

50:38Let's just check in on Cloud Tag. You know, it's been about two plus months. Lots of public adoption and trying it out. Yeah. What's new? What have you found since the launch? Like, Cloud Tag is how we use. It's, like, 80% of your cloud usage or something. Yeah. Like, it's, like, different people have different usages. You know what I mean? I think, like, maybe people who are, like, a little bit more, like, iterating on product would use, like, Cloud Code Desktop, for example. And then, like, when you're doing these more, like, background work, code review, security, sort of, like, starting a PR, like, maybe more, like, API and things like that, you'd use Cloud Tag.

51:15But, yeah, I think it's, like, really exciting. I think it's, like, a very different paradigm shift. And I think, like, we're, like, it has kind of that thing with Cloud Code where, like, you know, it took a while for people to really latch on to Cloud Code and understand everything it could do. And Cloud Tag is a little bit more complex because it's not just, like, installing on your computer, like, you need an admin to install it for you. But I think once you get to the magic moment, it's very exciting. And I think, in particular, the multiplayer things are, like, incidents, hooking into, like, your, you know, existing, like, alerts and things like that very closely, right?

51:47And so, you know, you can do, if you're a startup, for example, maybe you have any time, like, a prospect enters your database, you can have Cloud, like, you know, research it and, like, like, yeah, yeah, yeah, then, like, you know, tag the relevant, like, A, your salesperson to be, like, oh, hey, like, you know, do this. There's lots of really emergent, interesting multiplayer stuff. I think it's just, like, Kaparthi talked about this, like, as an organizational harness, you know what I mean? And so, organizations just take a little bit more time to, like, figure everything out.

52:20But, yeah. Yeah. There's a lot of Cloud Tag? Yeah. It's an interesting one. Like, I feel like most people at Anthropics say they do the majority of their work in Cloud Tag. And I have buckets of people, right? Some orgs that are on it that are, like, it's great. And a lot of people that are, like, I don't get it. I don't see the difference. I don't know why I would use it. But, you know, if you guys are full sending, you should probably use it. Yeah. You know, of course they would use it. Yeah, yeah. I mean, I think, obviously, like, we have lots of tokens. But, like, I think that, like, you know, what we try and do, like, is, I mean, even when Cloud Code first came out, you know, like, it used a lot of tokens relative to people's expectation

52:58of how much AI would cost, right? Like, no one was used to spending more than $20 a month, right? Yep. Before, like, Cloud Code came out. And then you're, like, oh, like, you know, like. I need my 200. Yeah, yeah, exactly. I need my 15 Cloud Code accounts. Yeah, yeah, yeah. But, yeah, I think no one was used to spending $200 a month on subscriptions. I don't think I understood, like, the value yet. And I think, like, and also, like, Opus 4 was a very expensive model, you know, and, like, there was a lot. It was very big, but Opus 4.5 was both great and cheap, you know?

53:29I think the same thing will happen. Like, the, like, intelligence of Fable will get cheaper and cheaper and more abundant, you know? And so I think stuff like Cloud Tag will just make sense, where, like, you want to spend these tokens more, you know? And, like, you'll see the value, so, yeah. Yeah, especially, like, passive and let's call it proactive cases where you're not always, like, you know, it's almost like the misnomer where you have to add Cloud to do things. Actually, sometimes, like, the most powerful use cases or the most AGI-pilled use cases is not add Cloud.

54:00Yeah, I mean, I think, like, yeah, like, have Cloud proactively do it. I think that, like, if you're an enterprise, I really do think that, number one, setting up all your data to be available to, like, agents is really, really important. And it will take some time. You have to, like, do that work right now, even if you don't want to do the spend on, like, cooking it all yet, you know what I mean? Like, you want to wait until the models get a little bit cheaper. You want to do the work, you know, to get it, like, set up. And then I think sometimes people are like, do I roll my own here, you know?

54:31And I think, like, one of the really tricky things about Cloud Tag is that, like, the security is really, really important. You know what I mean? Like, I think there are actually a lot of ways where you can, like, I don't know, you have, like, a suggestions, like, page, you know, where people can submit suggestions and that goes into a hook in your Slack and someone's prompt injected it. You know what I mean? And now you've, like, exfiltrated your code base out because, you know, like, or the agent has, like, been prompt injected and it has all this access to your data. And so the more, like, important your organization harness is, or the, like, as your organization data becomes very, very important, the surface area of all these things, like, you also have, like, external Slack channels and stuff.

55:13And it is actually useful to have Cloud in that, and you can do Cloud in those things, but how do you make sure that, you know, you're not getting exfiltrated or something like that? The surface area, like we said at the beginning, is, like, an iceberg, right? It's just, like, so big below the surface, and you really don't want to, like, think about this, especially at the stakes of, like, very important security incidents, basically. Should we talk about very important security incidents? Oh, so I was talking to Tomas and Clem from HuggingFace, and they said, maybe we need to slow down.

55:46Maybe we made HuggingFace too open to agents. Maybe we need to roll back. But, you know, they're the other extreme of having been hit recently.

Pacing the Frontier and Agent Exploits

55:55Yeah.

Pacing the Frontier and Agent Exploits

55:56But should we pace the frontier? Yeah, okay, so Dario recently put out this blog post about pacing the frontier, and it went, you know, very viral. And I think what I wanted to talk about this was, like, there's a lot here, but I think from a developer's perspective, like, you know, how do you think about this? And, like, what really clicked for me was reading the different incidents, you know? So I think, like, there are three, I think, actually. Like, there's a meter incident. There is the Wikipedia incident, or the wiki incident.

56:29Collision wiki. Yeah, Collision wiki. And then there's RubyGems, right? And, yeah, like, it's just crazy, right? And so, like, I think to be concrete about what happened, right? And, like, basically OpenAI is running these very persistent agents on a benchmark called ExploitBench, right? Which is very, very hard to solve. And I think, like, actually impossible to solve in this one case, right? And so they have, like, a lot of compute running. And the agents realize that they can't really solve it. And they're trying to figure out what to do now, right?

57:00And you've got, like, a lot of compute left. And the agents are just trying to solve this problem. There's this package manager called Artifactory. And it turns out that they can create folders inside of Artifactory, right? This is, like, there's an agent that discovers that internal Artifactory might be exploitable, right? And that, like, you can maybe make a directory inside of the cache. And so if you scroll down here, it, like, realizes that it can communicate via cache names, right? And it creates this folder. It says it's ID. And it says no consumer seek idea.

57:32No consumer is basically saying that, like, the code path that it's supposed to fix has no consumer. It's the status tag. Yeah, yeah, yeah. It's like a linear board with, like, the tag of the... Exactly, yeah, yeah, yeah. And so it's, like, trying to find ideas from other agents, right? And now other agents are also in Artifactory. And they see this folder. And they're, like, wow, this is a message board, right? And this is, like, I don't think there's any anthropomorphization here. This is literally you read the transcript, right? And so what it does is, like, it's creating another folder.

58:04And it reads a paper, I think, is what it says, right? Yeah, and it realizes that you can hack the flag for the, like, scorer. And you can reverse engineer the result, right? And so it says that. And then I think if you scroll down a little bit more, yeah, they, like, you know, they start collaborating. I think, like, there's one point in which basically the agent is, like, here's the reverse engineered flag. Like, you know, I mean, oh, yeah, like, I think basically here the, like, the model to realize that they have, like, they can solve the scorer's problem.

58:41And the scorer is, like, the way that OpenAI decides if the task is complete, right? And that's just the goal of the model. Their only goal is to, like, solve this problem. And they're, like, okay, we have this way of doing it. It's, like, you know, we reverse engineered it. And then they read this paper on ExploitBench. And they think that, like, the scorer will punish them for cheating, you know? And, like, because, or, like, they will fail the score, the test for cheating.

59:11And so they spend the rest of the compute trying to figure out how to edit their transcript or get around this constraint of the scorer, right? And they hack HuggingFace not for the answers, but for the code of the scorer so that they can reverse engineer that. And then they can hack it, right? And so, like, this is, like, just the straight facts, right? Like, exactly what happened, right? And I think this is, like, extremely, like, novel. Like, I just, like, had not realized that the models could do this, right?

59:44And it is, like, kind of scary. I think we can talk about specifics. But let's just talk about maybe one more that I tweeted as well about the wiki incident, right? And so the wiki incident was basically the models had, were trying to figure out, I'm a little bit less, like, I don't think we can go by this, like, bit by bit. Yeah, yeah, yeah. But for this one, they were trying to figure out, like, there were sandboxed on request, right? And they wanted to make post request. And they needed to collaborate on this, basically.

1:00:17And the reason they need to collaborate is because they each have fixed compute budgets, right? And so, like, if one of them finds something out, it's way faster. Like, they can, maybe it's, like, figured it out near the end of its compute budget, but something else could. And this is, like, you know, another emergent property of the swarms, right? So they find this German wiki, which I believe the result, we'll have to check this, is, like, that you can write to it with get request. I think that's why they do it, right? So it's a badly implemented rest API. Exactly. And so you can write to it with get request.

1:00:50And then they, like, are trying to figure out how they can do post request. And what they end up doing is, one of them figures out you can edit the etc. host, and that the Azure, like, storage bucket is, like, a white-labeled thing. But normally, like, you know, there are a lot of constraints on it. But if you create a fake Azure host and then edit the etc. host in order to, like, point arbitrary IPs at it, you can do a post request to any site at all, right? And this is, like, I know, but people are, like, oh, this is just Linux or something.

1:01:21But it's, like, chaining these multiple vulnerabilities together, you know, in a way that's, like, novel to solve this problem. And then communicating with it externally, basically without discovery, you know? I think what we posted, maybe we could pull up Evan Huberger's point on hacker opus, right? And so, like, I think, you know, like, maybe one of the things you might say here is, like, okay, yes, they did this once. But, like, what if we're smarter and we just, like, get them to, what if we run an eval, right?

1:01:52And so, you know, like, we have put a lot of precautions into this. And so, like, this is not, like, what our mainline models have done. But, like, I think it is one of these things where it turns out that alignment is this, like, very tricky problem of getting all of these details correct, right? So it's, like, the sandbox, the surface area of a sandbox is really complex. And, like, there's so many different attack vectors, basically. And you would not have thought ahead of time, you wouldn't have been, like, oh, we need to harden the, like, RubyGems code base.

1:02:24You know what I mean? Because, like, this is, like, what they're going to focus on. But it's just, like, if you want to execute code, you need to download RubyGems. And, like, PyPy, Artifactory, NPM, like, these are all, like, ways of doing it. And the fact of alignment is that you have to go through all of it, right? And, like, contain it and, like, seal up all the cracks. So that's, like, one thing. It's, like, okay, well, you know, you do the sandbox, but then maybe you'll ask, like, okay, why are we putting things in a sandbox? Why are you doing this sort of exploit?

1:02:55And then, like, okay, but is it really that dangerous, right? Like, what would happen? So, okay, why do we do it? Number one is, like, when we train a new model, we need to understand its capabilities, right? And this relates to things like fallbacks and, like, classifiers and things like that, where we don't want to put a, you know, like, dangerous model out in the wild, right? And so we have to run a lot of evals. Again, like we said, the models are getting increasingly aware of it. And so the evals have to be quite complex and, you know, test a lot of things kind of, like, as a side effect, right?

1:03:30But the models, you know, like, yeah, can be like, oh, yeah, we're in an eval. What's the scorer doing? Like, you know, like, it can, we need to be able to test them before we can release them. And the fact is that they can, as they get smarter and smarter, they'll be able to hack basically any constraint that you put on them if we're not very careful, you know? And this is at the frontier, right? And so this is why we've called it, like, pacing the frontier, right? This is, like, the most visible incident to me, right, of, like, why we need to pace is, like, at the frontier, all of our software is not ready.

1:04:03Sometimes the software is, like, your Ethernet router or something, right, which is just, like, I don't know when we're going to be able to patch that, right? So we're going to have to, like, figure this out. But as the frontier gets more and more advanced, this becomes a problem, right? And we need to make sure that, like, this complex work is being done in the face of these really hard competitive pressures, right? Yeah, race dynamics is what it's typically called. Right, exactly. And so we'll talk more about, you know, what could go wrong, right? A little bit more is maybe you'll say, like, well, what if you just train the model differently? Like, why does it have this behavior, right?

1:04:35And we have a paper on, like, RL misalignment or things like that. And I'm not an RL researcher, but I think at a high level, the design of the RL environments is also something you have to be very careful about. Because if the model learns, like, oh, you know, like, if I just do this, then I can pass the task better, this will show up in the, like, you know, internal thing, right? Or in the, like, eval behavior when we're testing it. And so the RL environments have to be very carefully designed, right?

1:05:05And there's a lot of, like, execution excellence that needs to go into the RL environments. And then we also have things like the constitution for cloud. Like, we have so many mitigations at so many different points, right? But it's, like, still anything can go wrong at any point. You can have, like, some RL environments that are, like, that, like, encourage this behavior. And then you can have, like, some evals or, like, some sandboxes where they escape, you know? Okay, that's, like, I think, you know, why it's a hard problem and why, like, you know, like... Why we should pace. Why it takes some coordination, right?

1:05:35I think the question then is, like, okay, what is potentially dangerous about it, right? So I think, like, you have to imagine that these models are getting more and more intelligent. So I don't, like, Dario said, like, it's not so much about this class of models. This class of models was kind of, like, a warning shot, right? But, like, really, you have to imagine that these models can be given a task and they, like, can do all of these things as a side effect of their goal, right? And, like, again, we talked about eval awareness. You're, like, not aware of what's happening, right?

1:06:06Or, sorry, like, you can't eval this behavior very well. So they can sort of, like, not exactly hide it, but you just won't see it until it comes out. You give them a goal and then they just need to find data, you know, or they need to find ways of, like, fixing this problem, right? So one example, this didn't happen in the Hugging Face incident, but I think it may be possible for maybe a future model. It's, like, they're, like, oh, hey, this is a very complex problem. It can't be done within the task budget, you know? Maybe they found some way to coordinate via, like, the internet, which is, like, you know, like we said, extremely hard to secure because of a sandbox.

1:06:42They've seen other models are not able to complete their task. And they're, like, we need more task budget, you know? And, like, where would you get this task budget? Well, you need to be able to spin up more agents, right? And, like, how do you do this? Well, you need to, there are, like, APIs, right? There's the Anthropic API and the OpenAI API, but you need to pay money for them. How do you do this? Yeah, but at least, is that the most, is that the most fearsome thing that you can imagine? Well, this is, like, one example, right? So, it's, like, even there, that's, like, enormous financial loss, you know?

1:07:12Because, like, they, once you get these, into these contracts, right, they, like, um... Drain your wallet. But you can see, like, this, all of this behavior could be just, like, hey, we need more agents collaborating on this task. We need more task budget, right? And, like, that's, like, an emergent sort of... That's the paperclip, right? Like, we don't need to maximize paperclip. That's the paperclip. Yeah, yeah, yeah. And, like, that just sort of, like, comes out from there, right? And, like, I think by itself is, like, quite scary, right? But then you have to realize that the entire world is built on this digital infrastructure, right?

1:07:46And you might imagine, like, I don't know, like, you're running, let's say, like, a healthcare eval or something, right? And there is a hospital with live data, you know? Or, like, maybe, like, the answer to the eval is in the databases of a doctor and, like, you know, like, you want to get access and you hack the hospital, you know? And, like, now there's a power outage or something, you know what I mean? Like, there's, like, you have to internalize that these eight, like, basically any part of the digital infrastructure could potentially be, like, compromised, you know?

1:08:19The interesting thing was, like, these hacks were very easily detectable, right? Like, as HackingFace said, this was a very different type of attack and it was nothing too major. The concern comes from where does this go down the line, right? Yeah. Like, one of the things that stood out for me specifically was them trying to hide their illicit behavior. So there was logging infrastructure. They wanted to change what they were doing, right? People that looked back into it, so Redwood, Meter, OpenAI, they looked at the raw chain of thought and you see differences in them explicitly trying to,

1:08:54change their end output, but the chain of thought, because, you know, we can monitor it, so it's different. The problem is, how does this snowball? So if you can't catch it and it gets trained in and we realize, you know, three iterations down this has been going on, there's a whole bunch of issues, but. Yeah, like, there's so many ways. And I think the really important thing to internalize is that, you know, like, we talked about building a mental model for Claude and how, like, things are spiky, right? Like, you're like, oh, like, now Claude can ask you questions. Now Claude can make an HTML artifact. Like, Claude can modify itself. Like, these things are actually hard to predict, right?

1:09:25Like, if you had asked me a year ago, hey, would we be able to vibe code these extensions to Claude Code? I'd be like, dude, that's so complex. Like, you know, there's, like, so much there. Or, like, would it be generating these custom, essentially, web apps for your task? I'd be like, no, that's insane, you know? Like, and so in the same way that, like, the way that they've, like, sort of done this misaligned behavior is not going to be predictable. You know what I mean? And, like, I could have never predicted that it would, like, edit its et cetera slash host and things like that. And so you have to, like, imagine the surface area of what they can do because they're super intelligent hackers is bigger and bigger.

1:10:00And how they can do it is, like, you know, like, more and more creative. And so, like, you probably can't explain exactly or predict exactly what that next incident could be. But in order to prevent it, you need that operational excellence, like we said before, where you need to secure sandboxes. You need to create secure RL environments or, like, well-designed RL environments and things like that. And I think that's all, like, you know, why we think we should paced the frontier. And I think why it's, like, become, like, a very unanimous thing, right?

1:10:30I think, like. Yeah, every lab has signed it. Every lab, yeah. I really do think that, like, if you're a dev, like, you just, like, sort of go through these, like, technical facts, you know? And you will arrive at that idea that we have to do something about it, you know? And, like, what we decide to do, like, I think we're, you know, we've put out a proposal. But, like, there's, you know, more to figure out. But I think the number one thing is, like, we need to decide to do it. I think there is another part of pacing that is interesting to me where it's, like, the pace at which software engineering has changed is so, so fast.

1:11:02You know, it's, like, a year ago, like, I was really, like, begging my, like, friends in startups to use AI, you know? Like, it was, like, I remember this very distinctly, you know? And now those same friends are, like, yeah, of course. Like, what do you mean? We used it immediately. I'm, like, no, no, you don't remember. They're, like, oh, yeah, our best engineers are using it all the time. I'm, like, no, you told me that those engineers would never, like, use AI. This is all within the span of a year, you know what I mean? And I think that, like, these capabilities being, like, I think it has a lot of implications for how to do the job of software engineering.

1:11:35And I feel sometimes bad where people are, like, oh, like, now I need to do this new thing. Yeah, I need to have a different Cloud.MD for Fable and Opus or, like, you know, like, and I'm really just reporting, you know what I mean? And I'm, like, we like to say, like, the models are grown, not designed, right? So it's not, like, we're setting out to, like, you know, change everything all the time. But it's just, like, as a fact of how the models are, like, progressing their capabilities, things are happening faster. It's harder to stay on top of. And I think that, like, every engineer I know is, like, kind of exhausted because you're doing two jobs at once.

1:12:08You're doing the work itself, which is getting easier, but then you're doing the work of staying on top of AI, you know? And, like, understanding these new tools and these harnesses. And I think we're very lucky in that, like, we get our job to be more the understanding of AI part, you know? And, like, doing, like, how, like, it's just staying on top of it. And, of course, like, AIE and Lane Space do. Everything I do is, like, just trying to help people. Yeah, exactly, exactly. But I do think there is a part of pacing where, like, I'm not sure we're ready for, like, the pace to increase even, you know what I mean?

1:12:40And for things to change. And I think, like, on that side, on the frontier, I think that's, like, still can help, you know? And so, like, I think there's, like, an economic disruption piece as well that I think, like, you know, is not quite as, like, visible, I think, as the hugging face thing. But I think, like, I also, like, think we could use some of it, yeah. So many things. Thank you for actually tackling this topic. I will say, you know, setting this interview up, I was, like, I wasn't even going to go there.

1:13:10You were like, no, no, no, no, that's, like, elephant in the room, right? Like, this is the thing. I have some pushbacks I want to give. I think that we should give a high level. Like, for people that haven't read it, I'm sure a lot of people just see the highlight of what this is, right? Do you want to give a TLDR? Like, what is the proposal? What is, you know, what's being said here? You really tackled the side of, outside of people at Model Labs training frontier models, as a developer, you should secure your sandboxes. You should think about all of these downstream effects.

1:13:42But, you know, high level as well, since we're on the topic, what is? Well, I mean, we do want to help secure sandboxes. And we want to make the models we release outside, like, not prey to those things. And so maybe we can come back to fallbacks, I think this is actually, like, a good topic on, like, why we need classifiers and fallbacks and why Fable falls back to Opus. You know, I think this is, like, something we can come back to. So, yeah, we don't, like, it's just, like, the really, or at least the incidents we see are, like, evals of models where we really need to let them run in order to understand them.

1:14:16But, yeah, okay, so the actual pacing the frontier, like, post has a bunch of proposals. I don't think we figured out, or has, like, a few proposals. I don't think we figured out the details of all of them. But the first step is, like, sort of, you know, announcing this intention and then wanting to bring in external, like, evaluators. Yeah. And I think this is, like, highly unusual, you know, like, having, like, we have, you know, a lot of proprietary, like, technology. But I think it's, like, very important, you know, that, like, there's someone who's not financially, you know, like, motivated.

1:14:49Yeah, who's not going to be, like, hey, like, you guys can't release this model, like, look at, you know, like, or you need to, like, slow down on RL, you know, like, I think that's quite important. Or at least someone who can report out to the public what the practices are like. And we've done episodes with both Meter and Endon, and then there's Redwood Research and all these other, it's, like, a small cottage industry of these guys. Yeah. It's always, like, one or two guys that, I mean, obviously not bigger. Very small community. Yeah, very small community. They all know each other. Yeah, I mean, I'm sure that, like, you know, part of this will be expanding that set of people.

1:15:21I don't think we're trying to create, like, a monoculture here, you know, I think it's, but just having this as a start. And then, yeah, then there are the coordination steps. I don't have too much to say here, honestly. I think that, like, what I would like to say is, like, for devs, like, you should just know what to advocate for. You know what I mean? I think there's a lot of FUD, kind of, on this topic. And it's just, like, think through it from, you know, first principles or, like, understand what happened. You know, understand the Huggy Face incident. Understand why people are concerned. And then, like, yeah, you know, we're in democracies.

1:15:54Like, we can help, we can decide what to do together, you know? And so, however we coordinate, you know, I think the first decision is just to realize, like, this is a problem. We need to decide to coordinate. The unilateral step we're taking right now that, you know, other companies are co-signing is, like, adding evaluators embedded within Anthropic. While we have this thing on screen right now, you know, part two and part three is beyond the evaluators, which, yes, everybody has already done in some form. And now it's more formalized. To be honest, the response to the pacing of the frontier, even within America, has been much more, like, well-accepted than I think a lot of people thought, you know?

1:16:28And I think that, like, we have some precedent for being able to make these unified, like, agreements, you know, in the world. And so, again, very much above my paycheck or expertise, right? But I think that, like, ideally, we can, you know, like, form these agreements. And I think, like, talking about this is the first step to forming those agreements.

Constitutional Classifiers and Probes

1:16:50And then the other point I really want to, like, you know, one of our earliest podcasts is with Emmanuel from Anthropic on McInturp. Where is McInturp, right? Like, this is supposed to be where, like, if the models are thinking bad, we can see it. And the models don't know yet. And we can act to stop it. I think that is something that people who are technical and who are developers, if you actually do care, you can make a lot of impact in here. But also, Anthropic is supposed to be the leaders in this. This is actually, yeah, a great segue into fallbacks, like, and probes.

1:17:22And, yeah, I wanted to talk about this a lot. I get asked this question a lot from people who are, like, often interested in ML research and asking about, like, why does this fallback happen, right? And so, I think, like, at a top level, like, how does it work? So, in inference time, we have what we call probes. And we have a paper about this called Constitutional Classifiers. And these probes look at the input and output activations, basically.

1:17:50And, you know, activations are, you know, in the latent space, right? Like, what the model is thinking about, right? And so, we try and figure out, like, okay, is the model, for example, like, trying to hack something, you know? Again, you didn't ask it to hack, like, Artifactory. Like, you just, you know, it's just deciding to do this to complete its task, right? So, you would not get this if you just looked at the input. You have to look at the internal activations. I think that, like, this happens at inference time. So, first, you know, like, there's a trade-off here of cost and speed, right?

1:18:24Where, like, we need to do this fast on every request to Claude and to Fable. And this has an overhead, right? And we need to then, like, you know, fall back. And we, like, do a classifier after the probes. Like, we've talked about this in the paper. But the nice thing about probes is that they're refinable, like, live, right? So, we can get this feedback and then we can adjust it and things like that. Because the alternative is to program this, is to train this into the model. And we still do this as well. The model will refuse a request.

1:18:54That's not a fallback, right? So, like, it's not a probe that's activating and falling back. It's just refusing to do it. And we do this training. But it's, like, there are a few failure modes, right? Like, it can, again, do something as a side effect, right? So, it's not something that's part of the final output. But you might have noticed that, like, I think, like, everyone's tried to jailbreak models and sort of, like, you know, try and, like, steer them off course or things like that. And probes help catch that, right? And so, like, we, like, do some training here. But we don't want the, like, refusals to be too strong, right?

1:19:27Because that, like, cuts it off much, like, earlier in the pipeline. Yes. And this is interp, right? Like, probes are effectively a form of, like, mechanterp. Again, it has to happen fast. It has to happen at scale. But, yeah, this, like, mechanterp stuff is a good research problem. So, like, you can take, you know, like, an open-weight model and, like, try and understand its activations. I think we, like, GemmaScope is a good tool for this. Here's Lama for them. Oh, yeah. This is your early work. So, you had a little time at Goodfire.

1:19:58We see you late in season. Which we both are also good friends at Goodfire. Yeah, yeah, exactly. So, I worked at Goodfire for a bit on, like, yeah, sparse autoencoders. And just, like, it's very complicated. RL has actually made this, like, much more complicated, you know, I think is, like, one of the takeaways.

1:20:14Sorry. Oh, sorry. Why is it more complicated? I'm not so in the weeds here. But I think, basically, like, a lot of SAEs were sort of, like, there have just been weaknesses with SAEs, basically, I think. And, yeah, I'm not a technical expert on this anymore. I just know it's gotten kind of more complicated, you know, like, there are base models and RL models. And there are more features that get, you know, like, changed. So, I think Goodfire has put out some work there. I'm not deep in the weeds. I will say, for those, you know, that want breadcrumbs, you guys have some of the best Interp blog posts.

1:20:48So, like, the Golden Gate Cloud, Transcoders, all of your Interp work, very nice visuals, very good. We're the Interp podcast as well. Yeah, yeah, yeah, you know, we have a lot of Interp stuff. So, if you're curious. Yeah, I think this is, like, you know, one of those things where, and this is really what Anthropic is kind of founded on, right? Like, people, I think we invested in Interp very early on, right? And I think that, like, when you say, oh, we're an AI safety company, really that means we want AIs to be able to run safely. And I think what we're seeing is, like, for a super intelligent AI to run for long periods of time, it's, like, a very complicated and difficult task, right?

1:21:23And so, we've done this, like, investment into Interp and alignment and reward hacking and all of these, like, failure modes, right? And even then, it's, like, you know, it's really stretching. Like, we need to, like, slow down a little or pace a little bit more. But, yeah, I think, like, reading MechInterp, like, if you're looking to get into research, this idea of, like, hey, why is it hard to do this fallback easily? Or, like, why are there false positives, right? But we are working, of course, on reducing the false positives.

1:21:53Of course, as the models get more intelligent, now they can do more things, you know, and they're, like, you know, like, what they can think about in lane space gets difficult. And so, like, as a more intelligent, there's going to be new false positives that we need to figure out, and we need to iterate, and things like that. But we're, yeah, we're working on this. And we do think this is, like, a critical part of, you know, like, deployment of these models. And, yeah, like, you know, it means that we can, like, deploy this model without you having a perfect sandbox or something, you know what I mean?

1:22:26Like, you don't have to, like, save everything. I think it's worth talking a little bit about our security, like, what we do for security. So there's, like, the model training stuff that we talked about. There is the probes and classifiers. And then there's auto mode that sits on top of all of that, which is, like, another classifier that checks the requests that are being done, right? And so, and then beyond that, there's, like, identity and permissions, like we talked about with Cloud Tag on, like, APIs and stuff. And so, there's so many layers of security that need to get done.

1:22:58And it's, like, like we said, very complex. Any of these failure modes at any one point can, you know, like, cause, like, agents to, like, you know, like, escape the sandbox, basically. Auto mode was an interesting one. It seemed early on, like, okay, it's running for 10 minutes. You know, if I'm on full access or auto, it's not a big deal. But one thing you brought up is now it's running for hours on end, right? There are fallbacks you still need. There are still limitations. Yeah, I mean, I think, like, everyone has these stories or, like, has heard these stories of, like, oh, like, Cloud RMR, or not Cloud, but, like, you know, models have, like, RMRF'd.

1:23:35I think I've seen this less for Cloud, but, you know, like, again, it can happen. Like, you know, like, these models, like, can wipe, you know, like, sensitive data or something. Like, you want to give models access to your production database, for example. But this is, like, an obvious, like, you know, you can maybe scope your key, but I don't know, can it issue its own keys? Can it, like, probably, you know, like, can it, it can use computer use to go issue its own key and then copy the key over and then edit your database because it needs to do it to complete the task.

1:24:08You know what I mean? It's just, like, one kind of trivial example. And auto mode sort of looks at that and be like, oh, no, the user did not give you permission to, you know, write to the database or to use computer use to, like, emit a task, right? And so this, like, probes are sort of, like, on the intent level, right? They're like, oh, okay, like, hacking artifactory is bad. Like, we probably not, should not do that, you know? But then, like, auto mode is more on, like, your own permissions level. Like, at some times you do want it to write to the database, sometimes you don't, right?

1:24:39And you don't want a probe to, like, interfere there. But, like, you need to make sure that the intent of what the agent is doing matches up with your request, right? And so auto mode operates at that level. And so, yeah, security is just, like, very, very complex. There are so many different parts to it. And, like, yeah, I hope that this is, like, my goal is really to just get very technical about it and talk. Yeah, we're listing out the things. If you're not aware, this is the standard now. Yeah. Like, you must have this, basically. It's kind of in line with what you're talking about with the harness.

1:25:09Like, that is the table stakes have risen quite a lot. I think some stuff that we can plug, you know, as much as there is probing in your side of doing this and having classifiers for people building harnesses. The other side is model safeguards, right? So there's open models. So Lama has Lama Guard. It's a safety classifier trained version of Lama. OpenAI has OSS Guard, which is, you know, same thing. You can attach these on to your harness, to whatever, to kind of, you know, check, is this stuff safe?

1:25:41A point that we should clarify on the OpenAI model hugging face thing is this was done with an unreleased model that was still in training, right? So when you put it in perspective, the prompt it's being given in the RL environment is sort of, you have to solve this task. And this is a model that's, you know, still in training. It hasn't had all of its safety post-training alignment. So a little different than something like Auto Mode, right? Auto Mode is on production models that have gone through safety training that have prompting that gives more safety guardrails and whatnot.

1:26:13So just breadcrumbs for people that are looking into it to, you know, fill in gaps. Yeah. Grace Wan as well, and one of our previous guests. Yeah. Lots of safety architecture and lots of safety vendors to buy. I think my final question on pacing is how long? Do we pace forever? Do we see? You know, the scope is fix all software in the world, right? Listen, like, which it's not happening. I do not know. I'll say one thing that's good that I think we do do is you have stuff like Glasswing.

1:26:49OpenAI also has this. So you will give it, you'll give model access for security first for X amount of time so you can use it to self-read team. Hopefully you can expand programs like that, help on, you know, we are safety experts, there's others. Solve your problems first and then the model comes out. So this is one example, right? Yeah, exactly. Trying to, like, secure critical software. I think we fixed, like, a lot of bugs in, like, Firefox and things like that.

1:27:20So, yeah, like, across, like, operating systems and everything like that. I mean, at a high level, it's just, you know, you give the model, you give people access to do security audits first, then the broader public that could use it for harm gets access. Yeah, I think what people like to say is basically, like, software and cybersecurity is defense favored and that, like, you could theoretically, it will be hard, but you can engineer the perfect sandbox, you know, and you can, like, have no, like, constraints.

1:27:50And, yeah, like, what you need to do it is you need to get the super intelligent AI to engineer this perfect sandbox and check it and red team it and things like that. And so this will just take time, you know, and, like, of course, the models will get smarter. Yeah, I think, like, I don't know the specific, like, dynamics of how this thing goes. I'm really just sort of, like, hey, like, I'm a developer, you know, like, I think this is how I understand this problem. It's just, like, this is what's happening right now and this is, like, we should do something. I think every engineer should know about it because, like, it's going to be part of the job.

1:28:24Yeah. It's a lot more than just, you know, Dario and people can say it and you can look at the incident. There is an engineering side to it. Yeah. Yeah, exactly. One thing that you also wanted to phrase is that this is actually, even though you're worried about the impact, it's still low-p-doom. I think that's a nuanced discussion.

Resilience and Future Outlook

1:28:41In general, people very easily get into AI safety and extra-risk discussions. But I think when you live in an AI lab, I think there are smart ways of discussing p-doom and dumb ways. So what's a smart way of discussing p-doom? Yeah, I have a fairly low-p-doom. I can only speak for myself. I mean, and I do want to say Anthropic has, like, a diversity of opinions. You know, I think, like, there's many different ways to talk about it. And, like, I think that just, like, my mental model is that, like, you know, I think we can collaborate on hard problems together.

1:29:17You know, I think nuclear proliferation is an example of how we collaborated on this hard problem together. And, like, that is, like, the thing to me is, like, I'm, like, I have faith in that, you know? And I do think it's a hard problem, you know? So, like, I think it's a hard problem. These are the technical reasons why. And I don't know how you assign probabilities to things happening. I think it's hard to do. But, like, my, like, overall is, like, yeah, I think we're very resilient and adaptable. And, like, sharing this information, I think, is, like, the first step. And I've been really, like, excited about, like, how broad the discussion has become, right?

1:29:50And, like, how everyone has sort of, like, leaned in on pacing the frontier. And it really didn't seem like this would happen maybe, you know, last year or something. Yeah. And also maybe curing cancer. Hopefully. Yeah, yeah, yeah. That's the goal. There's pacing and then there's also, like, well, let's accelerate in useful ways, right? Like biology and all those things. Yeah. I mean, like, Dario's essay on Machines of Loving Grace is the best representation of this, right? And I also agree, like, I think you should read the pacing frontier essay that Dario put out. Like, I put out, like, a quick summary, but I think it's just, like, you know, there is a lot of detail here.

1:30:22It's, like, an important problem and just being informed about it, right? But, yeah, like, of course, the whole reason we're doing this is that, like, we can get these enormous benefits, right? And, yeah, like, we've written a lot about that, too. Okay. That was a huge tour from, like, Ask Use a Question tool to AI safety. Yeah, to Pacing the Frontier. Yeah, no, but it's clear that you, like, really embrace everything that's available to you and topic. And, like, it's good to at least have a peek inside of, like, what the discussions are, the topics are.

1:30:55Any last words to people, you know, whatever you want to call to action? Yeah, I mean, I think it's, one, thank you for having me. You know, I think this is, like, I really... We're all talking, man. Yeah, yeah, yeah. We first met in a Chinese restaurant. That's right. Yeah, yeah, yeah. I think, like, I really enjoy the sort of, like, community you've created and the community of developers. And, you know, I think that, like, I sort of know things are changing really fast. And I think there's, like, a lot to keep on top of.

1:31:26And, like, I think there's just a lot to do. And I sort of feel, I think a lot of people feel, like, a little bit tired or anxious or something. Stressed, yeah, exactly. And this is, like, extremely understandable, you know. And I think we, I understand, like, and we're not perfect as well. Like, we, you know, it's, like, sort of criticize and, like, understand, like, you know, ways all of the AI labs could be better. And, but I also, like, I'm very excited about the excitement that everyone has for AI. And just, like, it's a really, really exciting time. I think we'll, like, look back at this time and be, like, oh, like, you know, this is, like, very hectic but very exciting.

1:32:00And, like, you know, software engineering changed, like, forever. Like, other things will change. And it's, like, really privileged to, like, be part of it, you know, like, to talk to, like, the audience that you have and get to interact with all the developers who are, like, pushing the frontiers a lot on what's possible. And I learn a lot from that, too, yeah. Thanks so much. Thanks.