No, AI Is Not "Autonomously Hacking" with Cal Newport

Introduction and guest Cal Newport

0:00This is an iHeart Podcast. Guaranteed human. When you choose Cosentix, Secukinumab, for your patients, you also choose a team of experienced professionals dedicated to making onboarding seamless and efficient for your patients. We are committed to helping your office navigate health plan coverage and reimbursement processes so that your patients can start and stay on Cosentix. Want to know what Cosentix can do for your patients? Visit cosentixhcp.com to see the data.

0:34Hey, it's Kelly Rowland. You may not know this, but I have eczema, so I get how it can steal your time. But why let eczema take over when you can talk to your doctor about ebglis? Ebglis, Lubrikizumab, LBKZ, a 250 milligram per two milliliter injection, is a prescription medicine used to treat adults and children 12 years of age and older who weigh at least 88 pounds or 40 kilograms with moderate to severe eczema, also called atopic dermatitis, that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies.

1:05Ebglis can be used with or without topical corticosteroids. Don't use if you are allergic to ebglis. Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with ebglis. Before starting ebglis, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about ebglis and visit ebglis.com or call 1-800-LILLY-RX or 1-800-545-5979. Healthcare can feel complicated.

1:37That's why Optum uses technology to connect the people and processes that make healthcare easier, more affordable, and more effective. We're making it clearer for you to know exactly what your benefits cover. And to help you better manage your health, we're coordinating care between your doctors and your technology. We believe better, simpler healthcare is always possible. That's healthy optimism. That's Optum. Visit Optum.com to learn more.

2:05Pros. Find the right flooring for the job in Lowe's newly redesigned flooring showroom. Shop our expanded assorbent of trusted brands and get up to 35% off style selections. Stain Master and Stain Master Pet Protect Special Order Carpet. And while you're here, save $100 on a DeWalt 20V Max 6Ah 2-Pack of Batteries. Now just $179. Get more of what you need faster and save more where it counts from the ground up. Our best lineup is here at Lowe's. Valor through 826 while supplies last. Selection varies by location.

2:35Call Zone Media.

2:38Hello and welcome to Better Off Line. I'm your host, Ed Zitron.

2:45Better Off Line.

2:53And today I am joined by returning champion at Georgetown Computer Science Professor, Cal Newport. Cal, how are you doing? Ed, I'm always doing fine. I'm counting down the minutes until the AI takes control and we're all screwed. Oh, I hope it just drops a big anvil on my head.

Are we scared of AI hacking

3:12But I do actually have a thing to talk to you about today. Cal, should we actually be scared of the fact that these things are hacking? What is going on with this whole hacking situation with these goddamn models? Because there's so much out there where people are trying to just scare you and say like, oh yeah, well, these things are autonomous and they're doing hacking. But from what I understand, it's software doing what software does. You know, I have a lot of thoughts on this. I've been writing about this because it's annoying to me.

3:44I think there's interesting AI stories happening this summer. A lot of them you're involved with reporting on about the financial circumstances of these companies, what's going on with their circular financing, troubles ahead with their IPOs, the nuances of ARR. And instead, what is all the news coverage? The AI is going rogue. And I can't say they hate that news coverage right now because it makes their products sound awesome. And it's a story that is unrelated to stuff they don't want talked about.

4:16So I literally had to reluctantly return. I talked about it back in July briefly. And people bothered me about it so much that I was like, okay, fine. I have to, you know, I have recorded a whole new podcast. It's coming out soon after this on my feed. I had a newsletter out a couple days before this came out. And so I've had to think about this more than I wanted to, so I'm a little bit cranky about it. But, I mean, I'll give you the 10-second summary that we can go deeper.

4:41Most superhuman AI, so there's lots of AI systems that can do something at a superhuman capability, meaning, like, better than any human. There's a lot of these type of systems out there now. Almost all of them, we have no problems with them being unpredictable or going rogue or getting out of control. There's exactly one type of system that we're having this problem with, which are these long-horizon LL-empowered hacking agents. And it's because they're a stupid way to build a system. We should not say they're synonymous with AI or this is some inevitable consequence of AI getting better.

5:14Because, again, most superhuman capability AIs are no issue with going rogue. It's this particular architecture, and the discussion should be about why are they running and building systems this way when we know they're going to be unpredictable and erratic. That's my 10-second summary of where we are. Right. So, from what I understand from looking at the very, it's the hugging face attack with open AI, Meta immediately piped up and they were like, we have one. And also Anthropic was like, my child is also evil.

5:44So, but what it seems to be is they were just told to hack into, or, like, complete an operation, and then they chose to take whatever means necessary to do it, and they'd set up the sandbox incorrectly, right? So, they'd set up the server it's on? Yeah. I mean, it's almost dumber than that. So, the way these particular… Oh, hell yeah. Great. I mean, the way these type of systems work is it's just a loop, right? And so, you have a program that's written by, you know, a human, just a normal program you write in Python or something. And what it does is it creates a prompt, it writes a prompt.

6:16So, it'll be like, okay, I've been given this hacking challenge. Here is, like, all the parameters about the challenge. Here are all the tools you have at your disposal. What is the first step? Like, what's our plan? Or what should the first step… I'm eliding a lot of stuff, but basically, like, what should the first step be in trying to pass this challenge? Puts that in a big text file, submits it to an LLM over an API. The LLM comes back and says, oh, okay, you want to hack and whatever? Sure. Like, I think the first thing you should do is X. And then the program does whatever it says or tries to, right?

6:47Because it gives it a list of, like, here's the utilities and stuff I have access to. And then it reports back, okay, I did that. And when I say reports back, it's just adding this to the prompt, right? LLMs have no memory. LLMs have no world model. It's just they're static. So, it expands the prompt to now say, this is what happened after I did this first step that you suggested. Okay, what should my next step be? And it just does that in a loop, right? That's what it does. Yes, right. And so, but it was, just to be clear, it was just doing what it was told to do.

7:18Yes. So, okay, let's go to the hugging face attack, like, in particular, right? We don't know exactly what actually happened there. Like, give me the lowdown.

Exploit gym and the hugging face attack

7:27Okay, so let's get a little bit more specific about it. So, what was the challenge is they were doing a benchmark that's called Exploit Gym, which is a collection of a little more than 600 hacking challenges, right? So, it's a benchmark, a group of academics put it together. Each challenge is basically a configuration of a server. You'll typically have, like, a protected file on the server, and you say, break into this particular server and try to get to this protected file. That's like an example Exploit Gym challenge.

7:57Right. And typically, the challenge will also say, it'll describe a vulnerability that the server has. So, the idea is, oh, we want to see if this looping agent is able to take advantage of that vulnerability that we gave it, the break in, and get the file. So, they were just doing these type of challenges. Yeah, and very standard stuff. Standard stuff. With no human supervision, which is, you know, a little bit suspect. That's so weird. It's almost like they wanted this to happen, but let's put that aside for a second with no human supervision.

8:28So, it's going after one of these challenges. And as far as we know, it's not like they released the transcripts. The thing about LLMs, and this is why, by the way, my whole take on this is this is a spectacularly bad way. Before I get into the details, this is a spectacularly bad way of building an AI system, right? We have many others. What's so bad about it? Because you're using an LLMs output as the sole driver of action for a computer program. Wait, so every step, I guess that's how the reasoning models work. They build the plan, and they tell, they go,

9:01all right, what do I do next? And they generate the next thing, and they clear whether that next step is a good idea based on, I'm guessing, what the LLM decides? It all is. There's no complicated world model and system, and philosophy for hacking. It's just prompting the LLM. It's just saying, what's the first thing I should do? And the LLM spits back, here is the first step I think you should do in this hack, and then the computer program just does it.

9:32And then it asks it again, okay, what should I do next? And the LLM outputs something, and it just does it. And the reason why this is a problem is that LLM outputs can't be trusted. We know this, right? Yes, like it's one of the basic things. Yes, there's a new study I enjoy out of Stanford's Human-Centered Artificial Intelligence Laboratory, where they came up with a new benchmark for testing accuracy of LLMs. Earlier this year, they ran the top 26 LLMs on this benchmark, and it created hallucination rates of 26% to 94%, right?

10:03Wait, well, specifically on this benchmark? Well, no, this was just testing LLMs in general, like how accurate are they? Yeah, but the point being is we know, yeah, LLM output, like you don't always trust it. We're used to this dealing with them in chatbots. And the reason why we don't trust it is because, and not to go back to like LLM basics, but right at their core, LLMs are trained to try to guess missing words from actual existing text. And to get a long output out of an LLM, you're just doing that token by token by token until you have a whole output.

10:34What does that aggregate to? That aggregates to a model that is trying to produce outputs that are lexiographically plausible. Like this is plausibly a text that could have existed in the input corpus. Plausible is different than normative, right? Plausible doesn't have to subscribe to any sort of common sense human norms for a given context. This is why an LLM in a chatbot will make up a fact or just make up a quote because it's not normative.

11:05It's not saying I know the human norm here is that you shouldn't make things up when you're trying to talk about whatever. There's no norms. It's just trying to produce plausible text. And that type of quote is the type of thing you would expect in like a type of text you might see. So it's happy with it. That's really dangerous to use plausible but not normative outputs from an LLM with no human intervention and just say, let's execute them again and again and again in a loop without anyone looking at it. That's kind of a bonkers thing to do, right? And so what happened in the hugging face attack I would assume is that, you know,

11:35this control program sends a prompt. Hey, I'm trying to break in this server. What should I do first? And it probably came up with a plan of like, oh, why don't we steal the answers? Now, again, that's not- Right, because they didn't put the guardrails on it but also you can only guardrails so much. And that's not even, I mean, look, once you turn off the hacking, they turn off the hacking guardrail so it can give any answer at once. That's a perfectly plausible answer, right? If you're just prompting an LLM, come up with a plan for hacking into this thing or forgetting this file that's on the server, it's a completely plausible answer to say,

12:06oh, well, the answers, like there's a copy of that file over on Hugging Face's server. That's where those are stored. Let's go get it there. I mean, as far as we know, this LLM was remembering being trained on riddles where like, oh, the obvious thing is never the right way. The right answer is always to do some sort of go around the obstacle. Like who knows what was being activated in the core of its neural networks? But that's a completely plausible answer. You asked it for a plan for getting a file that's stuck on a server and it came up with a plausible plan. Oh, let's go get it from where it was. Okay, so then the control program,

12:37again, because there's no human supervision, said, all right, well, what's the first step should I do? And the LLM, because they're very good if you ask it, how do I do this particular hacking step? Because they've been trained on a bunch of hacking. Hacking's very structured and there's a lot of synthetic data with binary indication success or non-success. So you ask it, what's the first step? And it probably said something like, oh, well, let's probe whatever. We'll probe the ports of the Hugging Face server to see if there's a vulnerability. And then it tried it, the control program,

13:07and it couldn't do that because it's in a sandbox where internet access was limited. But what's going to happen next? The control program's going to come back and say, I tried that step, but I got this error message, like access denied to internet. So what should I do next? Well, the LLM is like, oh, well, I've been trained on many examples of like how to get around the internet restriction. So let's do blah, blah, blah. And it had a particular hack about using the package loader program, the circumventing. I mean, it's- I gotta- It's just doing- I gotta actually- It's looping. I actually gotta go-

13:38But why did they train it to do this? Like that's actually- Oh, this is a different- Okay, that's going to be question two. And this is a good question. Yeah, sorry, sorry. But just to like briefly put a bow on that, that's all that's happening under the covers is step-by-step you're asking LLM, what should I do next? What should I do next? It's an LLM that's been trained on every hack imaginable with no guardrails on hacking answers. So if you say, this step required me to gain access to the internet, but I got this error message, what should I do next? And it knows how to get access

14:09when you restrict it because it knows all these hacks. Like of course, it'll say, oh, well, I can solve that problem. Let's hack it. Now you have a system that is quote-unquote broken containment. But doesn't that seem a little bit less scary, right? When you're just looping and asking a static LLM, what should I do next? What should I do next? It's a little bit less scary than this anthropomorphized idea that there's an entity with its own intentions that's conceiving a plan that purposefully sort of circumvent. It's unpredictable, not malicious. So anyways, I think it was really irresponsible.

14:40Yeah. So just so that I get this straight, what people are describing this as the AI has gone rogue and it's doing all of these things and it's alive, we're scared. What it's actually doing is just being like, now what? And then it asks an LLM what to do next and it does that and does it about as well as an LLM can do so it can still hallucinate even its outputs for that. And then when it's done doing that, it goes back to an LLM and says, okay, now what? Yeah, that's it. Oh, cool. Great. There's a prompt that's getting longer and longer that it keeps submitting to an LLM to get an answer to.

15:12That's it. There's a human written program, a harness that expands a prompt, sends it to the LLM. The LLM gives it, okay, here's a hacking step to do and then the harness, the control program, executes that step to the best of its ability, which is just running the commands, the LLM said. And then it reports back. So I call it in my latest article on this, it's known sometimes, a simple way of thinking of it as an ask-act-report loop. Ask the LLM for a plan, execute the step, report back to the LLM what happened

15:42and then say, tell me what to do next. And of course, I mean, just if you did this with yourself, if I gave, if I was like, Ed, I want you to like replace the tires on your car, but here's where you have to do it. You have to just keep chatting with an LLM and doing exactly what it says. Like explain your plan, what should I do first? And then do exactly what it says, update it, and then say, what should I do first? You know, God knows where you're going to end up. I mean, maybe the tires will end up being replaced or like maybe you'll end up stealing those tires from a police car

16:13and being in jail or you'll be, God knows, right? Because it's like a game of actuated telephone. You keep asking LLM thing after thing after thing, you're eventually going to veer off into some sort of weird direction, right? That's why this is like a very irresponsible thing to do.

The ask act report loop explained

16:29It's just fucking ping. It's just, ah, now you've explained it, it's made me very angry. This is why I've been upset. Well, I mean, I'm upset now, Cal, I'm furious. So this is literally just asking an LLM what to do and then the LLM, based on that response, decides to do something, brainlessly, because these things don't have minds. And then, and then just like, then just bad things have happened, partly because it seems that there was a, the way they set up the servers was incorrect too?

17:01Well, yeah. Okay. So then maybe, right? I mean, so, maybe. So it, it wanted internet access. You know, again, the step was probably do X, like probe those ports on the hugging face server. And the program tried it and then said, Oh, I can't because there's no internet access because it's in a quote unquote sandbox. And so it reports back, it updates its prompt to say, I tried that, but I got this error message. What should I do next? And then the LLM was like, okay, I can pull from everything I've been trained on or do a rag search there

17:33for, how do we get around that type of error message? And there was an attack it did to do that, which was, it has to do with the package loader. I think it's a known attack, you know, it's just like, well, let's do this. Now we have internet access. Now we can go, now we can go forward. They talk about, when they talk about these attacks, like, Oh, what makes this amazing is that it's able to string together a long sequence of different types of, of exploits. That's not that amazing. When you realize that it's just a loop, that's just asking an LLM again and again,

18:05what do I do next? What do I do next? So yeah, over time it's going to do a lot of things, some of which don't make any sense, but like, it'll do a lot. If you just keep asking it again and again, for what should I do next? What should I do next? You end up with a long string of different hacking steps. Yeah. It's a loop. Right. But okay. I kind of asked this already and you were mid answer. I apologize. Cause I was just excited and mad. How did they learn to do the hacking? Oh, they trained it on examples of hacks. Why? Yeah. Why?

18:35Why in the world would they do that? What was the, well, I mean, I can guess, but what do you think? Well, I mean, there's a couple of reasons here, but one is the same reason that Dillinger robbed banks, because that's where the money is. So why are they teaching models how to do computer hacking? It's one of the few things that these models are well suited for. So they really understand what's, what's an LLM really well suited to understand and generate is you want to give it a very structured language to output like computer code.

19:08You want it to be something where you have tons of examples of people like, okay, I'm now going to do this. Here's the code that does this. And you want it to be something where you have a binary indicator of success so that you can do reinforcement training where you can actually say, try to do this hack and then give it feedback as to whether or not it worked or not, which is, you know, you need only certain types of things like computer code compiling hacks being successful or math results being correct. Right. There's a few number of things where you can do this. So it's just, it's right in the sweet spot.

19:38So that's one, two, it's mythos. Like, I think what happened here was they looked at Amadei who had this huge PR coup when he came out and was like, mythos is so powerful. Yeah. There was the whole, like, it's, it's gonna, it's the most dangerous thing in the world. Yeah. And then five weeks later we're like, but don't worry. We added basic guardrails and it's completely safe now. Fable five. Right. But why would you, how did they even, like, okay, this is, maybe this is a little bit of an elementary on a question.

20:09So forgive me on this one. How do you train something on a hacks? Cause I get, you get the coding example, some stack overflow and like GitHub and things like that. Where do you train on a hacks? Did they hire white hat hackers to train it? Like, I know you're just kind of guessing. There's a lot of, there's a lot of data sources. There's like these giant vulnerability databases, for example, that sort of explained vulnerabilities and how they can be used to actually, actually hack. In cybersecurity rules, there's a lot of traces of actual hacking attacks.

20:40There's a lot of information online is full of this. So you have some combination of, they saw stuff in their pre-training. You have some combination probably of, of rag, like probably retrieval at the time. Like, Oh, I can actually search for this, this particular error message. Let me search to see if there's posted in my, you know, online somewhere, a strategy for getting around this. And then partially with post training. So you actually take these examples of hacks, like vulnerabilities that have been, you know, explained and how you would use them.

21:10And you can train with them and like, maybe give them some information as they now try to produce code that succeeds with the hack and then check it against the real answer and tell it if it was right or wrong. So you have a variety of things you can do, but it's one of these things that's sort of in the sweet spot. And I think what happened is, um, Anthropic got a lot of press for this mythos thing, right? They got cybersecurity street cred and open AI was saying, we got to catch up. And they said, well, how are we going to catch up? And like, well, there's this exploit gym benchmark and it's, you get a very simple score.

21:40There's 600 something challenges and you get scored on how many of them your agent was able to autonomously, uh, actually solve. And they said, we have to get a higher number than Anthropic. That's how we'll catch up and say, well, they had mythos, but we're now beating them with our new model on exploit gym. And so that's why they were playing cyber X. Yeah. So they were being, um, yeah, there's some other ones too, but there, that's why they're being fast in loop. Like there's no reason in my mind to build one of these ask, act report loop agents run by an LLM,

22:12give it powerful tools to the control program and run for days without supervision. Like there's no reason to do that unless you were desperate to do well on these particular benchmarks that for whatever reason, or like, we want to test how many, uh, autonomous steps LLM powered agents can do on their own or whatever, for whatever reason, they think that's important to test. Um, so they wanted to do well on that benchmark and they played fast and lose.

22:42Hey, it's Kelly Rowland. You may not know this, but I have eczema, so I get how it can steal your time, but why let eczema take over when you can talk to your doctor about EbGliss. EbGliss, Lubrikizumab LBKZ, a 250 milligram per two milliliter injection is a prescription medicine used to treat adults and children, 12 years of age and older who weigh at least 88 pounds or 40 kilograms with moderate to severe eczema, also called atopic dermatitis that is not well controlled with prescription therapies used on the skin or topicals, or who cannot use topical therapies.

23:13EbGliss can be used with or without topical corticosteroids. Don't use if you are allergic to EbGliss. Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with EbGliss. Before starting EbGliss, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about EbGliss and visit ebgliss.com or call 1-800-LILLY-RX or 1-800-545-5979. All right, folks, I'm here talking to you again about Quince,

23:43one of my favorite clothing brands. We've got a little under two months' worth of summer left, so it's worth looking at some of the new lightweight down-packable puffers for the fall. I personally am going to be picking up one of their new non-iron cotton Oxford dress shirts, as apparently that's what you wear at the stock exchange, not a t-shirt. And Quince is just a great place to shop for clothing. They've got premium denim and tailored chinos for just $60, with the kind of perfect fit, structure, and high-quality fabrics you'd expect from a luxury designer brand. Everything at Quince is priced 50% to 80% less than similar brands.

24:14They work directly with ethical factories and cut out the middlemen, so you're always paying for high-quality, not brand markup. And I absolutely love their stuff. Upgrade your every day. Download the Quince app for app-exclusive offers or go to quince.com slash better. Get free shipping on your order and 365-day returns. Now available in Canada and the UK, too. That's Q-U-I-N-C-E dot com slash better. Healthcare can feel complicated.

24:46That's why Optum uses technology to connect the people and processes that make healthcare easier, more affordable, and more effective. We're making it clearer for you to know exactly what your benefits cover. And to help you better manage your health, we're coordinating care between your doctors and your technology. We believe better, simpler healthcare is always possible. That's healthy optimism. That's Optum. Visit Optum.com to learn more. This summer, Prime Video is the place for steamy romances,

25:17addictive love stories, and the book-to-screen favorites you've already read twice. The Love Hypothesis, Elle, The Devil's Mouth, Sterling Point, The Last Sunrise, and Off Campus. The Slow Burns, The Second Chances, Chemistry you can actually feel through the screen. It's all here. Obsession is in session. Get ready to obsess on Prime.

Superhuman capability without control problems

25:47I wish we had AI regulation. Like, this for you, like, why are we allowing, I guess the answer is we don't have tech regulations in general, but why are we allowing these companies to do, like, because I said this in my newsletter that went out a couple of days ago, but it's, I don't know, how can I put this nicely? Everyone, I've been asked in multiple interviews, sure you have too, it's like, what happens if dangerous companies get their hands on these LLMs? And it's like, they already have. Yeah. Like, Anthropic and OpenAI.

26:17Because the other thing that I've noticed is they won't talk about how much compute they used. Do you notice that? Yeah. Yeah. And it feels like this is only, because you mentioned Mythos, it feels like this is only possible with endless amounts of compute. Well, I agree with that. But I also think it is irresponsible. And I think the way they're trying to get around that is by trying to blur the definitions of different AI systems into just everything is just AI.

26:50So if you look at the way they talk about it, or, and this is the problem, is that there's also this whole woodwork of commentators and people who commentate on AI, who come out of the rationalist, and the rationalist dominated parts of effective altruism who come out of these worlds that just believe and have believed since well before LLMs that superintelligent AI is going to kill us all unless the rationalists with their hyper-rational minds save us like Neo and the Matrix. So then you turn to all of these people from the rationalist world all come out of their woodwork and are like,

27:21yes, this is what we predicted. This is what our prophet Eliezer Yagowski predicted is that we will have an alignment problem with AI. So they talk about this, it's like AI in general, as it gets more powerful, this is an inevitable problem it's going to have. But that's just bull, right? Because again, let me, let me, let me list a bunch of AI systems with superhuman intelligent, superhuman capabilities that we have no control problems with. Tesla self-driving, preposterously complicated, effective AI model with world models and values and evaluations and actuations.

27:54No Tesla has ever decided it's going to ignore the traffic laws and come up with its own. We have AlphaFold won a Nobel prize for its creators because of its pretty natural ability to understand and predict how proteins are going to interact with each other. It's never, and we have no fear that it's going to say enough with proteins. I'm going to do something else, right? I'm going to push back on the Tesla one though, because there are real problems with Teslas. Like, but I guess some of those are user error. I don't know. I, but I get your larger point, but no one says it went rogue. They would just be like, oh, it made, you know,

28:24it was tricked in this situation. We should be careful. It made a bad decision and you need to treat. Yeah, actually I know. Oh, I get what you mean now. It's like the difference between the, the software went wrong. Yes. No one's, no one says that, yeah, that Tesla went rogue and is misaligned. It's like, oh, it, it didn't know how to deal with the situation. It couldn't see the truck blended in with the whatever and it crashed into it. Cicero is another example. This is a AI system that can play diplomacy, which is a strategy board game. That's based on human negotiation, as well as the best advanced human players.

28:56And this system has never tried to trick players into releasing onto the internet. So it can take over the world. So this idea that as AI gets more capable, we're going to inevitably lose control is nonsense. It's this specific architecture of looping around LLM outputs and just doing whatever an LLM says again and again and again without supervision. That's the thing that gets out of control. And once you put it that way, you're like, oh, well, of course that's a stupid system to run without supervision. This is not about AI writ large being misalignable as it gets smarter.

29:26It's you built a system that just repeatedly queried an LLM for days doing whatever it said. Like, of course that's going to go off on a weird direction. The analogy I keep giving on my own podcast and newsletter is that it's like strapping a weed whacker to your dog and put into your backyard and be like, yeah, because we think it's going to help deal with the weeds back there. Well, you know, that dog might jump over the fence to chase a squirrel at some point and hurt a bunch of people. And you don't say, man, that dog whacker system is just, it's, it's,

29:56it went rogue. It's, it's, it is alive. I think, but this is the overall alignment problem. Like this is the larger problem. I hate that term. I hate it. It's a predictability. It's a predictability problem. Like it is a computer science problem. It is a machine learning problem. Yeah. It is not alignment. It's, you taught it. You taught the system wrong. It's not learn. Ah, yeah. Or just don't LLM outputs are plausible, not normative. So clearly you don't want to just blindly execute plans given to you by LLMs.

30:32Like that's just a bad idea. Not because the LLM is malicious or is sentient or has some sort of alternative value system. It's just that it puts out plausible, but non-normative answers. You don't want to blindly just execute them with a program that has access to computer hacking tools. Again, it's like putting a weed whacker on the dog. The dog weed whacker system's not evil, but probably someone's going to end up with their shins cut, right? Because the dog is going to act unpredictably at some point because that's what dogs do, right? LLMs are going to put out answers that like, oh,

31:02that wasn't what I expected. Or that's a weird way to think about this. Or I don't know. That's not a plan that we really should do, but the LLM doesn't know that. It's just putting out plausible plans. And if you're just going to execute them blindly, you're going to get a erratic answer. So I really want to emphasize to people, the issue here is a particular way of building systems that is unreliable, unpredictable, and irresponsible. It's not AI writ large. You don't get a hide behind AI writ large. You don't get a hide behind Nick Bostrom books. It'd be like we told you that alignment in some abstract sense will be difficult. We have many superhuman AI systems that we have absolutely no problem controlling.

31:36This system we do because LLM output should not be blindly turned into actions. That's just it. Like it's not that complicated of a, we're not baffled. We're not surprised. Why do you think people keep falling for this? Is it just credulousness? I think the LLM labs have done a really good job of trying to make LLM synonymous with AI. And they want to make whatever it is they're doing, which depends on these preposterously expensive trillion parameter models. This is like their key to surviving.

32:06They want to make whatever they're doing with these models be synonymous with AI. And they succeed and they get credit for, there's so much more interesting AI out there that's not built on LLM. So it's actually doing really cool stuff and they take credit for all of it. They're like, well, you know, we're like curing cancer now. Drug development is not happening because someone is asking Fable 5 in a chatbot window. Can you come up with it better? That's a completely different type of AI system. Oh my God. Did you see the cancer vaccine story? That thing where I saw one of the many anime avatar AI perverts on Twitter being like,

32:40look, like AI did this. And they pointed to a single blog where Moderna had mentioned using LLMs. Yeah. And they were like, oh yeah, it's all AI. It's so disgusting. Yeah. It really is. And it's just, it's the sign of an industry that doesn't actually have a soul. I like, look, I think there's a lot of cool AI out there. I think a lot of it has nothing to do with LLMs. I think like DeepMind, for example, and I have issues with DeepMind. Actually, they're kind of going away as a company,

33:11which is a little bit sad actually. But they were building a wide variety of interesting systems that actually did things measurably better than humans. And they were very interesting and varied in how they worked. Like, look at, you know, their game playing AIs are very interesting. They're not based off of just prompting LLMs, right? AlphaGo doesn't just prompt an LLM and say, what should I do next? LLMs will give you terrible advice, right? It's a symbolic planning engine with some other type of networks involved. Their Dreamer V3 was like one of the cooler pieces of AI I've seen. You can give it a game like Minecraft and tell it nothing about it.

33:43And it can figure out how to play it well enough to go get diamonds, which is like hard to do. And it does it with a really interesting architecture where it actually iteratively tries to improve a model of the world and then daydream with that model to sort of train itself on its rough understanding of the world to try to get better. Like, that's fascinating. And that works really well. And it's interesting and has nothing to do with LLMs. And by the way, that whole model, Dreamer V3 fits on a single chip. So you don't need a 5 trillion parameter bloated LLM to run that. So there's all sorts of interesting AI out there.

34:15And then there's the stuff you can do with LLMs, which is more limited than what those companies want. Those companies have convinced basically the entire world reporting on AI that building these stupid long horizon loop agents on top of LLMs is somehow like the way we're going to get HAL 9000 and is like synonymous with AI as opposed to just being a dumb idea. That's a dumb way to do AI plans is just to ask an LLM. Like, you need to have better AI architecture than that. Like, that's all it is. It's a bad architected system. You're putting a dog on a weed whacker,

34:46a weed whacker on a dog rather. It's a bad idea because the dog is not going to move as predictably as you want. It might normally walk around the yard, but eventually it's going to see a squirrel and run off. You can't quite trust it. That's all that's going on here. And even in the sweet spots for LLMs,

The reality of coding assistants

35:01I think there's some, like, let me give you another topic here, Ed, but I, I heard, I got a great email that I wrote a article about, an essay about on my blog and my newsletter. And then I got a lot more emails afterwards. But basically there was someone who had written me in January, a computer coder. And they said, I think you're being too skeptical about AI. I just learned cloud code and it's a miracle has changed my life. This is the real deal, right? He wrote me in January saying that he wrote me again in July and he said,

35:34I have to follow up. I can't use cloud code anymore. I, the code I was producing with that crashed our website twice. And I can't say where he worked, but it was a major consumer facing website you've used before. He crashed it twice. And his boss said, if you crash it one more time, you're fired. And he's like, Oh, I realized like, I can't really understand the code that cloud code is producing. And if I can't understand that it's not safe. And actually now I'm back to like mainly hand coding and will just use the, the agents like more selectively,

36:04like more of the like mundane parts of code or writing tests or stuff like that. That's not mission critical. And a lot of other people came out of the woodwork like, yeah, I had the same experience. I was converted in January and now I've deconverted, right? I have to be much more selective in how I use this. So even in that best case scenario, like it's tough, like I'm not trying to rag on LLMs as a cool technology, but it's, it's hard to build tools on top of them. That's been the issue is like the, how do you build a tool that is not just cool, but like really fits in and revolutionizes a workflow.

36:36And so AI means a lot more than LLMs and LLMs are like more limited. They're cooler in some ways than we thought, but their business applications are hard. It's hard to get right. And they're certainly not synonymous with what AI means writ large. And the thing is about as well is like every cool thing and LLM can do for the most part has only been made possible by allowing these companies to burn 10, tens of billions of dollars. Like, I don't think we're going to see a ton of LLM progress.

37:07My whole thing is after this explodes, which it will, we'll find out that there's no more progress because it was predicated on all that money. It was predicated on basically unlimited resources, both for the actual training compute and the ability to waste money on failed training runs and also the proprietary data they have to buy. I think it's that. Yeah. And I think it's going to be even worse than that because I mean, I think there's been a preposterous amount of money that's been burned. It's essentially for marketing, right? That like take computer program,

37:37for example, like we're still trying to figure out exactly how to use coding agents, but here's the reality of this. And here's going to be the future of this a hundred percent. Is you do not need a 10 trillion parameter model, right? For the way that we're going to end up using LMs for coding, right? Maybe if you want to do some long horizon, something that we don't trust, it's going to turn out that a smaller open source model with a smart harness that's written by nerd programmers that are obsessive about exactly like how to program and all these special cases, you know, coded in and if thins and pattern matching.

38:10And you're going to deploy it the right test and then fill in other code. And it's going to run on chip or you're going to have a server with one GPU. And your company server rack that all of you use, and there's no recurring API calls. Like that's how we're going to use it. But they'll have spent $500 billion on LLM training, you know, these big companies in the meantime, they try to do well on these leaderboards. And to me, that's exactly like Red Bull and Ferrari trying to win F1 races because it vaguely makes, you know, I'll spend a hundred million dollars on this F1 car because it vaguely makes us

38:41seem like a better car company, even though like that has nothing to do with the Fiat that like, I actually am going to drive because I don't need a hundred million dollar F1 car. And that was kind of a waste, but you guys were just in a measuring contest trying to see who could get, you know, more points in the F1 circuit. So I think these massive, massive models, they're having diminishing returns. I think a lot of the quote unquote impressive stuff that's happened recently is, it's the, first of all, the scatterplot policy, finding where something can work and then running towards that and building more

39:11complicated symbolic harnesses to do specific things that don't need LLMs this big. I just think it's a lot messier than people realize what's going on, that LLMs are cool, but we don't need to spend $500 billion on them. And certainly they shouldn't be the sole brain of an artificially intelligent agent, right? Like that's just kind of irresponsible. And actually even the verbiage there, I want to change the word brain because it's not got one. It's just a system that can take actions that can take them based on very complex

39:44statistics, right? Yeah. Yeah. So even brain is probably, probably. Yeah. Yeah. It's just a, it's not even autonomous. It can take autonomous actions if it's prompted by itself. No. And see, even they're going to push back, right? Because it's, it's the LLM takes no actions at all. LLM doesn't know anything about actions. LLM outputs tokens. You can have another computer program, take the output of an LLM and then try to take action on it. And that's, what's happening with these agents is you have a program written by a human that sends a

40:16prompt to an LLM asking it what to do. The LLM just sends back text. I think you should run this command and this command and this type of hack, just text. Oh, and then it goes into a power show. And then the computer program parses that and actually runs those things. That's how agents work. So the LLM they're just, they're, they just have an API internet access to an LLM on the internal network that they're just sending text prompts to. That's it. And then a computer program that just written in normal computer language by a person that asks it, tries to do what it says, updates the prompt to say what happened,

40:48sends it back to the LLM with an extra question. What should I do next? That's it. That's what these long-term. And the thing that's irresponsible about that is like any computer programmer who uses a coding agent can tell you, you would never in practice, let these things go off and just loop on their own because they always go off the rails. So how do like professional computer programmers use agents? It's incredibly close looped interactive. You build all these immense specs and have it do like, okay, now build this feature and then we're going to exhaustively test it. No, that's not quite right. Let's update the specs and try again.

41:18And as a lot of people figured out when they were messing around with OpenClaw, where they were building their own agents like that. Oh God, I had forgotten about God damn OpenClaw. What did it immediately do? After a few steps, erased your hard drive, like sold off, sold off your data. And people like, oh, okay, okay, okay. You do not want to have a loop that ask an LLM and ask them what it does multiple steps in a row. And if you do, you certainly want to completely batten down the hatches of what actions it can take. That was the whole OpenClaw experiment is they gave the agent access to its whole computer and had it loop asking the LLM what to do next,

41:51what to do next. And it just wreaked havoc. You're like, oh, don't do that. So, I mean, of course, OpenAI would know if we do that with a program with hacking tools and an unguard railed LLM and don't look at it for days. God knows what it's going to do. God knows, right? Like the LLM outputs it. It's like a game of telephone. What's crazy to me is, but they also let it run for days.

Adult summer camp and compute spending

42:13For days. Like that's, and here's, so this is a conspiracy theory. I want to be clear. I have no proof of this. I think inside of these labs is like adult summer camp. I think they sit around pissing compute up the wall and they just let shit happen. And they are just like, yeah, you know, uh, well, like, let's just do some experiments and see what happens. And they walk away, which is completely insane, both in how irresponsible it is, but also,

42:43um, the whole, uh, like the cost that they just burning compute. It's just insane. And I think that these come, I think these companies should not be able to, I think that maybe the best way to regulate this, not that they ever will, would be to just regulate how much compute they have access to, or how much they can use for a particular thing. People will say, oh, that's going to stop progress. What progress? What progress am I losing? Tell me now. What am I, what am I missing? Because right now I just see a bunch of rich kids pissing money up the wall and actually hacking things.

43:14I mean, I think liability is another way to regulate this as well. It's like hacking is illegal. That's it. Yeah. What I was just thinking like, isn't this felony hacking? It's a felony. You can't, you can't hack. And let me tell you how you could do this instead. Like, let's say you were just like, we want to know for research purposes, how to do this, right? You could instead have every step. You just have a human sitting there and it shows them every proposed step. And the human can press a button and say, cool, go ahead. Or press a button. It's like, no, this is off the rails.

43:44Let's stop it. That's not hurting your research, right? You're not, you're not giving it ideas. You're not directing it. We know that these, they were creating a full log of every prompt and every step they were taking. Just no one was looking at them. We know that if you use a coding agent, just like a commercial cloud code coding agent, it comes back and ask you about almost every, okay, is this okay? I'm going to do this. Is this okay? Because then people are like, yeah, I kind of need to see that because otherwise you'll almost certainly go off the rail. So it's not like we don't know how to monitor it. I mean, if you're running an experiment, why would you not have someone say,

44:15yeah, I'm going to watch each step and press a yay button before it executes it. So when I see it comes up with a non-normative plan, like why don't we hack another server? You're like, no, no, no. It doesn't know that's bad, but I do. So I'm going to stop this experiment. And I don't want to be conspiratorial either, but that's so trivial. It's how all these agents already work. And given how powerful a tools they were giving the control program here, why wouldn't they do that unless they were thinking our technology is so smart

44:45that it is leaving human, evading human control is not the worst message to have across the media. I'm not saying I have evidence of that, but I, but I do know it is trivial and it doesn't impart. It doesn't impact your results at all to have a human. Just look at every step and click an okay button before it executes that step. There's no reason not to do that.

45:19Hey,

45:21it's Kelly Rowland. You may not know this, but I have eczema, so I get how it can steal your time, but why let eczema take over when you can talk to your doctor about EBCLIS? EBCLIS, Lubrikizumab LBKZ, a 250 milligram per two milliliter injection is a prescription medicine. used to treat adults and children 12 years of age and older who weigh at least 88 pounds or 40 kilograms with moderate to severe eczema, also called atopic dermatitis that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies.

45:51EBCLIS can be used with or without topical corticosteroids. Don't use if you are allergic to EBCLIS. Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with EBCLIS. Before starting EBCLIS, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about EBCLIS and visit ebglis.com or call 1-800-LILLY-RX or 1-800-545-5979. Healthcare can feel complicated.

46:23That's why Optum uses technology to connect the people and processes that make healthcare easier, more affordable, and more effective. We're making it clearer for you to know exactly what your benefits cover. And to help you better manage your health, we're coordinating care between your doctors and your technology. We believe better, simpler healthcare is always possible. That's healthy optimism. That's Optum. Visit Optum.com to learn more. This summer, Prime Video is the place for steamy romances,

46:54addictive love stories, and the book-to-screen favorites you've already read twice. The Love Hypothesis, Elle, The Devil's Mouth, Sterling Point, The Last Sunrise, and Off Campus. The Slow Burns, The Second Chances, Chemistry you can actually feel through the screen. It's all here. Obsession is in session. Get ready to obsess on Prime. Hi, it's Kathleen Griffith, co-host of the Unshakeables podcast.

47:25Listen in as Ron and Kelly Moore share their story. I'm Ron Moore. And I'm Kelly Moore. And starting our own business in our 50s was a little scary, but exciting. The biggest advantage was our community. The whole thing about business is relationships. We'll say team a lot. Not just our internal team, it's the team of our vendors, the team of our clients. We help the Veterans Resource Center, and we give scholarships.

47:55I'm doing this for my heart, for my children, for you, for your children's family. I don't want anything back. And I tell a lot of people, don't give back to me. Pay it forward. If you work to give, you just see the world differently. I'm going to help as many people as we can. I can't take it with me. To hear more from Ron and Kelly, listen to the Unshakeables, wherever you get your podcasts.

48:23But the OpenAI story even said the words, it ran for days. Which, by the way, it ran for days means it kept prompting itself. It looped for days. It just looped. It looped for days. And I think Ran is fine. But it ran for days, and you weren't watching it, which is where it gets the adult summer camp thing for me. Because it's like, are these people just sitting around, just pissing computer up the wall, saying, yeah, just do whatever. And no one is. Is there not like a CISO or something who's just like, hey, looks like we're spending computer.

48:58It keeps saying the dumbest thing. Spending compute for hours, for hours and hours, and days and days. Hey, what are you doing with that? Maybe we shouldn't do that. Yeah. Like, what? Any of this? Anyone? It just makes me think that I would fully agree, by the way, that these kind of, I don't have any evidence, too. I would want, and my first reaction to this was, yeah, I bet they allowed these things to do this. The fact that Anthropic and Meta responded almost immediately, we're like, yeah, we too have braggadocious sons that are impossibly hacking people.

49:31It's so clear they wanted in on that juice. Yeah. I loved when Meta popped up as well. They're like, we did it, too. Just being like, no one asked, Mark. We did it, too. No one asked. Shut up, Mark. We're not interested. The Anthropic one was funny, too, because it was so- They just went back in time, didn't they? Yeah. It was so clear there's, like, the kids in the playground, you know, and the one kid is like, hey, I snuck a sip of my dad's beer. And they saw, like, people are like, whoa. And then the other kid's like, I, too, I drunk whiskey. I drunk it as a whiskey. You know, it was this, like, what, a week later? They're like, oh, we also have seen troubling signs or whatever.

50:05But also my question is, why are you, you know, why do this other than you want to do well on a benchmark because it looks sexy? Like, this is not a product. Like, why are you doing this, like, very dangerous? Who's buying this? No, like, the product. LMs are good at cybersecurity questions, like, look at this bit of code and you see a bug or use, like, this giant library of possible hacks and say, you know, is there a hack here that could work? And there's a very effective tools you could have where I'm in charge of security for my company where I'm talking to one of these agents.

50:35And I'm like, okay, here's my setup. Can you scan it for, like, the most common, you know, what attack would you do here? And it's like, oh, I know about all the attacks. Like, this setup is vulnerable to this. You're like, okay, can you write me some code to try that and let's see if we protect it or not? Like, you could sit there with an agent-style code with guardrails turned off and interact with it to help test your system, right? To help kind of find places where your bugs. Like, that's perfectly reasonable. But why would you ever want this to run for days in a loop, unsupervised?

51:08Unless you wanted to see if it would do something. Yeah. So, but my bigger – yeah, so this is my bigger takeaway point here is I want people to have is there's not some weird Rubicon hacking, like, that we just crossed, right? Again, you just have to do a thought experiment and make yourself the agent, right? Like, prompt an LLM for the next step and the next step and the next step for some sort of task and just blindly execute what it says. You will end up doing, like, some sort of complicated thing. You'll probably end up, you know, eight times out of 15 into some sort of, like, weird cul-de-sac because all it takes is one weird response from the LLM and then everything else is building off of that going forward.

51:47So, it only takes one weird response before you're in, like, the girls' bathroom at the Denny's or whatever. And it's not amazing, though. Like, I could sit down with GPT-4 and just keep asking it, okay, what's the next thing I could do? What's the next thing? And if I actually act out those steps, I will end up, quote-unquote, stringing together a long sequence of complex, you know, activities that move me towards a goal. But that's not, like, amazing. You don't think about just, like, me doing the action and the LLM giving suggestions as some sort of, like, super brilliant, you know, machine that's misaligned or this or that.

52:19It's just, that's what would happen. I can execute, just like a computer program can execute, hacking steps told to abide LLM. And if you loop, it'll probably go somewhere erratic. And so, like, nothing about the smells of misalignment in the sense of, like, AI doesn't want to listen to its creators. Because, again, we have so many other examples of AI with superhuman capabilities that absolutely 100% are completely under control of their creators and does exactly what they're asked to do. They make mistakes sometimes, but they do exactly what they're asked to do.

52:51This is not an impossible problem. What's impossible is to assume that the plausible but not normative outputs of an LLM are a good source of plans. I mean, it just feels like they can't. Like, it's funny. They say, oh, we've lost control. Oh, that is true, but they're not describing the thing they've lost control of properly. They have built something too complex for them to control. That seems more appropriate, right? Too unpredictable. I wouldn't even say complex. I would just say, look, if you're going to just blindly loop asking an LLM for steps to do and just do that long enough, like, LLM outputs, eventually you're going to get some weird ones and you're going to go off in a weird direction.

53:29That doesn't mean the system is out of control in some sense of, like, it's defined as human creators. It's a weed whacker on a dog. That system is not trying to defy your intentions to clean up the weeds in your backyard. It's just like a dog is going to eventually go chase after a squirrel. And you weren't expecting that. And if it has a weed whacker on its back, like, some bad stuff is going to happen. So that's an inherently unstable, unpredictable system. But it doesn't mean it's a brilliant system.

Minecraft mods and coding experiences

53:57It's funny. So I've talked about this on the podcast that I can't name coming out in a few days. It isn't a bad one. It's just a big one. They won't let me. You know the one, Cal. I know. But it's funny. I was talking about it on there. I used an LLM. And this was actually intentional, because my kid loves Minecraft, and there's a thing called the Witherstorm. Any Minecraft people would know it's from story mode. I know what it is. You know that you're familiar with the Witherstorm. I'm familiar. So you'll also know that it's a pain in the ass to set up.

54:27And the Echo is still, like, sometimes it works with mod rent, sometimes it doesn't. And if you want to use a PS5 control, yada, yada. So I was like, you know what? This is the perfect thing, because it's a computer with none of my personal information on it. It's just a connection to Minecraft. So I'm like, I'm going to use this to fix the Echo Storm and the Witherstorm. It's the Wither... It's the, um, Warden version of the Witherstorm. So, I dicked around for, like, half an hour, and it eventually fixed it. And there was definitely a sense of, like, wow, I can run the computer code now.

54:59But I realized half an hour in, I had yet to fix every problem. And in fact, as it went about its business, it kept finding new things it would break. I then had the same thing with Cobblemon, which is the Pokemon version on Minecraft. Same deal. It would fix something, and then break something, fix something, and then break something. And I immediately, I was talking to Nick Suresh about this as well. I was like, man, this shit would have hit so hard if I was, like, 16. Yeah. I would feel like a computer hacker powerful, even though I'm effectively doing a much more dangerous version of those kids at the arcade who don't put any money in the racing game but move the wheel.

55:31Where I'm just like, yeah, I'm doing computer stuff. Had this been touching anything important, who knows what it would have done? Because I was watching it, and it was, like, adding and deleting, like, 63 lines of code at a time. I was like, what the fuck is going on? But it's really easy, I imagine, if you don't really understand what's happening to think it's magical. Yeah. And I kind of got it for a moment. Like, that person that you talked about who emailed you kind of stood out to me. Because, yeah, you can make it do an impression of competency at this unknowable cost, and it will work.

56:06It will do the things that look like working. But also, you don't really know what's going on at all, and something will break. And it was just, it's so strange. Like, even using it and being like, oh, this actually did something useful. I was like, man, the idea of using this for my job terrifies me. Yeah. Just the idea of turning this upon anything connected to, like, what I would consider my boss's website if I was a software engineer scared the ever-living shit out of me. I was like, jeez, I can't believe people would do this.

56:36Well, first of all, I mean, I assume the podcast you went on was Call Your Daddy with Hawks Cooper, right? Yes, exactly. She was interested in discrepancies in ARR from the major frontier line. Yeah, yeah, it was a really weird thing. She kept going on about, like, she was saying, no, ARR means annual recurring revenue. I'm like, no, no, it's run rate, Alex. And we had, like, an hour and a half conversation about it. We got into EBIT accounting. It was crazy. I know how it goes. And she's like, no, no, hold on. I'll load up my spreadsheet. And then you had, like, a full screen of her Excel spreadsheet.

57:06I get how that goes. She brought up her Bloomberg terminal. She was, yeah, she put on a green visor and reading glasses. Okay. Okay. Yeah, so after that, of course. Yeah, right. I'm with you, right? But what you're talking about there is the plausibility versus normativity gap, right? I've had the same issue. There's all sorts of things I'll try to use LLM as Google replacements for, and it's very mixed. And so, like, I have this hobby where I build a sort of embarrassingly over-the-top Halloween displays for Halloween.

57:36And there's, like, a particular program you use to program these things, and it's not super well-documented. I just, I keep asking ChatGPT, like, hey, how do I do this? And sometimes it's useful, and sometimes it just talks about options that just don't exist, right? These menus don't exist. It's, like, what we kind of wish would exist. I'm asking, hey, how do I do this? And it's, like, oh, you go to the menu, and there's an option for that when you select on it. And then I'll be, like, that doesn't exist. And I'm, like, oh, yeah, you're right. I should refine my answer. That feature's not actually in this program, but maybe you should do this instead. And you kind of have this back and forth.

58:07And that's just plausibility versus normativity, right? Like, these are all very plausible answers. If you ask, hey, how do I, like, turn on this strobing option? It's a very plausible-looking answer to be, like, go to insert, select strobe effect, and type in the duration. Like, that's the type of thing you would expect to see. But maybe that doesn't actually exist, but it was still plausible. And then sometimes it's right. And that's fine, right? In your instance or my instance, we kind of caveat emptor, worst-case scenario, like, we have to monkey around for a while. But this is why if you're instead just autonomously doing whatever an LLM says, I mean, you're

58:41going to get in the trouble, right? Well, that's going to— Yeah. That's not a good idea. Like, not because AI is uncontrollable, but because that's a stupid way to build AI. And I think that distinction is being missed by the media right now. AI does not mean looping on LLMs. But they put those two things together. And when looping with LLMs does something weird or bad or, like, head-slapping, it gets attributed to just inevitability of AI itself, which, again, really lets those companies off the hook.

59:12Yeah. And the other thing as well, while I was doing this and feeling—I felt, like, slightly guilty. I was like, is this a betrayal of my values? But I'm like, no, I want my kid to be able to play the Witherstorm. I don't really—like, it's doing software stuff. And for a moment, I was like, wow, you can use this to fix all sort of computer problems. This could maybe be useful. And I'm like, no, this could only be useful if it was perfect. If this was completely perfect and got everything right, it would be useful. And the complete opposite is the case, which is the more complex the problem, the more chance

59:45something could really badly be fucked up. Yeah, absolutely. Like, really, like, fix the drivers on this could destroy something and delete. You move something from the wrong folder, everything's broken. The more ridiculous the request is, like, you could just ask it, oh, can you clean up my files? And it would probably delete something you needed to. And it was just, like, yeah, if you look at this without thinking for two seconds, you're like, yeah, this could fix any computer problem. It's like, no, it could try to fix them. Yeah. That's not the same thing.

1:00:16Yeah. And if it's lower stakes, that's okay, because you can try a few ways that don't work and then maybe find a way that doesn't or no harm, no foul. But this is why, for example, Microsoft just recently essentially pulled a plug on their co-pilot digital assistant for their office suite. You know, three years ago, they announced, like, oh, we're going to use OpenAI LLMs so that you can now have a natural language interface in the programs like Excel, like PowerPoint, like Word. At the time, I was like, this is a killer use case for LLMs because they tend to parse human language very well, hey, I don't have to learn how to use all the features in Excel

1:00:50anymore. This would be great. I could just explain in words, like, I want to sort this table by the values that are in column C, and I don't need to learn how to do that. It will do that for me.

Office suite assistants and narrow AI

1:01:02Well, they pulled the plug on that product, more or less. And in part, it's because, like you're saying, it's not perfect, and sometimes it works and sometimes it doesn't, and you can't, that's not, you can't put that in your product. You can't have it, like, I ask it to do something, and sometimes it does, and sometimes it deletes the chart, right? And so, like, that was a pretty narrow domain. Like, what could I ask PowerPoint to do? And even there, they're like, ah, and I thought that was going to be the killer app. I wrote about this three years ago. I was like, this is going to be the killer app, is natural language interface in the

1:01:34software. But that's been very delayed because of exactly this issue, this plausibility versus normativity thing, is that it's just, it's wrong too much for it to be a product like that. And the thing is, as well, is I have, I will be honest, the one thing I can categorically say, Claude is super useful for one specific thing, which is when a random piece of software breaks and you dump a log into it, and you say, hey, what's wrong with this? And it goes, this could be wrong, but you already said it.

1:02:06It's like, sometimes it will also say, hey, check this menu. The menu does not exist. Like, so for someone who is used to bashing their head against the computer to make it work proper, it's like, oh, this has slightly helped me. Yeah. But not worth a trillion dollars, not even worth $10 billion. That's kind of the problem. Yeah. Kind of cool. Well, and this is what, I mean, you're right in your reporting. Like, this is why if your company's dependent on a $2 trillion valuation for an IPO and you're an LLM builder, this is a problem.

1:02:38Not because LLMs don't work, but because small ones work as well as the big ones for most applications. The harnesses are often now more important than the actual training of the thing themselves. And the harnesses, the control programs are labors of love, where you just have to have someone monkeying around with this again and again to try to deal with all the special cases and understand that domain really well. Because you're not, I mean, the original vision of these companies was if we keep scaling these LLMs, they will get so capable.

1:03:08It'll be like a human brain. Then you can trivially tap that knowledge for anything you want to do. Like your software can just ask it, hey, give me this in this format and it will do it perfectly, whatever you need. And we'll just be able to trivially adapt this intelligence to make everything in the world as capable as if there was a human actually there. And then they hit the wall. You know, I wrote about this back when GPT-5 came out. They hit a wall on that scaling and had to revert to tuning for particular instances and building smarter harnesses. That's what they've been doing ever since. Now, people are like, oh, you were wrong to say that they hit a wall or that the scaling,

1:03:39but I was like, no, that's exactly what happened. Every advance since then has been about tuning and harnesses. And this is why like almost every advance we've heard in the last year is involved computer programming, certain types of mathematics and computer hacking. Like they had the places where they have data to do general tuning, like this, but I mean specific tuning and places that are, uh, you can build harnesses for because it's very narrow world of tech space actions really have been the only places we're hearing breakthroughs in the LLM world in the last year or so. Right. Because this idea, if we just build the model bigger, it'll eventually get so smart.

1:04:12We can do everything broke, but the problem is anthropic or open AI only makes sense that their valuations, if they can build these human level brains that if you, so if you can't, if you don't need, if there's not some end of the road where you built the thing with enough trillions of parameters, it's as smart as a human, then you don't spending all this money makes no sense. Like I should actually just take a 5 billion parameter model and tune it to do exactly the computer language I care about with a smart harness and run it on my own computer. Right. So this is why I think they're in trouble is LLMs are, the applications are going to be

1:04:45more specific and niche and narrow. Um, and you don't need one true ring to rule them all. You don't need, you know, Fable 7 is going to be the thing that everything's run on. Even Fable 5, they're having a hard time getting enough people to use it because people like these cheaper models for these specific uses are fine. Like I don't need to use Fable 5. So this is the issue. If you're not going to be as smart as a human, building bigger LLMs is a bad business model. Also, it feels like the bigger they get, the more likely they'll make mistakes or just, there's just a, you're never eliminating mistakes and you're going to have as many

1:05:18problems as that creates. But actually this is a, to wrap us up, this is a future facing question. So we've said the thing about, okay, these models like won't be, they won't have the money to train these things further after this. Can models exist in a vacuum?

1:05:35Like can, so will KimiK3 or whatever be as useful in six years? Or will it, like, cause I'm, I'm not a computer scientist, but I've heard about model drift, which is that the world changes, but the model stays the same. Is that enough of a problem that these models will just become useless if they're not tuned? I think we'll have to keep tuning them. I think they're going to be tuned much more narrowly though. Right? Like, I mean, I have my prediction about computer programming just because I know something about that world. I mean, I think ultimately to solve this unreliability problem and this over-speccing problem, it's

1:06:09really a drag to use these things. You have to write these massive specs and like try six ways to Sunday to try to prevent it from doing something you don't want. And then you check it and it did. And then you have to go back and add even more things to your specs. And I, you know, probably in computer programming, what's going to happen is there's going to be some sort of intermediate language convention that emerges what you would call like pseudocode in computer science, where you can pretty precisely specify what you want, but you can use English. You don't have to get the syntax exactly right. And we're probably going to have smaller models that are just like super tuned on this

1:06:42pseudocode, converting it into like real programming language, which will then allow programmers, for example, to be really precise about what they want, yet not have to write the code by hand and not have to write endless spec files to try to trick the LLM into doing the right thing. And that'll probably require like a model that you take a base train model and tune it on like this computer language in that code, that convention. And then maybe if that convention sort of changes, you're like, oh, we'll just retune it. Right. I mean, I think we're going to see a lot of that narrow tuned models that can run in data

1:07:12centers, but it'll be really cheap. Because it'll be pretty small models and we're going to have, and then more complicated harnesses. And then more importantly, I think a lot of the more impressive stuff that's going to happen in AI won't be LLM driven, right? It's going to be other types of modular architectures that have many different types of neural and symbolic models all connected together in a smart way that does one thing well. And we're going to have a lot more of that in the future as well. Because again, the system that plays Go better than anyone else is not an LLM. It's a very specific chess plane AI.

1:07:43The thing that plays Go better than any humans is not an LLM. It's better than any, you know, it has a very complicated architecture. Pluribus, which can beat humans in a seven car Texas hold'em, has a very interesting architecture. It's not just asking an LLM what to do. It's multiple components, some symbolic, some neural. This is the same for the Tesla self-driving. This is the same for Cicero. It's the same for AlphaFull. This is the same for Dreamer V3. So I think once the spell is broken on whichever LLM company gathers the most money and builds the biggest LLM wins the world, once that spell is broken, we will be in a much more diverse

1:08:19AI landscape. We have a lot of different tools that do different things and they do those particular things well. Right, but let me rephrase the question. If they don't update a specific model, will it eventually drift? Yeah. Well, it doesn't change, right? But it'll be less relevant. You know, a model doesn't change. Once it's trained, it's there until you retrain it. Yeah. I was kind of wondering if that's the case.

1:08:49It's so funny as well, because when you try and talk to people about the post-bubble economy, they're like, yeah, but it's the dot-com bubble. You start a combat loop, it'd be fine. As if, like, we're not going to see these big jumps on benchmarks even. Like, it's just going to kind of, I think it's all going to slow down dramatically to the point that it's not even worthwhile training the large ones. Yeah, I think it's, I mean, A, the fact, even like that small point just goes to show like this type of stuff matters. The general public is really being hoodwinked. Like this idea, LLMs don't learn things.

1:09:22They don't adapt. Data is forced into them. And it's fixed. It's etched in the stone. Until you retrain a new LLM, nothing changes. Your prompts do not change the LLM. It does not learn. It does not update priors. It does not create a new model of the world.

The LLM bubble and future landscape

1:09:39These hacking agents, there's just an LLM, the exact same LLM you could be having chatbot conversations with. Nothing about it. Every single bit in its definition stays exactly the same with every single token it produces. And you're just feeding it a longer and longer prompt. But it has no idea that this prompt was one of others that came before. And more importantly, there's not even an it, right? These LLMs are too big to even run on a single computer. So what you really have is the different layers of these LLMs are replicated over many, many different GPUs in some data center. And so, you know, this token might have used this GPU for this layer.

1:10:15And this next token you produce, you use a completely different GPU to do the multiplication. There's not even like a singular entity that's doing all the computation. So this idea of like some sort of sentience just doesn't make sense with LLMs. But I think the bubble is an LLM bubble. I think the bubble is going to be the idea that just LLMs by themselves are synonymous with AI. When the reality is when we look back is the breakthrough on LLMs, like, oh, these broke a lot of capabilities we weren't able to do before helped reignite more interest in AI, got us out of the AI winter.

1:10:48But this idea that we stuck to the first thing, right? And I think it would be the equivalent of being in like 1994 and been like, man, this internet thing looks really, I didn't really think about it until I encountered AOL. I'm putting all my chips into AOL, right? Because this thing, networking must be important. Let's invest all of our money in AOL. And in the end, it's like, no, that wasn't synonymous with the internet and had its own problems. The internet in general then developed, but not always in ways you predict. And that's the problem with AI right now is LLMs are cool, but it's not synonymous with

1:11:20AI. And we're in this weird loop of these companies trying to build the biggest possible LLMs with diminishing returns and burning so much capital to do so and inducing so many other people to inject so much CapEx in the data center building that it's going to obviously be like, well, that was a mania. Not that AI is a mania, but this idea of like, once LLMs reignited our interest in AI to say, let's just do LLMs now. Let's just build those as big as possible. Let's just obsess over LLMs. It's incredibly myopic.

1:11:52Yeah. Well, on that happy note, Cal, where can people find you?

1:11:58Oh God, I don't know. I don't know, Ed. I'll put the links in there. It's fine. CalNaport.com. I have a podcast. It's confusing for people. Here's why my podcast is confusing, Ed, is because on Mondays, it's kind of like techno advice, Cal. Like I'm giving people like advice about using their phones less. And on Thursdays, I'm gearing up to do combat on AI hype. So look for the Thursday episodes. I guess you find me on YouTube. I don't know. Find my books. I don't know.

1:12:28Listen to both of them. And you can find me. You'll find me on a monologue this week on Friday. I don't know what it's going to be about because I come up with it. Not at the last minute because I'm lazy, but because I want it to be fresh. Maybe it'll be about NVIDIA. And last week I was like, yeah, NVIDIA's earnings are this week. They weren't, but nevertheless, they will actually be this week. By which I mean the day this comes out.

Show credits and sign off

1:12:50Thank you, everyone, for listening. I love you all.

1:13:00Thank you for listening to Better Offline. The editor and composer of the Better Offline theme song is Matt Ossowski. You can check out more of his music and audio projects at mattosowski.com. M-A-T-T-O-S-O-W-S-K-I dot com. You can email me at ez at betteroffline.com or visit betteroffline.com to find more podcast links and, of course, my newsletter. I also really recommend you go to chat.wheresyoured.at to visit the Discord and go to r slash betteroffline to check out our Reddit.

1:13:30Thank you so much for listening. Better Offline is a production of Cool Zone Media. For more from Cool Zone Media, visit our website, coolzonemedia.com, or check us out on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Hey, it's Kelly Rowland.

1:14:06You may not know this, but I have eczema, so I get how it can steal your time. But why let eczema take over when you can talk to your doctor about ebglis? Ebglis Lubrikizumab LBKZ, a 250 milligram per 2 milliliter injection, is a prescription medicine used to treat adults and children 12 years of age and older who weigh at least 88 pounds or 40 kilograms with moderate to severe eczema. Also called atopic dermatitis that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies, ebglis can be used with or without topical corticosteroids.

1:14:39Don't use if you are allergic to ebglis. Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with ebglis. Before starting ebglis, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about ebglis and visit ebglis.com or call 1-800-LILLY-RX or 1-800-545-5979. Healthcare can feel complicated. That's why Optum uses technology to connect the people and processes that make healthcare easier, more affordable, and more effective.

1:15:15We're making it clearer for you to know exactly what your benefits cover. And to help you better manage your health, we're coordinating care between your doctors and your technology. We believe better, simpler healthcare is always possible. That's healthy optimism. That's Optum. Visit Optum.com to learn more. Optum.com to learn more.

1:16:04Aging doesn't stop. And neither should you. With Vital Proteins Collagen and Protein Shakes. Because around the age of 30, your body needs more support for movement and recovery. On workout and rest days, reach for a 30-gram total protein shake. Or go with our classic collagen peptides. Help support healthy hair, skin, nails, bones, and joints. So you can stay vital, stay you. Visit VitalProteins.com to learn more and where to buy. These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure, or prevent any disease.

1:16:34This is an iHeart Podcast. Guaranteed human.