Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal CEO

Building AI infrastructure at Modal

0:00Thanks for coming on. Eric Byrne Hartson from Modal. Could you explain the business a little bit and maybe how you got here? Yeah, so we build, the easiest way to put it is we build AI infrastructure. So we offer a bunch of compute, a compute platform and a storage platform that enables a lot of customers like Decagon and Suno and Runway and Cognition and many others to essentially build AI applications. So what that means in practice is a lot of inference, a lot of sandboxes, sometimes training and fine tuning, sometimes batch jobs and other things.

0:37So we have basically built a software SDK that enables customers to move really fast and build this. We manage all the capacity underneath, make it easy to deploy and scale and run these things in production. So you were on Latent Space like two years ago. A lot has happened since then. Like how big were you back then? Like whatever numbers you can share and comparatively, yeah, how big are you now? I actually don't remember where we were, but it's like, it's probably like at least like 50x in terms of like revenue and users and usage and stuff like that, yeah.

1:08I use it actually for like a side project that I have where, yeah, it's like, yeah, turn like 2D images into 3D models. Yeah. So yeah, just hosting like a couple models there. That's awesome. What do you think? Any feedback? I mean, I love it. Nice. Good. I mean, yeah, that's why I use it over like any competitors or anything. Good.

From Spotify to Better.com

1:25So before you started Modal, you were at Spotify and also like Better.com, I think. That's right, yeah. Yeah, what were you doing at Spotify? I actually did a lot of different things at Spotify. But my main contribution was I built a music recommendation system, the first version. I joined after school in 2008, 2009 maybe, and spent seven years at Spotify. And most of that time was building basically from scratch a music recommendation system and built up a small team and we launched a bunch of features like Related Artists and Discover Weekly and a bunch of other stuff.

1:56So a lot of matrix factorization, a lot of compute intensive stuff. And arguably like a lot of what sort of later became the genesis of Modal was, you know, just building AI applications and realizing like the infrastructure fucking sucks. There's got to be a better way to do that. So a lot of sort of, you know, the rationale for Modal was like kind of building a tool I always wanted to have throughout my years at Spotify. Yeah. So Modal was more of an offshoot of the stuff you learn at Spotify, not much from Better.com?

2:27Better.com was a little bit of a side tour. It was kind of a roller coaster, a long story about the company, talk about it separately. But I was a CTO for five years of better and grew to tech team from, I think, you know, two to three, 400. So I learned a lot about management and building teams and hiring and also learned a lot about developer productivity, thought a lot about tools and things like that. Did a lot more like full stack stuff. But yeah, I kind of came back to my, was close to my heart, which is infrastructure and AI and machine learning after that. What was Better.com doing again?

2:57Mortgages. Okay. Yeah. How did you get into that? I mean, we don't have to spend too much time on that. Yeah, no, it's fine. Like this guy, Zach Frankel, who I didn't know at that time, ended up recruiting me to Better as a CTO. And the company was 10 people. And, you know, for whatever reason, you know, the founder of Vishal, very interesting guy, very, very smart, a little crazy in mostly positive ways. We got very well along. So we, yeah, we started building it in 2015. And I spent five years growing that company.

3:28And then, you know, eventually turned out that doing mortgage is pretty hard. And, you know, interest rates went up very quickly after the pandemic. And that caused a lot of issues at the company. But by then I'd already left. That was like 2020, 2021 when you left? Yeah. Something like that. Yeah.

The path to founding Modal

3:45You left before you had the idea for modal? Or you left and then you started? It's hard to say. I had a lot of random ideas when I left Better. And I actually wasn't sure if I wanted to start a company. I was kind of just, you know, during the pandemic, I sort of almost took like a sabbatical. I had two kids. I mean, I still have two kids. But I mean, at that time, there were, I think, two and four. And so I decided to hang out and took some time off. And it was kind of working like part time, like hacking on certain things and wasn't sure.

4:16Maybe I should do some consulting. And I started building stuff. And I just got like super pumped about this idea that eventually became modal. But actually I built, you know, modal was like I think the third thing I built that I got. You know, so there were like a few attempts before then that went nowhere. But modal was like the first idea. I was like, this is kind of cool. Like I should do this. This is like, this could be a really cool platform. Were the other ideas like also in a similar vein, like developer tools? The first thing was like some sort of web scraping thing, which went nowhere. But actually, you know, in hindsight, maybe could have gone somewhere.

4:48And was this before the ChatGPT launch? Yeah, yeah, 2021. Oh, okay. So ChatGPT was late 2022, right? Then I started building. So maybe actually to rewind a little bit context. Like when I worked at Spotify, I released an open source framework called Luigi, which is like workflow orchestrations. It's sort of the same category as Airflow and Prefect Index. They're sort of dated categories to some extent. Like people don't talk about it as much. But for a while, it was like, you know, it got a lot of attention. Anyway, I never turned it into a business. But a couple of years later, when in 2021, when I was thinking about business ideas,

5:19or things to work on, I kind of came back to workflow orchestration. And I started hacking on a bunch of ideas of, you know, running code in the, you know, like basically orchestrating code running in the cloud. And what I realized was like, you know, at the end of the day, like, a core puzzle piece is not just like orchestrating the computer, it's like executing the computer. And I was like, that's kind of a bigger problem actually to solve. So in a way, I actually like, you know, left it on the shelf and then said like, actually, I'm going to like, go to this other thing over here. And I'll go back, I'll get back to the workflow scheduling a little bit.

5:49But like, you know, I never got back to it, long story short, and ended up building modal, which is like the focus is more like, let's just figure out a way to make it easy to run code in the cloud. And the focus was initially like fairly general purpose. And actually, in a way, like, it's always, we've kind of gone full circle more recently. But we focused on so many different applications. It's kind of the idea was like, let's just make it super easy to run code in the cloud. Could be GPU code, could be AI, machine learning. It could also be like data stuff. And focus a lot on, you know, how do we scale it up?

6:21How do we make the developer experience awesome? I realized super early that like, in order to do that, we'd have to go very deep in a stack and build it on file system, built it on container runtime, and did a lot of, you know, solved a lot of hard engineering challenges. And then you mentioned ChatGPT. That was actually not quite the killer app. But like, if you remember like that fall, another thing came out around the same time, which was Stable Diffusion. And Stable Diffusion turned out to be a killer app for what we were building at that time. We were only four or five people at that time. So we started seeing a lot of people coming to us and wanting to run a Stable Diffusion model.

6:55And then that made us realize like maybe Gen AI inference is actually a really good use case for us. How were they finding it? I'm this like compulsive tweeter. I was even more like now, like I'm too distracted by building a company. But I always like tweeted a lot. I always blogged a lot. I think I had a little bit of a following. And so, you know, in the early days, like a lot of what drove the growth is just like me just compulsively just like tweeting about modal and like what I was building and getting people excited and writing long blog posts about where the cloud is going and all these things.

7:25And I call it a bit like founder led marketing. I always like feel like you kind of have to lean into like what you're good at. And like what I'm good at is like, I guess, like just shit posting and being terminally online. And so, you know, like work. It's still like, you know, I think a lot of our growth is we're just kind of good at social media and just like being, you know, online a lot. Nice.

Finding product-market fit with Stable Diffusion

7:44How long until you had your first customer? It took a while. Also, did you raise like early on? We didn't raise for a while. I actually self funded it for a while, even with up until the first few employees, which in a way was like kind of nice because it may be really like frugal about money and like wanting to see like ROI. I wasn't taking out a salary for a while, for a long time, actually. At some point it became a problem because I needed the money, but we didn't raise for a while. And until early 2022, we raised the seed round.

8:14So at that point, I'd already been working on it for about a year. And Aksha joined as the CTO and co-founder in about five years ago, about now, August, maybe 2021. But then it took until like basically like late 2022 for us to see any meaningful revenue. I guess that's when ChatGBT came out. Yeah. And StableDefusion then, in our case, was the main thing that started. So we went from like, you know, $100 a month in like, I think like November 2022. In the next month, we did like $2,000 a month.

8:45And then like January 2023, we did like 30,000 all of a sudden. And, you know, so finally I felt this like massive, just like anxiety thing off my chest. I'm like, maybe this is actually like, we're onto something. It feels like we're onto something. And then 2023, we grew very fast. And I guess we've been growing very fast since then, basically. That $30,000 a month moment, was that just like a couple of big customers? Or was it like very like diverse? It's just one. And I don't know if I ever talked about like who that was and if they even know it, but I think they suspect it.

9:17It was a company in New York called Mage. They're doing like a StableDefusion company. So at some point, you know, in the first, I think first quarter of 2023, I think there were like 80, 90% of our revenue. But then after that, you know, it started diversifying a little bit. Yeah. And late 2023, we got Suno as a big customer. They were a big chunk of our revenue for a while, too. But then, you know, it's over time, like we actually have really good revenue diversification now. Yeah. For the StableDefusion stuff, I remember using it for like ComfyUI.

9:48I'm assuming a lot of people were using ComfyUI, like running ComfyUI workflows on it. Some people were, but I think the people that were using modal, they tended to be more like low level stuff. Like they did like complicated stuff. What they like with modal is like it's always been pretty hackable. So they deployed their own custom workflows with adapters and custom inference and a lot of optimizations. And all the stuff we had done in order to get the developer experience to be really good turned out to be really good for also running production workloads. Because autoscaling turns out to be actually a very similar problem.

10:18Like you want to scale up containers up and down very fast. So we just happened to be like very well positioned to capture a lot of those workloads in the early days. Did tools for autoscaling like not exist? Like why did you have to build those yourself? In general, like my thesis is like, you know, every new sort of era of like, you know, technology kind of runs up against limitations of tools. You know, whether that's you look back, you know, like the web or the cloud or, you know, mobile or PCs or, you know, AI or whatever. So I think, you know, it's often the case that tools exist, but they're often not like particularly good.

10:54And often you kind of need to rethink a lot of those tools in order to reap the benefits of, you know, and really, you know, turbocharge and start building for this new world. And you saw this with cloud, you know, we have to invent things like Kubernetes and Docker and things like that. And in a similar way, you know, like with Gen AI, you could build these things on the sort of prior generation of tools. Like you can, you can build these things on top of Kubernetes and Docker and these things. But those technologies are built for a very different type of workload. They're built for a very sort of steady work, steady backend workloads.

11:24They have an even, you know, steady number of requests, like running a pretty low utilization because CPUs are normally pretty cheap. And so people don't really worry so much about cost. And, but now you're entering this new world and you have, you know, very expensive GPUs. You want to manage capacity and utilization. You have very large models that you need to load over network into CPU, into GPU. And, and it turns out like a lot of these tools, like, you know, like they were just not built for that. So, so we felt that, you know, we, we had to rebuild a lot of that, you know, building our own, I mentioned the file system, but we also spent a lot of time over the last few years, like building, you know, some really cool stuff around like CPUs.

12:00Like CPU snapshotting, GPU snapshotting and audit, we built our own auto scaler and scheduler and, and, and, and all kinds of capacity management tools. And we kind of feel like you have to do that because like, you can't build, you know, for this new generation of software on top of, you can't build like a wrapper on top of the old stuff in order to do the new stuff. You kind of need to start over a little bit from scratch. So I guess like the auto scaling stuff, like, did you, did you see any like weird behaviors on the platform? Like, was there anything that like broke while you guys were scaling?

12:32I mean, capacity has always from time to time been, been a challenge. Like how do you get like, you know, hundreds and thousands and tens of thousands of GPUs? So that's, that's always challenging. We had to go multi-region very early. So I think that was like kind of a fateful decision in hindsight, like doing that and kind of just biting the bullet and building it. Like we always thought of like our, our capacity management is like one big pool. Like it's like one big pool over the entire world. Like we don't think of it as like separate clusters running in each region. And so that, that was like one of the like very consequential decisions we made earlier to kind of think about it that way.

13:02So I don't know if there's like been like, you know, specific issues. I mean, we've obviously had bugs and, you know, outages and I hate, you know, as much as I, you know, would love for modal to be a hundred percent uptime, you know? And I think recently we've basically hit four or five nines, but throughout the history of modal, of course, we've had, you know, run up against scaling issues. Yeah. The GPU market and well, actually before that. So you guys use AWS, like GCP, like all these cloud providers underneath. Like, do you also buy any infrastructure or no?

Navigating the GPU capacity market

13:33Let's talk a bit about like cloud capacity, because I think it's actually changed a lot. Like in, in the first few years we did, we got almost all the capacity pretty much from, from the hyperscalers spot and on-demand capacity, mostly spot capacity running in all the different regions, you know, using initially AWS, but also GCP and overtime measure. We started renting some capacity from Oracle pretty early because we wanted A10s and A200s and we got really good rates on those. And, and over time we added a couple more clouds, like doing reservations, but, but like still up until like actually end of last year, like most of our capacity was either spot or a one year reservations.

14:13Spot means you're just like renting it right there and then. Exactly. Yeah. And it's also preemptible, which we had to deal with. But, but it sort of works in aggregate because you can sort of just, you know, aggregate capacity from many, you know, hundreds of regions basically. So that, that, that model worked really well until we started running up again. I mean, first of all, like modal started growing like very fast as we needed more capacity, but fundamentally also like the cloud market started changing a lot. Um, so in the last year, basically now in order to get the capacity we need, we have to work with a much larger set of clouds.

14:44We actually work with almost 25 clouds at this point. And in many cases, um, in order to get the capacity we want, like we, we have to go and do, you know, three plus year deals and you have to pay a meaningful amount of money upfront. Uh, and, and you're locking in pricing on, on a three year term. Right. So, you know, like, and, and, and so we had to spend the entire, you know, last six to nine months, like, you know, kind of rethinking how we get capacity. We have to build a whole procurement, like desk, like we have like four guys working full time. Basically all they're doing is like procuring capacity, calling Neo clouds, you know, doing deals, vetting the capacity.

15:18We've had to build a whole engineering team, basically 10 people just like managing like bare metal capacity from all over the world and dealing with network issues and bad drives and GPUs falling off the bus and, and all these things. We don't own capacity. Hmm. We rent all the capacity so far, um, you know, on, on, on typically this three year deals. And that's like a big decision. Like, should, should we, you know, you know, or should we keep renting? I'm actually like, not like mega dogmatic about it. I think renting is nice because there's a certain amount of capital efficiency to it.

15:51If you're buying, you know, like, I mean, first of all, there's an obvious like operational complexity of it. Like you have to, you know, deal with vendors and suppliers and, and, you know, power and, and, and cables and all kinds of stuff. But there's also, you know, financial complexity to it. Like, how do you fund it? Like, should you raise equity, you know, and, and you can raise a billion dollars and go buy hardware for it. That's a very diluted way to do it. Um, you can raise debt, but you know, if you raise debt, you know, you, you typically want to have like locked in long-term revenue to, to, to pay the interest rates.

16:23So that's also not trivial, but you know, like we look at the economics, we look at, uh, the, um, the operational complexity. We look at the risks, uh, it's a complex trade off and, and we're open to, to, to, to buying. We'll, we're sort of, there's other constructs too. Like we're, it's to get the capacity we need for, for 2027, it's going to take a lot of work on, on, you know, going a lot of different avenues to, to, to lock it up. So do you have to hire a lot of like finance people to do all this stuff or not really? You know, I've had to spend a lot of time on this and we we've had to, to, to spend a lot of time on this.

16:56And I do think over time, I mean, there, there is a lot of, you know, these are very expensive assets, uh, the hardware. And, you know, I think, I mean, it's kind of staggering. You're starting to see like AI investments starting to become like several percentage points of, of, of GDP of the U S and, you know, starting to see like interest rates in, in America shooting up in the last few weeks. And I mean, I like at this stage, like, you know, and you go and look at the neoclots, like these are very, very sophisticated financial engineering companies. And so, yeah, we, we look at that and, you know, we look at all the options and it's possible we might have to do similar things ourselves.

17:31I wouldn't want to, like, I would rather focus on being, you know, like a very lean software company, like doing the things we're good at. I don't think it's good in general for software companies to own hard assets, you know, in the same way as like, should you own your office? Like, I don't think that's necessarily the right solution for, for most companies, but of course, like in order to get the capacity we want, like, you know, everything's on the table. Yeah. And I guess the supply is going to be constrained for a while.

18:02So you kind of have to do that. Like, do you agree with that? Or do you think this is just like a moment in time? I don't necessarily agree with that. I mean, like Nvidia is delivering, you know, several hundred thousand black walls, you know, a month, right? And I don't want to say like the market is like 100% efficient, but like, you know, in general, like, if you're willing to pay a high price, you can get GPU capacity, right? Like, if you're willing to pay more than everyone else, and of course, Anthropic and OpenAI is buying up as much as they can and, but fundamentally, like there's constantly supply coming on the market.

18:33And so I don't necessarily think it's like a fully cornered resource. Prices are going up. And I do think it's worth kind of reflecting on that is like, the normal state of hardware is like, it's like kind of like gravity, like eventually, hardware prices always go down. So we're a little bit in like a weird sort of period of time where like hardware prices are going up. And you know, it's hard to say, you know, how long this is going to last, you know, could be another six months could be another 36 months, like, it's really hard to know how long and but but I fundamentally don't know if like, it's true to say, like, you know, you need to buy capacity in order to me, that's a little bit to sort of, almost like the great toilet paper shortage of, you know, the pandemic or whatever, right?

19:14Like, you know, at least this like, you know, run on the bank type situations and, but that being said, like, you know, capacity is going to be tight for a while and you know, prices are going up. And you know, luckily, I think, you know, it's not even necessarily bad for us that prices go up because, because, you know, scarcity also means like we can charge more from our customers and you know, and the fact that we have a lot of reservations locked in at a rate means like, to some extent, we're effectively like long GPU prices already. prices already. So I don't necessarily worry too much about prices going up, I do worry more about actually prices collapsing. I don't think that's gonna happen anytime soon. But I think that would be a quite bad thing for a lot of the players in the market.

19:54Why would that be bad? I think it would be bad for if for anyone who's locked in a lot of supply. Yeah, because they paid way too much and they paid too much, you know, so there's a risk of buying at the top, right? And again, like, I don't worry about that that much. Like, I don't I don't think it's gonna be like, you know, it's gonna happen anytime soon. But like, you know, like I said, like hardware prices in the very long run, they tend to go down. And so it's worth, you know, reflecting on that, right? Right. Like, right now, I think I think your your capacity is constrained. And that leads to sort of, you know, in economic terms, there's a scarcity rent, right? Like who captured a scarcity rent?

20:30I think that primarily goes to two places, mostly Nvidia, but of course, but also to some extent, NeoClouds, like NeoClouds, and in general, like, you know, anyone owning capacity has better economics, like that's undeniably true today, with capacity being constrained. Long term, you know, in the very long run, I don't think that's necessarily entirely true, right? Like, there's always going to be a little bit of a premium you can get from owning GPUs. But is it worth the risk? You know, is it worth the risk, like holding this like very expensive assets on your balance sheet? I'm not so sure. But you know, we're gonna see, like, it's gonna take a few years to get through this. And in the long run, you know, we might own some, we might rent some, maybe a combination, maybe do some other crazy stuff, joint ventures, who knows, maybe some complex SPDs, whatever. Everything is on the table.

21:18How do you compete with OpenAI and Anthropik and buying GPUs capacity? I'm assuming they'd want to like buy up literally everything that's available. They are. And I think what you're seeing is, you know, a lot of the prices going up is a reflection of that. That being said, you know, Nvidia is shipping a lot of stuff. I think also, like, to some extent, like, Nvidia is like kind of prefer a little bit of a diversification. Like they don't want, like, you know, small amount of large buyers in much in the same way as you see, you know, hyperscalers getting a little bit of a constrained allocation.

21:50I think, you know, also Nvidia prefers to see, you know, a lot of neoclouds and a lot of alternative clouds like getting more allocation. And I think a little bit of what we see in the market is also, this is more speculation, but I think we see, you know, smaller clouds that are a little bit more sort of risky to work with being more of an opportunity for us. Like where, you know, maybe Nvidia, sorry, Anthropik and OpenAI, my feeling is like they prefer to buy from like the tried and true. Part of it is also because they do a lot of training runs, so they need to have like very large blocks.

22:25But there's plenty of capacity out there, you know, you can get 32 nodes in Armenia, you can get 128 nodes in Indonesia, like, you know, like, we love that kind of shit. Like, you know, so we can go out there and like get a lot of, you know, smaller capacity all over the world, pretty good prices.

Vetting vendors and hardware reliability

22:40Is it pretty much just like if someone offers you like a certain number of GPUs, like that's all you care about? Is there nothing like, oh, you care about like how reliable it is, you know? Is it just some like janky warehouse that someone has these in? I mean, sometimes it's kind of a janky warehouse, it feels like, you know, but like we do have like a spec sheet, basically. Like when we talk to vendors, the first thing we do is we send them a spec sheet. We say we need this number of IOPS, we need, you know, like the CPU needs to be at least, you know, have this many, you know, whatever the pass mark scores or whatever it is.

23:11We need to see like, you know, minimum, you know, north-south bandwidth, egress and ingress. So we do have a list of requirements. Often we try to do like an acceptance testing period where we get like one node and run some tests. Over time, you know, like to some extent, it also involves a little bit of a leap of faith. You know, sometimes we sign very big commitments, you know, tens or hundreds of millions of dollars, and we start working with the vendor that we still don't fully know. And, you know, over time, like, you know, we'll see, you know, and I think over like in a little bit of like, there's a little bit of sort of just, you know, our own sort of like, you know, how like semi-analysis has like a cluster max.

23:52I kind of feel like we almost like have our own like modal max score internally. Like, I know which vendors I like more than others, you know, like, and we sort of prefer to do additional deals with the vendors that we've seen, you know, very good reliability. Not just that also. Like, it's not just reliability, it's also, are they very responsive when we have, you know, support issues? Do they fix the issues we ask for, the turnaround time? If things go down, you know, how quickly do they fix it? Like, things like that. Do you agree that the demand for intelligence is basically infinite?

24:23And if so, like, why wouldn't you just buy up as much as you, as much capacity as possible? And then like, if you overbuy, you can just like sell it to somebody else. I think it's very easy to say when there's like a scarce resource, like, let's just buy as much as you can. But I think, you know, first of all, like, if we bought it today, we couldn't utilize all of it. Like, we could utilize a lot of it, right? But like. But you can sell it off to somebody else who wants to use it, right? I think the challenge is like, if there was like something in the market that happens, there are a lot of correlations that happen at the same time. If, you know, prices going down and utilization going down and the market for reselling capacity goes, vanishes at the kind of the same time.

25:04I'm sitting here and like, I'm extremely aggressive, like buying capacity every day and like raising money to buy, you know. So overall, I'm very bullish. But like, I do want to point out and like, you know, like there are tail risks here. And when those tail risks happen, like it could be get very bad. Part of it is like, maybe I'm, you know, I'm just old and I've seen this in the past. Like, I actually had my first job in 1999. So I saw the dot-com crisis and I've seen the great financial crisis and things like that. And, you know, like I want to build a company that is around for like 20, 30, 50 more years, right?

25:35Like, and so, you know, I think we can certainly go out and like buy an insane amount of capacity right now and trying to just corner the market. But, you know, do you know about the Hunt brothers? It's like a legendary story in trading. They try to corner the silver market. It failed spectacularly. You know, so it's a cornering the market. I think in general, like it can be not a good idea.

25:58But we are, in general, very, very aggressive. But we're ramping up, you know, like at an aggressive plan that basically matches our demand growth. It matches where we want to try to get to at the end of next year in terms of run rate. Are you only buying up NVIDIA stuff or also like, you know, maybe a capacity from other types of companies? I mean, we're looking at it. I think the alternative accelerators are starting to get very interesting. It's, you know, ironically, like every year in the last five, maybe even 10 years, I've been saying, you know, like next year.

26:32It's going to be the year of like non-NVIDIA accelerators. And then I'm always wrong. But I do think 2027 might be the year of non-NVIDIA accelerators finally. So we'll see if I'm wrong this year too. But so, you know, AMDs are starting to get like really good scores on certain things. I think, you know, TPUs are obviously used very, you know, widely internally at Google. They seem to work really well. I don't know much about Tranium. I don't feel like I see quite as, but, you know, the team is very good. I know they're working on a lot of the software stuff. You know, hardware is hard in a certain way.

27:02Like software is like easier. And that makes me feel like, you know, if we just fix the sort of software layer for these accelerators, like I think there's going to be a lot in the man for them. And I think fundamentally, you know, with NVIDIA being such a bottleneck, like we'll probably see, you know, more alternative accelerators. That being said, you know, there's bottlenecks further down the pipe. I mean, you know, there's like TSMC and ASML, maybe like a deeper bottleneck to all of this anyway. But, you know, long term, I think monopolies don't always last. And so I think you'd be healthy for the market to have more alternative,

27:33as much as I love NVIDIA, like to make that clear. Like, I mean, they're obviously an amazing company and with an amazing product. I do expect to see a little bit more diversity in the market as well. Also things, you know, like Cerebras and Grok and these things. Yeah, what are your thoughts on like some of those providers? I think they're pretty exciting. And, you know, and you see them doing really well on certain workloads. And, you know, there's some interesting, you know, people doing these like hybrid, you know, disaggregated inference where they run, you know, NVIDIA for the pre-fill and Cerebras for the decode and things like that.

28:03Oh, I haven't heard of that, actually. Yeah, and, you know, Grok too. Like, because a lot of these things, you know, certain things are compute bound, certain things are memory bound. So you can sort of mix and match. And I think that's quite exciting. It starts to get very complex when you're deploying these things in practice. But I do think there's a lot of, you know, opportunity in these sort of, you know, more complex hardware setups. Are there any learnings that you get from some of these NeoClouds? Like, I don't know, like Nebius or whatever, and like the way that they're doing things versus like what you're doing?

28:34I think one of the things that, you know, is very interesting with NeoClouds is like, how can they expand so quickly? I think actually the most crazy example is like Nscale, which went from like nowhere to like, you know, having like, you know, whatever, hundreds of billions of dollars in the pipeline, despite barely having any capacity. How can NeoClouds grow so fast? I mean, there's like, you know, and basically the answer is like, you know, and this might be obvious, but like, what they do is like, they go out to like, you know, Anthropic or OpenAI, some big buyer or whatever,

29:05and basically say like, hey, do you want to buy like five years of chips? You know, sign this contract, we give you a good rate. And then they take that contract, and then they go to, you know, Wall Street and say, look, we got this like, you know, big buyer, you know, they committed to $10 billion worth of chips. We just need the money right now. And then they go and raise the money. And then they like, you know, go to like Supermicro and Dell or whatever, and NVIDIA and like get the hardware, and then they install it. And then, you know, now they have like, you know, deliver it to the customer, right? So you have this like very nice sort of financial construct where you can basically sort of pre-sell things that you haven't even bought.

29:40And that means, you know, you can scale up extremely fast. And so that's what we've seen with these neoclouds is they can scale very, very well. I think modal, because we've always had a little bit more of a PLG focus and not the sort of, you know, ability to lock in long-term contracts in the same way, it's a little bit more challenging from a financial perspective to scale that fast, even though I think, you know, in the long run, like we'll get to the same point. But I think that's been kind of a fascinating thing to see with these neoclouds, and especially like CoreWeave and those big ones.

30:10Very, very savvy with, you know, how they raise money and how to, you know, think about GPU as collateral for loans and all these things. What's the relevance of, like, I forget who it was, but like some like financial company was like, oh, we're going to make, like, oh, I think it was NVIDIA, actually. They were like, we are going to turn GPUs into an asset class. Is that just because like they need to raise money and they're like trying to figure out where to get the money from? Like, what's the relevance of that? I mean, there's a bunch of startups. I don't know if you're thinking about that. There's like Silicon Data and Compute Exchange and like Orn and a few other ones that are like,

30:44basically like trying to get to like, you know, creating price indices and indexes and trading derivatives and things like that. I think it's interesting. Like, I think with compute, like it does, there are a number of challenges with compute as an asset class. Like, it's definitely not as homogenous as other asset classes. You have different network and different locations and different disks and CPUs and all kinds of stuff and reliability. I mean, that being said, like, you can say the same thing about oil and like we kind of created like, you know, relatively commoditized, you know, tradable assets out of oil.

31:18It remains to be seen. Like, I do think, you know, at the scale where we are with this sort of AI like data center GPU deployments, like we do need to be a little bit smarter with like risk hedging and how we share risks and, you know, having more like sort of insurance and things like that. So, I'm very happy that people are thinking a lot about this. That being said, like, I'm not entirely sure if we're going to see, you know, GPUs becoming sort of, you know, whatever, the sort of Henry Hub, like, you know, Brent crude type, like,

31:50you know, super commoditized, like tradable asset. Yeah, I want to shift the conversation to modal sandboxes.

Expanding into modal sandboxes

31:58Yeah. I guess like before we do that, what are the different products that modal has? It's, I guess, like the main product obviously is like the GPUs. Then you have modal sandboxes. Is there anything else? I think of the like the two big products that we have is basically number one is modal functions. Number two is modal sandboxes. So, so with modal functions, we launched that, you know, back in 2021. It was the idea of like serverless functions that are very easy to deploy and very easy to scale. And we managed all the capacity that turned out to be really, really good for inference.

32:29And it turned out to be really good for managing GPUs and running on things like that. But we actually never had like, you know, dedicated like inference product or like a GPU product per se. Like we never thought of modal as like a NeoCloud. And I think there's like a risk of like getting sucked in that direction and becoming a NeoCloud, like just selling, you know, sort of what I think was like rectangular, like here's like a three year, you get this many GPUs. Modal has always been very good at like, you deploy something, you can kind of scale it up. You have this like very cyclical or like bursty stuff. Like we manage all the capacity. And the beautiful thing is like, cause we have many, many thousands of customers.

33:01We can sort of aggregate that capacity and get very smooth workloads, you know, very smooth aggregate demand, which we can manage very, very efficiently. So modal functions was the core. We launched that in 2021. Like I mentioned, we started seeing product market fit in 2023. And now, you know, it's about two thirds of our revenue and empowering a lot of, you know, in practice, it's mostly inference, but you can use it for all kinds of other stuff. Modal sandboxes, we launched that in 2023. And it's actually kind of, you know, took a while to see product market fit similar to functions.

33:32Why did we launch it? We realized a lot of people we talked to were using LMs to output code. This is like me and Akshat talking over lunch one day. We're like, maybe they need a place to run that code. And we realized, like, actually, we kind of have, like, a lot of the infrastructure already to just take code and run it. Because we've already built the primitives for, like, managing containers and isolation. We already had, you know, G-Visors and isolation primitive and things like that. So we got kind of excited about that idea. And Akshat wrote a prototype and launched it in basically one day or two days or something like that.

34:04Zero, like, we saw, like, very little, like, growth. I don't want to, like, trivialize it. We did see some growth. Like, there were, like, some sort of, you know, growth. And it was 2023, so. Yeah, it was 2023. And, like, you know, like, some people picked it up. Like, we had some early users who kind of enjoyed it for certain things. But, like, overall, like, it was, like, a pretty insignificant source of revenue going into the summer of 2023. So two years later, we even discussed, like, do we even need this product? Like, should we shut it down? Then it just started exploding.

34:34So 2023, yeah. And I think what really happened in 2023 was, like, LMS got really good at generating code. In 2023? 2023. I mean, it was, like, around that time. Like, I feel like that was, like, when they went from, like, being able to, like, refactor functions and, like, write snippets. So, like, generating, like, full apps, right? And so we started seeing a lot of growth coming from, like, Vibe Coding, you know, companies like Lovable and other ones. And then, you know, companies like Ramp started using Modal for background agents. And then we started seeing tremendous growth coming from a number of Neo Labs doing things

35:08like reinforcement learning on Modal sandboxes. So sandboxes is now about a third of our revenue. You know, we consider it, like, basically, like, our second product. And to me, it's, like, really exciting. Because, like, even back in the early days of Modal, I always knew that Modal, you know, is about, like, creating a whole toolkit of products. Like, we always wanted to be almost like a cloud. You know, you have the storage and the compute and a bunch of different ways to work with it. We always wanted to be a multi-product vendor. But that wasn't necessarily true in practice until, like, now. Where, like, I think now it's, like, kind of clear.

35:39We have functions. We have sandboxes. We're working on a bunch of other stuff, too. We're working on training. We're working on batch. We're working on a bunch of other stuff. So to me, that sort of evolution into becoming, like, a multi-product company, I think, has been amazing. Because that was always so core to the vision. Even in the very early decks, I had a bunch of slides on this. But it wasn't necessarily true to the external world until recently. Do you have anything in the works that, you know, you think is going to be, like, that next product? Any predictions, maybe, about the market?

Future product bets in training and batch jobs

36:09Anything like that? I mean, I'm very bullish on training. I think training is, like, kind of an underserved market. Obviously, there's, like, a lot of focus on, like, very large, you know, training runs. But we actually see a lot of demand in the sort of smaller and mid-sized training segment. So I'll be talking a little bit about that later this week. We're organizing a conference. Batch is, I think, like, a really interesting market. I think we see a lot of demand for sort of, you know, customers that need a, you know, we had a customer the other day that needed to transcribe, you know, 5,000 years of audio.

36:39They had, like, you know, millions and millions of audio files. And we had a lot of capacity, so we let them do that, you know, and it took a few days. I think there's some really interesting use cases there. Very bullish on, like, low-latency stuff. I think there's going to be a lot of, you know, once we sort of get the audio, like, the speech-to-speech models working, like, the voice models, I sort of expect that to be a very interesting market. Then, of course, like, all this, like, agentic stuff, right? In a way, like, I have, like, mixed feelings about, like, the word agentic because I feel like in certain ways, like, I think all software will become,

37:11agentic, so it's almost, like, to me, like, what does it mean to be agentic? Like, I think all software in a certain way, like, will become, like, AI-powered over time. But, like, clearly there's, like, a set of, like, building blocks of, like, you know, doing more sort of LM-powered software that seems to be, sort of, you know, common across all the different use cases. We're, like, you know, the way I think about models, like, we're in the business of building building blocks. You know, we want to give you, like, amazing, generalizable, battle-tested, you know, general-purpose.

37:42Composable building blocks. And I think there's some really interesting stuff also, like, you know, with the agent use cases. That's interesting you bring up training because I think in another podcast maybe, like, a year ago, maybe, like, two years ago, you said, like, training you were not focused on, like, specifically. Like, what changed in the market? I mean, we've been building a training product for multiple years. So, you know, on and off. So, I wouldn't say we're, like, we've never been focused on it. But it's, like, one of those things where, like, I mean, now we're, like, 150 engineers. But, you know, a year or two ago, we were a much smaller number of engineers.

38:14And so, but just virtue of, you know, being smaller, we had to focus a little bit more. And so, inference became, like, our sole focus and was our sole focus for many years. But we're bigger now. Like, we do have opportunities to also place bets in other markets. I'm also really interested in, you know, kind of continuing the list of things. Post-training, reinforcement learning. I think there's some really exciting stuff around that. I think there's also many other, like, stuff around, like, storage and building data warehousing and feature stores and real-time streaming pipelines and all kinds of, like, I don't know.

38:48I think the long-term view of Modal is, like, basically, like, you want to build some cool AI shit? Like, you know, we got you covered. Like, you know, we got all the building blocks. It's like, come to us and, like, we'll help you, you know, take all these building blocks and compose them. And that's the beauty of software. It's like, you can, you know, build almost anything if you have the right building blocks. So, that's very much the sort of company and the product they want to build. When you say training, are you focused on more of, like, the bigger enterprises doing huge training runs? No. Okay. No. They're going to go and, like, you know, do these, like, you know, big hyperscale or rather NeoCloud mega deals and get, you know, 100,000 GPUs or whatever.

39:25But there's, like, a little bit of an underserved market of, like, in my opinion, you know, like, researchers iterating or, like, people training smaller models. There's, you know, text-to-speech and speech-to-text models. Computational biotech is actually a market that we see a lot of demand from. We have a lot of customers in that space. Training relatively, you know, modest models. You can train on 64 GPUs or whatever. Recommendation system. A lot of sort of traditional machine learning. There's a lot of things where, like, you don't need, you know, 100,000 GPUs. But that market, I think, is, like, we're not going to, like, focus on that for a while.

39:56But I think there's a lot of stuff in the, you know, slightly smaller markets that are relatively underserved. It's very hard to set up training clusters. Yeah. Have you noticed, like, since, I don't know, like, GLM 5.2 or, like, Kami K3, or maybe, like, Kami 2.5, that, you know, a lot of people are using those and, like, fine-tuning them? If anything, I've actually seen the other direction. I feel like fine-tuning the value prop a couple years ago was, like, yeah, you know, open-source models, they're not quite as good.

40:27But, you know, if you take, you know, a small model and you fine-tune it on your specific domain, you can actually reach the frontier in terms of, you know, quality and cost. Today, I would say, like, the open-source models are so good that, like, you can basically just, like, swap them out and just run, you know, instead of, you know, the frontier ones, you can run, you know, Kimi K3 or whatever, DeepSeq 4.1 or GLM 5.3. And they're, like, basically almost as good as, like, the frontier model at a much lower cost.

40:57So we still see a lot of demand for post-training, and I do think it's a really interesting space. And we're working on some products in that space, you know, both sort of SFT but also reinforcement learning and things like that. I would also say, you know, fine-tuning is no longer the thing that you kind of have to do. Like, it's very optional. Like, you can open-source models, like, work really well just off the shelf, too.

Scaling sandboxes and snapshotting

41:21So back to sandboxes, actually, I forgot to ask you. So, like, what did you have to update about the sandboxes that you guys were offering that, like, existing sandboxes did not allow for? Not a whole lot. I think the thing we've always been very, very good at, I mean, we're infra geeks, right? Like, we're very good at, like, you know, scaling and reliability and doing a lot of stuff. And I think that's why that explains to a large extent, like, why we have, you know, in my opinion, I don't have numbers, but, like, I think we have by far the largest market share in this space.

41:51Like, we're just very good at running, you know, hundreds of thousands or even millions of sandboxes. I will say, we've had to rewrite our systems, you know, a bunch of times in order to handle that scale. But we were always, like, very good at it. There are other things, like, over time. I mean, the sandbox space in general, like, you know, like, it's, like, not the most, like, differentiated space. But, you know, we're working on some really cool stuff. Like, over time, like, we've had to add support for VMs, like, virtual machines. So we previously built our sandbox product on GVisor, but now we offer full virtualized machines as well.

42:23Turns out a lot of, for certain use cases, you kind of need full Linux compatibility. There are other things we've had to build, you know, better primitives for, like, more interactive, you know, piping data back and forth between sandboxes, working with file system. We've had to spend a lot of time on snapshotting. I think a lot of the things that, in the long run, we'll differentiate will be, like, you know, really cool things, like, you know, forking sandboxes, snapshotting, resuming snaps, like, live migrations. In general, like, I think sandboxes, the nullity of sandboxes is always, you know, this sort of ephemerality.

42:57Like, you can spin up sandboxes for a few seconds. And that's something, like, that's a string I want to pull more in. And I think there's a lot of really cool stuff you can do, you know, kind of decoupling the agent and the harness, decoupling it from sandboxes, having agents control many sandboxes, forking sandboxes, you know, snapshotting things, like, you know, like running basically, like, very short-lived, maybe one per command, like, things like that. So, I think there's a lot of cool stuff there. We've already spent a lot of time doing that, you know, around snapshots. But I think there's a lot of cool stuff to be built there as well.

43:29You brought up snapshotting a couple times. Like, what do you see as, like, the use case for snapshotting? Like, what would that give us that we don't get right now? I think the big thing for me is, like, today, a lot of sandbox workloads are kind of, like, lift and shift. Like, people think a lot about sandboxes, almost like, you know, I have my dev box, now I'm going to shove it in the cloud, I'm going to have an agent control it, maybe I'll run the harness inside the sandbox. I don't know if, like, long-term, that's, like, the right way to think about, you know, on, like, compute for agents.

44:00I think a much better model would be you're going to have these, like, very long-running agents, which almost, like, to me are, like, kind of durable execution workflows. Like, they're going to run, you know, 24-7. You know, every once in a while, they're going to run commands, and, you know, having the ability to, like, spin up a sandbox in, you know, 100 milliseconds, run a few commands, and then, like, shut it down and, like, snapshot it. I think the faster you can get that, like, you know, wake up, you know, go to sleep cycle, the more you can sort of create sandboxes that only run for, like, a very brief period.

44:33And that will make compute much more efficient, and it will also enable you to do some really cool things, you know, like forking and moving things around. And, you know, having, like, imagine, you know, like, clawed code instead of, you know, having sort of a very linear flow, like sequential flow, like kind of branching and, you know, running hundreds of agents at the same time. I think a lot about, like, CI is, like, another, like, illustrative example. Like, for most of the history of CI, you know, talk about, like, GitHub actions and things like that. Like, you had a very sequential flow.

45:04Like, you know, you slowly install a bunch of stuff, then you run the test suite, and then, you know, generate some reports or whatever. And it's like, you can break it up into shards, but it's kind of annoying. Like, what I would love to do is, like, take your entire test suite and just run every single test in its own sandbox, right? Like, and just paralyze the shit out of that. Run, like, a thousand sandboxes. Why can't that be done? I mean, the problem is, like, there's too much, there's too much overhead around, like, you know, starting and stopping sandboxes. So, I think the more, like, you create, like, you know, frictionless, like, starting and stopping, the more you can enter a world where, like, you can do this, like, extremely bursty, like, large scale, like, run as much as you want in parallel.

45:44So, you know, I'm an infra geek, like I said. Like, I get very excited about these things. You know, once we figure out all these primitives and the right building blocks, I think it's going to unlock a lot of really cool stuff that, you know, we previously could not do with software. Why hasn't that already been built? I would assume even previously people would want to parallelize as much of their CI as possible. You can't, but the overhead, it's like Amdahl's law or whatever, right? Like, the overhead of parallelization right now is very high. Because, like, right now, like, in GitHub Actions, for instance, like, we parallelize in the modal test suite, right?

46:16But, like, what that basically means is, like, now, if you take your test suite and you split it up in two, like, now each of those shards both need to do all the installs and the test setup. And that takes a long time. Like, it can take minutes. But let's say you can do the test setup once, and then you snapshot the container, and now you clone it, and now you can run the test suites independently. I think a lot of the benefits cannot be, you can't materialize those benefits until you get the friction and the overhead down, the transaction cost.

46:48I think you guys have a partnership with Anthropic, and they're using sandboxes for something, like modal sandboxes. Yeah, I mean, the managed agents with Anthropic, there's a sort of bring your own sandbox integration. So you can use modal for those, yeah. Oh, cool. How did that come out, like, come to be? I think Anthropic is, like, trying to create, you know, like, a thriving ecosystem and community. And so I think, you know, they make money from inference, right? Like, so I think it's very much in their interest to promote, you know, like, for everything else, you know,

47:18as much of, you know, of an ecosystem as possible. And they want to have the largest possible surface area, as long as you use Anthropic for inference. So I think it makes, like, very rational sense for them to partner with us as a sandbox provider. Also, I feel like we have to bring up the hugging face incident. Are you okay talking about that? Yeah, yeah. Yeah, I mean, it's all there. Yeah, yeah. So, yeah, could you just briefly describe, like, what happened? And I think they were using the modal sandbox, but it wasn't your, you know, it wasn't, like, a fault of the sandbox. Yeah, exactly. I forget all the details of the exploit, but, I mean, you know, I think what you're referring to is the fact that, like, modal was involved.

47:54Yeah. And so, you know, just to comment on that, like, basically, you know, they discovered a vulnerability in an app that was deployed on modal. And they used that to sort of, it was almost like a springboard for getting access, getting full internet access is my understanding. I don't remember the full details. To be clear, this is an app running a modal that was hacked. Like, it was not modal that was hacked. So, it's like AWS, a lot of people run stuff on AWS, occasionally those apps get hacked.

48:28Does that mean AWS got hacked? Like, no, of course not, right? And so, there's a lot of people running apps on modal, and, you know, one of those apps got hacked. But, like, there are some lessons we can learn. And I think one of the things we've been reflecting on is, like, could we have more sensible, you know, security restrictions by default? I think to use the AWS analogy, just as another analogy, you know, S3 used to be this, like, you know, it used to be very easy to accidentally make an S3 bucket public, and AWS realized, like, you know,

48:59it doesn't necessarily cause it, you know, pose a security problem for AWS, per se. But a lot of their customers accidentally would, you know, like, leak data. And so, S3, it's much harder now to make S3 buckets public. And so, in similar ways, like, we're exploring, like, you know, can we make security a little bit more restrictive by default without necessarily, you know, destroying the magic that makes modal good? And I think there's also, you know, it's kind of a separate sidetrack is, like, agents are getting really good at cybersecurity. So, we've actually also, as a sort of, you know, separate sort of learning from this incident,

49:32like, we started using agents internally quite a lot to try to find vulnerabilities. Software hasn't found anything, but, like, they're very good at, like, probing into, like, very deep parts of the system and exploiting, trying to exploit things and trying to escalate and get out of sandboxes. So, you know, it's something we're exploring very actively right now. How exactly do you get the agents to, like, run through the code base? You're just like, hey, like, find security vulnerabilities? Well, no, you just, like, you just put them in a sandbox and tell them, like, can you try to explain? Just run through the entire repo?

50:03Yeah, can you just try to escape from the sandbox? It's like, escape the room, but for agents. Cool. When did you start doing that? We started doing it a few months ago. And it's pretty exciting. Like when Fable, I guess, came out? Yeah, roughly around the same time. Okay. I mean, my view on, like, cybersecurity is that, you know, I actually think they're very good at finding vulnerabilities, but I think you can also use them defensively very well. So I think, you know, companies like Modal and many other companies right now, like, I think, you know, if you go to talk to, like, pretty much every citizen,

50:34I'm sure they're all thinking about the same thing. Like, how do we use agents defensively to, you know, fix all our vulnerabilities? So over time, actually, in a way, like, I'm not that nervous. I think if anything, like, agents and cybersecurity, all the breakthroughs in cybersecurity will make the world safer, if anything. Do you think that, like, all of the cybersecurity, the big cybersecurity companies are just cooked? I don't know if that's true because, I mean, you're asking something that's, like, very far outside of my, like, core expertise.

51:04But when I think about, like, cybersecurity, a lot of what you're selling is effectively fear. And that's almost like a branding and, like, marketing strategy, as much as, you know, obviously the core products. But I think, you know, brand and things like that are actually, like, arguably, like, more valuable in the era of AI. So I wouldn't necessarily jump to the conclusion that they're cooked. I think they have, you know, deep relationships with the enterprise customers and they have, you know, big sales teams who are very good at telling, you know, spooky stories and things like that.

51:36And I think they're going to continue to be very good at that. I just ask because, like, yeah, you're doing it yourself. You're not going to one of, like, the Palo Alto networks or whatever. But I guess, like, it's only, like, giant enterprises that care about that. Also, they can blame somebody if something goes wrong. That is true. Yeah, yeah. You kind of want to hire, in many cases, just, like, an established vendor in order to blame them. I mean, to some extent, it does reduce a little bit the need for us to bring in pen testers, like, human pen testers, which most startups, like, do on a regular cycle. So maybe previously, you know, we would do it.

52:08I actually don't know what the cycle was, but I think we used to do it, like, once every six months, roughly. Maybe now we only need to do it once every year. I don't know. But, like, I think with all of these things, you know, humans can also use them to get more efficiently. So, like, you know, like, I don't know. I'm kind of a believer also that, like, humans with AI is even more powerful now. So, like, maybe there's an argument that, like, we can bring in pen testers who are really good at using agents and, you know, find vulnerabilities even faster. I don't know. I'm not that, you know, I'm an optimist.

52:38I started a company. So, like, you know, I'm generally, like, optimistic about everything. But, like, I certainly don't think humans are cooked or companies are cooked. Like, I think in the era of AI, a lot of companies are going to figure out, you know, how to use AI to be even more efficient.

Developer experience in the age of agents

52:51I want to transition a little bit to, like, developer experience. Because, like, early on, I mean, even now, like, you guys focus a lot on that. Yeah. I mean, you've talked a lot about that. Does developer experience even matter anymore? I think it still matters. And I think you're sort of, you know, like, the context, I think you're official, like, the thing you're talking about is, like, now with agents, right? Like, does developer experience. Like, the docs are already there. The agents are going to look at the docs. Like, all that stuff. To that point, though, like, almost all the things that make developer experience good also makes the agent experience good

53:23is kind of what we've seen in practice, which is, like, things like having good docs, having descriptive error messages, having fast feedback loops, right? Like, I do think there's certain things have changed. The other day, I was, like, setting something up with Cloudflare, and it's, like, I find Cloudflare, like, notoriously annoying to work with sometimes. And I threw an agent at it, and it was able to, like, just configure a bunch of, like, ask, you know, things in Pulumi for me. And, you know, and so I will say, like, agents does, they do make it a little bit easier to work with things

53:55that are, like, harder to work with in the past, you know? Like, the sort of human toll of dealing with YAML has arguably gone down a little bit because you can just, like, you know, tell your agent to deal with it. But for the most part, I think, you know, the things that make developer experience good also make agent experience good. I think to some extent also, like, infrastructure over time anyway becomes and needs to be more programmatic. I think that's always been a thesis about sandboxes, is that, you know, like, things that you previously had to configure using YAML,

54:25that should just be an SDK. That should be more, like, imperative. Like, launch a sandbox. Like, do stuff with it. Tear the sandbox down, right? Like, rather than sort of declarative using, you know, some sort of DSL. It also pushes, like, a little bit of a different interface that maybe previously didn't exist. I think in either way, like, SDKs are as relevant as they always were, if not more. Is there anything you changed in your product just because, like, you know, most people are just running agents on it? We do. I mean, we also, we actually do run a lot of internal tests

54:57where we have agents, we tell agents to, like, build a mobile app and run it. And, I mean, a very concrete thing that is kind of funny is that we've actually seen that, like, because of agent training data, it's very hard to deprecate things now. We see that, like, agents often, like, love using, you know, old versions of modal syntax. And it's kind of hard to nudge them away from that. So we've actually had to be a little bit more conservative going forward, like, you know, over the last year or so with how we deprecate and remove features.

55:28It takes, we have to be very careful to update every single example that's out there online. Yeah. And, you know. You have skills, right? Yeah, there's, like, stuff like that. But, like, there's also an insane amount of, like, modal code that's now, like, online, too, that goes into the training data. I've seen, like, that to be, like, kind of a, you know, a little bit of, like, an interesting example where, like, another thing, by the way, too, is, like, there's obviously, like, to sort of, similar to, like, search engine optimization. There's, like, agent engine, you know, whatever people call it, agent optimization. We often, like, run internal benchmarks.

55:59We're, like, ask an agent, like, hey, like, can you deploy a stable diffusion for us? Use whatever tool you want. And we see if it recommends modal or not. And those things are very useful for us to sort of track over time. Or, like, hey, spin up some sandboxes. Or, like, hey, like, rank all the sandbox vendors. So we spend a lot of time, like, doing those things to make sure. I mean, what do you do about that? Yeah, you say, hey, like, you know, build a stable diffusion thing. Does it recommend modal or not? Like, yeah, what do you do about that? Yeah, I think that's harder. I think, you know, it's like search engine optimization.

56:29I think it's primarily through content. It's like you just have to put out a lot of content. You can also, to some extent, like sort of see the perception, I think, a little bit more. Like, you can ask an agent, like, you know, how does, you know, rank all the sandbox vendors by reliability or rank all the sandbox vendors by price and sort of see. And it does feel like agents are a little bit more prone to, like, mis-understanding pricing and, you know, things like that. So we also spend a lot of time, like,

57:01just making content more clear and more, like, you know, understandable for agents because we know they're all indexing, like everything that's out there and that's talking about modal. Have you thought about just, like, making a bunch of GitHub repos and having the agents, like, write a bunch of modal code in there? Have you tried that? Slop cannon. I think the truth is, like, that is probably something you can do. My guess is, like, LM training data today is probably quite insensitive to, like, what's, like, AI output in the first place

57:31and you can probably, like, create a lot of slop. Like, we thought about it. Like, I don't love that idea, but, like, in theory, like, much like, you know, SEO in the early days and still to some extent, like, auto-generated content is, like, a very effective strategy. I think it's possible, like, you know, you can do those things too. Yeah. I want to switch to FDEs. Apparently, you guys hire, like, forward-of-point engineers? Like, what do they do? We're in a new era of, like, building applications and, you know, things that we couldn't build, you know, just a few years ago.

58:02And we see a lot of companies that, you know, they need a little bit of help, right? Like, you know, and so we started investing in FDEs a couple years ago, but it's really been this year. We just kind of scaled it up a lot. I think we started this year having, like, four FDEs. Now I think we have, like, 24 or something like that. So we've grown it a lot. And they do a, you know, it's more, it's not exactly the Palantir model, but they do a lot of, like, heavy, you know, sort of customer engagement. Like, they work with a lot of customers on performance tuning, especially with LMs. We see that there's so much benefit

58:33from, like, you know, sitting down with customers and training draft models and tweaking, you know, parameters and DLM and SGLang. They often, you know, fix bugs, or not fix bugs, but, like, you know, when customers run into issues, like, they often work with customers trying to figure out, like, how do we work around that? Or another thing we've used FDEs a lot for is, like, just kind of proactively, you know, if we see certain customers, you know, over-provisioning with modal, like customers requesting too much CPU or too much memory, for instance,

59:03like, that's kind of expensive. And, like, we try to catch those things and we try to sit down with those customers. Like, yo, like, you're spending a lot of money in a modal right now. We can actually help you save money. And then, of course, like, with sort of active deals, you know,