⚡️ Which GPU Clouds Are Actually Good? | ClusterMAX 3.0

Building ClusterMax at Semi-Analysis

0:00Hi. Hi, Jordan. Congrats on launching ClusterMax. What's your brief history as a personal introduction and also journey into building ClusterMax? Sure. Yeah, I joined Semi-Analysis last summer, last June, and been working on the technical staff. We've grown a lot over the past year. Anyway, the personal background is that I spent 10 years working at Hewlett Packard Enterprise. I was designing a lot of the hardware systems that these NeoClouds were buying back before ChatGift launched, and then with the big craze in 22, 23, when a lot of them came into the market. I was a big Semi-Analysis reader, read every word, knew a bunch of the guys in group chats on the side, and saw a lot of my opinions represented in ClusterMax 1 when Dan and the team put it out.

0:58And the best way to get a job at Semi-Analysis is to find something that Semi-Analysis is doing and criticize it. And then Dylan hires you to fix it, which was what happened in my case. You got to tell us, what were the criticisms of ClusterMax 1 and 2? Well, I did two all by myself, so I can give you the criticism. No criticisms there. What changed from one to two, and then let's click into what's new and three. You can see it on screen, guys. It's perfect as is. I don't know what you would criticize.

1:30Oh, no.

Version Three and Automated Testing

1:31Okay. Yeah. ClusterMax 1, I mean, there was just a lot of providers missed, like 24 total providers, and a lot of them, big ones, you know, they should have been testing and weren't. Small team, didn't have the time. And then there's a big focus on Slurm. To me, Kubernetes was really important. Need to, you know, test that as well. In terms of like criteria and stuff, we haven't changed a lot, but yeah, all the way back to ClusterMax 1, you know, the ranking page looks a lot different, or there's just a lot less names.

2:01And releasing the market view also, I believe at the time it was around 120 total providers. Yeah, 111 emerging Neoclads there in the middle. And we didn't release the full market view this time because we pay well that behind our Neoclod model, or we track all of these providers. A lot of them are public. A lot of them are going IPO soon. So we've got this whole financial model for the industry now, but yeah, 3.0, we're up at 323 total providers that we track. Um, and I have some people this morning in my inbox reaching out saying that they've got some 3000 GPUs and they want us to test and we gotta, you know, anyway, um, it's been a big evolution.

2:40I think the big thing in terms of testing is that with ClusterMax one, you know, all the testing was hands on. This is pre cloud code, pre codecs, but getting serious. And so we did a lot of the testing hands on by like just running benchmark scripts manually. And now we've been able to develop like this whole repo over the past year and a half of industry standard benchmarks, custom benchmarks, stuff for reliability and burn in. Um, you know, it's, it's like very automated now where we can launch, uh, this stuff with the coding agents and have them run and fix a bunch of issues autonomously without us involved.

3:17And, um, yeah, it's, I mean, just the, the breadth of coverage as well as the depth of coverage, I would say has improved over time. I mean, I guess introduce it. What is ClusterMax? What is Neoclod ratings and rankings? Um, what are like the main things you guys are actually testing for here?

Defining Neoclouds and Rating Criteria

3:35Yeah. So there's 10 criteria described on the website. Basically a Neoclod we define as anybody who rents access to compute, um, sells access to compute. So Neoclod implying that there was hyperscalers and legacy clouds that existed before ChatGPT. Um, this is AWS, Google, Oracle, Azure. Also some of the old school clouds, like a Rackspace or DigitalOcean, or even like a Cloudflare or Akamai. Anybody who's like running a big data center footprint and renting access out, um, there would be a cloud.

4:08And then the Neoclods are the new ones that have come along. We say everybody's becoming a Neoclod because what makes Neoclods different is that they're specifically focused on AI. So they deploy GPUs and other accelerators for AI workloads. This is training inference, increasingly RL. And, um, that's what we try and test on is who's going to give you the best performance, security, reliability, and, uh, just like ease of use pricing availability. We try to go through comprehensive assessments instead of individual benchmarks. Back when ChatGPT came out, a bunch of people were reaching out to semi-analysis saying like, Hey, I'm trying to decide between five providers.

4:49Like, how should I think about this? How should we go about it? They were, you know, bothering dealing with the same questions over and over because like, Hey, I just raised $20 million. I'm about to go give 10, 15 million of it to a provider. You know, it's my whole seed round. Like, this is a really critical decision for me. I need to get everybody's feedback. And so instead of Dylan and team, like answering that question individually cost them every single time. I was like, why don't we write an article to give our like course grain first thoughts?

5:19Here's our like quick tier list rankings, um, podium, and then you can engage us for more detailed stuff, you know, consulting DD, whatever it is for your specific, you know, requirements. And we can dig in further from there. There's a lot there. Uh, uh, the, the one that comes to mind is the, is the criteria of, of the 10 criteria. Uh, the one that comes to mind is the one that you guys quoted, which is earlier. So let's give her a saying, uh, every, all of them have terrible cybersecurity.

Security Flaws in Neoclouds

5:49What's the, what is the state of cybersecurity and neoclouds? I mean, you said it terrible. Yeah. Um, it depends who we're talking about. Look, uh, we, yeah, we, And also I'll, I'll just say hi to Dylan who is, who's just joined us. So sorry for being late. Dylan, what's the state of neocloud cybersecurity right now? So, so, I mean, we, you know, I think, um, security is really bad. Even back from the first ClusterMax, um, there was, uh, pretty significant security issues. Um, people not properly implementing network security controls. And so we were able to see other people's, uh, jobs, see other people's data, see other people's storage, et cetera, in the first generation of ClusterMax.

6:34Um, we escalated to the firm as well as Nvidia, uh, you know, and, and rank that company underperform, of course, and strongly recommended people don't use them. Um, and in the second generation of ClusterMax, we saw similar stuff. I think a lot of people got better because we highlighted stuff. Um, but in the third generation of ClusterMax, we also tested, I think, you know, Jordan and his team tested this and, and saw pretty major security issues. So we posted a, uh, uh, a newsletter titled, uh, most neocloud, uh, suck at security. Um, and then, uh, you know, a handful of hours later, Ilya tweeted about most neocloud security sucking. Um, they're obviously a very security conscious firm. Um, you know, they, they have a lot of folks who are from the former cybersecurity world.

7:22So I imagine they've done their own work on a number of clouds there, uh, to see, um, and test. And it's, it's pretty clear, right? Like what it's so bad in one case, there was literally, we were literally able to see national intelligence. intelligence of a certain countries like stuff, right? Like they had multiple national security agencies and military style, uh, intelligence stuff running in that neocloud. And it's like, good God, we're just like here to test the cluster and we can like see this stuff. Imagine if we were malicious.

7:54I'm sure there were people who were malicious or could have been malicious, um, on that cluster as well. So thankfully they fixed it, I think. Um, but like, you know, we shouldn't be the one that has to find whether or not someone is, uh, you know, good or bad at security, but you know, I think the, the industry generally needs that service. Um, and so that's, that's a big area of our criteria. Yeah. And I think on the technical side, like this is, our checks are very simple. Is your software version up to date? Are your drivers up to date? Like, is it configured correctly? I mean, um, these are relatively simple checks, but when people are running,

8:35people are running versions of software that are two, three years old and they have well-documented CVEs online, it's like, you don't need a frontier model to create a POC exploit of this. Um, you just need to know the right place to look. And so as part of the article, we also put out a free CLI. People can install cluster max, C max, and then use it to check the software versions on their clusters and identify, you know, a quick, like you get a link in the terminal output to show you where to upgrade stuff, trying to help people keep the software up to date.

9:10This is the new cloud security. That's the thing is the network keys, right? There's, um, InfiniBand and ethernet have various, like, at least on InfiniBand side, it's like M keys, P keys, all these things are those properly configured and deployed. Um, it's, it's, it's shocking how bad most, you know, clouds are though. Like it is really bad. Well, you guys, I mean, I guess any, anything interesting, any major jumps, any improvements that they've made from one, two or three, anything you want to highlight?

Hardware Evolution and Raising Standards

9:36Our, our, our bar on cluster max is, you know, it's first generation was like super low. Um, and each generation we've made it harder and harder. Hence the trend has been for people to get downgraded. There've been a few folks who've been upgraded. Of course. Um, Google's done a really good job at getting upgraded across the cluster max generations. Um, I think they started out as bronze and moved to silver and now gold. Um, we're just silver. Sorry. Oh no gold. Yeah, yeah. So Google is, Google's consistently upgraded while there have been a number of clouds who have been downgraded across the generations because they sort of stalled.

10:13They improved a little bit, but they didn't improve a lot. And, and we keep trying to raise the bar, um, and, and teach everyone, Hey, these are all the things you should do to be better. Um, yeah. And the, the bar for this time was like latest and greatest. So are you deploying GB 300 NBL 72 in 2026? You know, this is incredibly popular chip. It's what all the frontier labs want to buy. It's what all of the new labs want to buy. And the typical customer that comes to us and says, I've raised a hundred million dollars. I want to spend it on compute. They want a GB 300 NBL 72 cluster.

10:47And so some of the providers that were in gold originally do not offer GB 300 NBL 72. Why not? Well, you got to contend with a whole bunch of different things. There's a new network and the L 72 implies it's a scale up domain in the rack. You got to go to arm CPUs. It's a blackwell GPU. You have an 800 gig network. It's mandatory direct liquid cooling. So from the physical data center to how you deploy the operating system, how you like manage the drivers in the network, just about everything's different. And when people say, Oh, we just like made a strategic business decision choice, not to do GBs.

11:23Well, what, what happens when Vera Rubin comes next year, who are you going to trust? And so the trend is like, if people are going out and they're deploying 10,000 GB 300s and we test the cluster and it works and we engage with the support team and we, we give them feedback on some things. We think we have issues with, and they fix it in an hour, not two weeks. They're moving up the rankings. Other people that are not deploying these latest and greatest GPUs, or they're doing it and it's completely broken. It's taken them a long time to fix it, or it's, we find security issues, or we hear from customers, huge issues with reliability, which is probably the number one criteria that like these top customers care about.

12:03Um, they're moving down the list. So that's, that's a bit of the dynamic there. So assuming most of these, you know, bronze silver honorable mentions are good to rent from what's the state of availability cost compute.

Compute Scarcity and Profitability

12:16So say you are in Neolab, you just raised tens of millions of dollars, you want to start to rent. Um, how does that market look now compared to one, two, three years ago? Things getting better? Worse? Good fucking luck. It's, it's really tough. It's, it's the toughest it's ever been to rent compute. Um, obviously anthropic and opening. I've been buying smaller and smaller cluster sizes. Um, but also all of the inference providers are more and more profitable.

12:48It was one thing when say base 10 was running at 10% gross margins, which I think might have been like six months ago, right? Um, on inference or nine months ago. But it's another thing when, you know, base 10 fireworks together and many other companies, even small companies like morph are running at 60% gross margins. On open models using VLM or SDLA and using just a little bit of optimizations of beyond that, um, and able to do an awesome job. And then, you know, companies like infra act and radix arc as well. So, so anyone who can like get access to a GPU can immediately make money.

13:20Um, and so ultimately it's like, well, great. I'm a Neolab who wants to train an interesting, cool new model. Well, I'm also competing against people who make money now. Whereas before I was mostly competing against money, losing, you know, opening money, losing anthropic money, losing labs. Very few people were making money off of GPUs. Now everyone can make money off of GPUs. And so it's just very, very difficult to get any sort of compute. What level are the labs? Yeah. What level are the labs going down to? Compared to like major, you know, they, they used to only go like 8K, you know, uh, 8K GPUs from a provider.

13:56Um, you know, and, and, and it's, it's steadily gone down. Um, we've even seen as small as 1,000 GPUs get rented by the labs now. Um, and so 1,000 GPUs is closer to what a Neolab wants. Um, and so you've squeezed out the market a lot. Um, and then when you look at like the base tens, the fireworks, the togethers, and all these other inference providers, they're cool with like, they're cool with like four nodes. Like they're modal, right? These guys are cool with like even four nodes that are cloud. Um, so ultimately it's like, you know, obviously their orchestration is tough.

14:27Um, modal's talked a lot about that actually, right? Their orchestration across like 20 different clouds. Well, it's yeah. So, so, so ultimately it's like a very, um, it's challenging for anyone to get any sort of compute if, you know, modal can make money hand over fist off of even four nodes from a company. Yeah, it's bad. I mean, like even individual, you can't spot rent single nodes anymore, like down the whole stack is cooked. Um, what about the state of the labs? How do you segment the large players, OpenAI, Anthropic, Google, where, where is the split of who has the most compute?

15:01Uh, anything interesting on the roadmaps of where you think who's investing the most? Uh, I mean, it's clearly OpenAI and Anthropic that are leading the way in terms of both renting from like large scale bare metal hyperscalers. In other words, you know, OpenAI is a driver of CoreWeave's business, also Microsoft. They're also exploring with other hyperscalers. They're doing self-filled as well, like massive stuff with Oracle and then Anthropic. Obviously they got their big AWS partnership with Rainier. They've also got a lot of TPUs with Google Cloud.

15:31They signed stuff with CoreWeave. They've signed stuff with, you know, others. And, um, so they're, they're the big driver. I mean, you just look at ARR and something has to drive that. And so it goes to the compute side, um, with that.

Managed Services versus Bare Metal

15:46Another aspect of this is, um, you know, a lot of these providers like, like, uh, Crusoe have fantastic bare metal, right? Um, but that's not what we're testing in ClusterMax. We're not testing, um, bare metal. I mean, it is part of the criteria. Um, but a lot of the criteria is a managed cluster, right? Managed, managed Kubernetes, all of that sort of stuff. So it's, it's a different level of, um, service that people want on top of the GPU. Um, the original ClusterMax was titled also like a big chunk of it was called how to rent a GPU. We were also just teaching people what they should do to rent a GPU.

16:17Um, and in our view, most of these smaller folks, you know, if you're super small, you don't care about, you know, your node being bare metal. Um, if you're super, super big, you want bare metal because you're going to put your own, you know, uh, fork of, of Slurm slash Kubernetes, um, and do your own stuff on managed services and such like that. Um, but most customers want a managed Slurm managed Kubernetes, um, and for the cloud to like hot swap nodes for them. When one fails, uh, you know, the biggest customers will manage stuff like this, but you know, the very, the, the midsize and smaller customers want that sort of stuff managed for them.

16:52And so ClusterMax is focused around that. Um, so companies like Crusoe, um, will rent bare metal, right? They have massive contracts with companies, um, or they'll do great in data centers or they'll, they're starting to do well in, you know, inference API, but on managed services, they're not doing well. Um, and, and that specifically is what ClusterMax is targeting is the managed service layer, right? Cause it's otherwise it's like too complicated to, uh, you know, it's, it's, it's many different areas people want to rent from. Some people want bare metal, some people want mayor services, some people want tokens out.

17:23Um, and so ultimately it's a range of options. Um, as far as, you know, even Google is renting some compute from folks, right? They signed a deal with core, core weave. They signed a deal with SpaceX. Um, so, so pretty large amounts of compute that they've signed over, I think over a hundred billion dollars worth of compute that they've signed from various, uh, customers. Um, Nebius also has deals with the labs and such. And so, um, but these guys are only going for bare metal. Um, open AI and Anthropic are definitely lion's share buying the most.

17:53Um, Google, Google is definitely falling behind at this point. Open AI and Anthropic have as much compute. If not a little bit more by the end of the year, they'll both have more compute dedicated to R and D than DeepMind. So yes, Google still ultimately has more compute than open AI and Anthropic for now. Um, but a lot of that is sold out to other places and dedicated to other offerings. Um, but DeepMind's R and D compute versus open AI and Anthropic's R and D compute. The two labs have more compute now because they've reached that scale.

18:23Um, so this whole like, oh, Google has the most compute they'll win is, is actually not really the case anymore, right? They still have a cost advantage because they build the TPU and, um, it's much lower cost, but ultimately it's, it's, it's less. DeepMind has the less compute than open AI and Anthropic. Yeah, there's a, there's a big, big shift. Like one of the, uh, uh, the biggest insights for me at AIE was running into some DeepMind researchers. And they were very depressed because they were like, look, our leadership keeps selling the compute that we want to other people.

18:54Yeah. Yeah. Yeah.

Big Tech Shifts and M&A

18:57I mean, imagine being XAI, right? Like. Well, yeah. Okay. You know, I thought SpaceX was a new cloud. Like what the hell? Why are we, why are we unavailable? They're on here hard. Yeah. Yeah. Well, as soon as we can test, they'll move up the rankings. You know, um, they'll be a competitor. I think, uh, SpaceX and the Google deal, for example, is super interesting as well as the Anthropic one, because it goes to what Dylan was saying about the fact that the labs are considering becoming new clouds on this opportunistic basis, um, Meta's considering the same thing.

19:28We see Mistral and poolside in there as well. The companies have this skillset on building data centers and managing them. And then when there's an opportunity to monetize the compute, either through renting it to other people or through serving inference, they want to pursue that, but only to a certain level. And they always want the optionality, bring the compute back and use it for those research or hero runs as soon as they can afford to. They want to use the maximum amount of compute they can afford to research right now.

19:59And all the extras are, you know, opportunistically making the money to fund all that research. And so SpaceX's contract with Google, for example, has mutual 90 day cancellation rights. So SpaceX's research starts to work. They can pull the compute back. Google stops making money off of that compute on some inference endpoint. They pull the compute back. It's a really interesting structure where both are kind of, you know, in this little standoff saying, who's gonna, who's gonna blink first?

20:29Who, who believes in AGI more? Um, as going. I think, um, the other aspect of, of, uh, at least for SpaceX, right? If you're an XAI employee and you got bought by SpaceX and then SpaceX sold all the compute, at least you're not depressed because like, you know, you're like, okay, look, Elon's building a shitload more. I'm gonna have access to this. And he's doing this because it helps us be more financially solve it. So for the IPO, um, with Google, it's like, why the hell am I selling the compute?

21:00You know, I have, I'm already a very profitable company. Why, why can't we just use the compute to build models and sell those models at way higher margin instead of giving it to Anthropic or open AI or other firms? Um, you know, SSI is also a customer of Google. Dylan, what do you think about the, um, the dynamic of like Google funding the people that leave to go do a neo lab? To buy GCP. In other words, like Dave Silver, Jeff Dean, they spin out, they get funding from Google ventures to then go spend on GCP.

21:33I mean, I think it's, it's, it's a fantastic move. If, if, if you look at just GCP as its own company, right? Thomas Kurian, uh, continues to pump his, um, you know, his revenue, his operating margin, his operating income, but for Google as a whole, it's really dumb, right? Like you should not incentivize your best researchers leaving. You should not incentivize. So Jeff Dean has a breakthrough. Jeff Dean sold at $20 million a megawatt when you could run models on it for $50 million a megawatt. So if Discovery Loop has a big breakthrough, do you think Google is able to keep them in their orbit?

22:05Or do you think Jeff Dean goes and takes that breakthrough and does something without Google being exposed to any of the upside? Like they would be an investor. They're, they're obviously the cloud service provider. I mean, I mean, go look at, uh, Anthropic, right? Like, yeah, true. Google did invest in Anthropic. Um, but if Google didn't invest in Anthropic, would Anthropic be here today? Because Google is a really early believer. Um, like Opus three and, and Sonnet three and three, five, like those, I think were just trained on Google's infra for sure. Um, you know, whereas, and then they later brought on Amazon, which was, uh, you know, Amazon's also great, but at least Amazon, you know, is just like,

22:42like they don't have a business of training models and selling them right in the same way. They have some folks, but like nothing, nothing crazy, but Google does. And so it's like, did Google like stand up, uh, ultimately their biggest competitor, uh, and fighter. Now Google, now Anthropic is not only renting from Amazon and Google, but also from numerous other clouds. We have deals with like 10 plus clouds now. Um, and then Anthropic is also building their own data centers, leasing their own data centers, uh, starting to fill them up, um, working with fluid stack to deploy TPUs in their own data centers.

23:15So sure, Google's still getting revenue, but it's like, well, I used to rent from you. Now I'm buying TPUs from you, which is less margin for you. And now in the future, I'm gonna, I'm now building my own chip. I'm gonna do everything without you. So, so like, it's, it's pretty clear what happened with Anthropic. Um, Jeff Dean, if he does something amazing, um, in discovery loop, then why, why would they continue? Why would they be stuck to Google, right? They can use any cloud. Um, switching clouds is ultimately not impossible. It's not a, it's, it's not, you know, data gravity and all this stuff that people used to talk about with clouds.

23:45Uh, in the pre AI era is a lot less relevant in the post in the, in the AI era because AI can just help you migrate everything. And a GPU, a GPU is not just a GPU, but like, you know, a GPU is largely, um, workable as, as a fungible asset across clouds if you're good. So X, what do you think? You think Google's cooked, man? I mean, I think you guys are much more of an expert than I am. Uh, and there's like, yeah, there's Google's just like a conglomerate of a bunch of different things, right? Uh, so I, I, I do think like the, the data advantage, I mean, there's data gravity as, as, as NeoCloud, but, uh, the data advantage of Google is the, the thing that most people could keep coming back to.

24:24Right. Um, and, uh, there's just organizational challenges and like, you know, now there's new, new leadership. Um, you know, God bless to Corey who has the hardest job in the world to, uh, to fix this. But like, I mean, he's a very, very capable guy. Um, and like, I think, I think they, they know, like, like, like, you know, it's, it's like not a secret that like, uh, they do need a reset. And, um, to me, Demis, like basically stepping out of the leadership seat was just like a complete black swan.

24:55Like, I just, like, I never thought it was possible. Like, maybe I just don't know him personally where like, he's like, yeah, actually like, I just care a lot about isomorphic, but like, um, I mean, you know, he, he, he founded it. Like he's, he's, um, you know, he's, he's sort of like the spiritual leader of DeepMind. And, and so like this, this, like, this is probably the first major, uh, Frontier Lab transition, leadership transition fully. Um, can we say that? Like two days where Sam left, right? But, uh,

25:25Yeah. I mean, he never left in, in anyone's like hearts. Uh, but yeah. This is the Tim Cook of like, hi. Yeah. Ilya leaving open AI. No, he's not the leader. Yeah. Yeah. Yeah. Yeah. Yeah. It was like, kind of like the, yeah. Yeah. I mean, yes. Okay. So like, yeah, obviously people split, right? Like anyway. Um, so, uh, that's my rough take. I don't think I'm going to have like, uh, spicy takes on you guys. I did want to pick up on one thing. Actually, this is like just the chance to like catch up on like the last six months of like

25:57crazy news. Yeah. You guys mentioned poolside. You've never, never, never actually written about poolside because they're not that big. Um, but, uh, there's, they do have Africa. Uh, they got bought by, uh, their, their, their, their, their employees got bought by NVIDIA for 12 billion. What's going on? Okay. Yeah. Um, it's really cool, right? Cause NVIDIA is paying for all these researchers so they can make Nemo Tron better. Um, no, this is defense side is poolside poolside. Like, you know, from the beginning, like they, they just like all the cloud, Neo clouds

26:29kind of sucked. And so they started building a lot of their own infra and they got better and better at building their own infra. And then they were trying to deploy their own clusters and had a partnership with core weave on this and stuff like that. But they've just been slowly moving down the stack. Maybe the first AI lab to, uh, become, you know, running their own infra. Even before open AI and Anthropic were trying to build their own infra, poolside was trying to. Which is wild as a side projects. It's like, people have to do this as a full-time job. This is a side project. Well, now it's their full-time job, right? Cause they sold all their researchers to, uh, NVIDIA for Nemo Tron.

27:02Um, cause NVIDIA clearly wants Nemo Tron to be a lot better. I think they've sort of lost hope in a lot of the, uh, um, open source folks. I mean, sure. Reflection is still trying, um, thinking machines, open source some stuff, but ultimately I think NVIDIA is losing sort of hope in American open source. So they think they just have to do it themselves. Um, so they've bought a bunch of employees. Um, and yes, I thought that was $7 billion, not a, not 12. But anyways, regardless, um, I thought it was like a $7 billion and then a $1 billion

27:32investment out of 12. We have the, we have the details. Yeah. Yeah. 12 billion. I took the bigger number. Licensed and employee. They're sick. The number is sick. Always, always take the bigger number. But you know, I think at the end of the day, it's like NVIDIA can say, well, actually we only paid one fourth the price because all of this money is just going back to buying GPUs. Um, so did they pay, you know, $7 billion, six for the employees and one in an investment? Or did they actually pay, you know, one and a half because all seven of that is going into buying GPUs. Um, and in fact, them doing this is going to incentivize other people to invest a bunch

28:06of money into them. And so actually it's a, it's, it's probably a positive thing to hire all these people. Like they, they actually didn't spend any money because now they've created a new customer, a new cloud that's going to build a massive cluster, right? Like imagine, you know, they've got $7 billion now of cash. Um, they raise another few billion dollars. Let's just say they have $10 billion of cash on the balance sheet. Um, and they're able to get a loan to value of like 75%, um, you know, 25% down and 75% loan. So now they have $40 billion. They can build a gigawatt. Awesome.

28:36This is another gigawatt scale neocloud now. Um, so they've just, you know, a new competitor. We have that lower down in the, in the article. Um, it's, uh, yeah. I mean, I, I don't know. I don't know. Uh, if that, I mean, this is the playbook. If this is the playbook, honestly, um, uh, lower down, lower down, um, uh, with the guy who's leaving it, Robert Bonar. Um, and like, if this is the playbook, a lot more people should be doing this. Should be doing what should, should be building, should be doing deals with NVIDIA like this

29:07because NVIDIA will king make you just cause they, they want more neoclouds. Yeah. I mean, not everybody releases the second best open American model and spends years building it. I mean, we're kind of brushing over the fact that poolside did build something interesting. And to me, both this and hugging face are defensive acquisitions by NVIDIA who does not want their competition, snapping up a lab that actually has something going for them and having some momentum where they can build the real experience that's required to build real training and inference clusters.

29:40In other words, it's so important for building an infrastructure company that you actually understand how the workload works and you have opinions that are informed by training models and running it gets represented in the, not crypto miners who decide, Hey, I found some site. I got some power. I have some money. They don't have the ability to execute like somebody who has proven they can train the second best American open model in the world, which poolside did. Right. Yeah.

30:10Say more on the, I was going to say, you mentioned hugging face in there. How do you guys see the hugging face acquisition? Anything you want to comment on there? I think it's quite, quite similar. I think it's defensive. I don't think that NVIDIA wants the influence in the ecosystem of other companies coming in and owning somebody that's like critical infrastructure. There's a galaxy brain take here, right? Which is, uh, now, now hugging face, you know, can, can sue the fuck out of opening eye for hacking them. Um, and now NVIDIA can, can, has, has that, has that hammer.

30:40No, I don't think so. I don't think so. I don't think so. Um, but I think, I think, I think hugging face is obviously like, uh, you know, leagues ahead of everything else. Um, some, some folks on my team have started submitting a bunch of PRs for model scope, uh, to work across the ecosystem. Um, By leagues ahead in what sense? I just like for, for being open source repository for. I see. Yeah. Just like what hugging face does. Um, but then ultimately a lot of them are, uh, you know, like it's, it's, it's, you know,

31:14the question is like, does hugging face, you know, there's a way you can just like deploy hugging face models, um, on Amazon Tranium if you want. Right. You can just click a button and it'll deploy them. Um, does that go away? And is it only a GPUs from here on out? Um, or is there any closing up that happens? Um, we'll see, but ultimately it's, it's, it's, uh, I don't know. I don't know. I think, I, I also love that the price was, uh, the emoji code. So sick. Well, and the team's incredible. Look, I, I think we're, we're kind of brushing over the fact that hugging face and poolside

31:46have incredible talent in both these organizations. And like a lot of people would see that's the motivation for NVIDIA to acquire them. I say it's defensive because if you look at NVIDIA's business, they're generating almost $50 billion of free cashflow a quarter. So 200 billion a year, and there is a corporate strategy team at NVIDIA that gets some amount of that money and says, what, what we need to deploy it, right? They can make investments. They can make acquisitions and 10% of free cashflow to acquire hugging face and poolside. I mean, it's a no brainer to me that these would be good assets on your team at working.

32:22At least not for your competition. Let's say hugging face stays independent and they follow the letter of the law and the acquisition and they keep supporting all these other chips and stuff. If they were acquired by somebody else, they would be actively working against NVIDIA's interests. And I think that's not what they want. So I think you're going to see more of this. If companies grow up a little bit, they build this team, a hundred or 200 incredible people. Um, they want to bring them into, make sure that they stay in the NVIDIA ecosystem as opposed to going and competing against them with somebody else.

32:53Yeah. It makes sense not to discredit anything poolside's done. We're a pretty big fans. We had them on the pod a month before they got acquired. For people interested, the model factory worked approach to training foundation models. Uh, very, very good way that they built all this out. Yeah. To me, you know, coming, you know, maybe, maybe this brings us back to ClusterMax, uh, the way that you, that all NeoClouds should be proving like custom X4 is like, just train a front-end model, please. Uh, your, yourselves. Right.

33:23Uh, and then like show everything works. I don't know if everyone should, everyone should have the skillset to train a front. That's just like kind of, there's a lot of open recipes and stuff. Right. We need mosaic. Let's, let's talk about what it takes to get there. Right. If you're going to train, I think also, I think poolside and probably hugging face have probably both entertained, uh, acquisition options in the past to the tune of similar money. And it's yeah. So specifically NVIDIA tried to buy both side, uh, sorry, uh, if you try to buy hugging

33:54face, uh, at the start of the year. Right. In case people didn't know. Well, I mean, I think also NVIDIA is on a bit of a acquisition craze. There were rumors of thinking machines and a whole bunch of NVIDIA acquisition. Yeah. I call it founders kind of have the, they have the option to like cash in when they want for a lot of these people that have built these things. Right. So when do you decide to cash in? I think poolside is probably looking out there and going, okay, I've built the 10,000 GPU cluster I needed to get Laguna out. But if I want to compete, I need a research cluster.

34:25That's a hundred thousand GPUs, 400,000 GPUs. Like, how do I keep up with the open frontier into next year? I think that's pretty hard to keep going. And the money comes and they're like, okay, sure. Let's just cash in and build it with the other people. That, that seems fine. Um, but the, the other thing worth pointing out is like on the technical side, we've, we've poolside papers are really good on the infrastructure stuff. I've talked to the team. If you're going to train a frontier model, you need to have health checks on your cluster.

34:56You need to have high reliability. You can't just have this model, like not make progress as you're training and constantly be failing. Um, poolside has that. They have like an incredible suite of, um, they have, they just have incredible infrastructure. So the, uh, the idea that people with that talent would be able to provide that as a service to other people, I think is like just obviously true. Uh, and there's only limited people that have that experience. As we're saying, like, even when we get into the silver and bronze tier here, um, people are not training frontier models on most of the silver tier and all the bronze tier.

35:29Let's say, uh, a lot of inference, bit of RL, a lot of research, but I wouldn't trust any of those guys, frankly, to build except Azure, right? But to build a hundred thousand GPU cluster, like who has proven they can do that already? Yeah. None of the five companies. Yeah. Yeah. Um, okay. Um, I do want to maybe, uh, dive in on one or two things that maybe you don't, uh, don't actually cover. Maybe the best way, uh, given the time that we have left is to just address the criticism. Um, uh, you know, you've really, uh, obviously this is a very valuable list.

36:03Like people do be purchasing decisions off of this, uh, periodic lads, uh, did come out and say like, yeah, actually like, uh, you know, this, this helps us decide what, what, what we spend on. Um, uh, yeah, I mean, uh, you know, what is the criticism that most people have?

Addressing Criticisms and Methodology

36:20I think, I think there's, um, a lot of criticism for a lot of things. Like first and foremost, uh, this is testing managed clusters. Um, so that does not mean that a company is not good at other things, right? Crusoe, it was downgraded for managed clusters. Uh, but they're one of the best data center builders in the world. Um, if not like the best, they have, they have, they have nearly the most contracted or, or the most contracted gigawatts in their pipeline and a bunch of sites that they're building on really rapidly. But this is not a criticism against their, you know, data center building capabilities.

36:53It's only a criticism against their managed cluster offerings that they were, they're still one of the better ones. Right. But they were downgraded. Right. Um, you know, likewise, you know, other companies together has great inference endpoints that has nothing to do with managed services. Um, you know, and, and so I think there's a lot of, uh, confusion around what this is testing, right? This is for managed clusters, not for data center construction or bare metal clusters or for inference endpoints, which is starting to become a blurrier line because everyone is doing these things. Right.

37:23Um, the other question, the other big major criticism we have is like, oh, you know, people are like, oh, are they taking money for this? It's like, no, we do not take money for any ranking. Uh, that would be illegal. Um, I myself have personal investments in Crusoe. I have personal investments in fluid stack and neither of those companies. Um, you know, I, I did SPVs in both of them. Neither of those companies are in the top, you know, five clouds. Um, because, because that's not what, you know, I'm, I'm not here to be biased, right?

37:53Jordan will say he's like, he's like, he has free reign to like do whatever he wants to do technically. He doesn't even know all the companies we do contracts with. Um, if I look at the top 20 companies in the world, there are five companies that are not our customers. Um, you know, like Eli Lilly and Saudi Aramco and stuff like that. But ultimately it's like, those are our much bigger customers than core weaver Nebius could ever be. Right. Because, you know, they, they just have way, they're way bigger companies with way bigger budgets. Um, you know, and so like, AWS is a bigger customer than core weaver Nebius, for

38:24example. Right. Um, and yet that has nothing to do with the rankings, right? Everyone in the infra supply chain buys our stuff. Um, it just is what it is, right? If you're tracking data centers, you want to know other people are doing their data centers. If you're tracking accelerator supply chains, if you're trying to understand the cost of various kinds of chillers, if you're trying to, uh, understand the supply chain for networking and optics, you're going to buy our research and our data services. That does not mean, um, and you'll often commission us to do, uh, work beyond that. Right. Um, you know, but that doesn't mean that we're going to ever bias our rankings. Um, you know, so, so sort of, I think, you know, there's a lot of criticisms around

38:57that, but ultimately like, first of all, if we were taking money and ranking people and not disclosing it, that'd be illegal. Um, but second of all, right? Like, you know, it makes no sense, right? The, the value we have to the industry is that we tell the truth and what we believe. Um, if, if we didn't do that, then no one would care what we would say and we'd be screaming into the void. Um, so our reputation is like what matters the most, right? Over a quick buck. Yeah. Maybe the one way to put it is like, uh, you're, you're, you're a buy side service and people, and people are accusing you of taking money for the sell side.

39:27And they don't understand the buy side is like probably 1000 times larger, uh, in terms of revenue for you than the, than the sell side.

39:35I mean, I think people trying to poke at it's a paid list, you know, and that's just one IQ. The, the broader point is like, okay, people are misunderstanding what you're trying to rank here. And a lot of the claims are just, well, it's crazy that this is a better cloud. Uh, X is a better cloud than Y. Right. But that's, that's not what you're actually testing. I hear two, uh, clear criticisms about the writing from people where I listened to the criticism because I, you know, respect their work and I, uh, you know, they're not

40:06like anonymous, just shouting into the void. And those criticisms are, uh, you guys didn't write enough. You should have released more data. You should have written more, or this is 30,000 words. I can't read this all. So it's really hard for us to balance between those two. But it's, it's a new, yeah. You're evaluating new clouds. Of course. Like, um, so for, for the, for the former, I'm, I'm, you know, interested in that criticism and, and we, we, we wanna do better.

40:37And so we have this research product called the cloud TCO model that anybody can come and license. It's a yearly fee. We write notes every week about, um, the trends in the industry. We have a dashboard, which we took a redacted screenshot of below where you can get all of our data, all of our performance data is on there. Um, yeah, it's, it's like, you know, you, you get us working on the research for you year round, weekly, rather than just, uh, for this individual, you know, kind of once a year

41:09release or every six months or something is what we do. So that's for the people who want more detail. Um, there's another crowd of criticisms, which is like, um, you know, it's oftentimes someone who like owns one of the stocks and we rank them poorly or positively. Right. Like I remember last time people were like, like, by the way, like our view on a company doing better or worse in the stock market has nothing to do with, uh, Jordan and his team. Right. There's, uh, you know, we, we do, we do estimate revenue for all these companies going

41:40on a forward basis. Um, and then you can like license our data and see what you think that should mean. You should trade. We never recommend us to trade. Um, but there have been times where like, for example, um, we thought Nebius was better than core weave. And the reason we thought Nebius was better than core weave, even though we think core weave is still a better cloud than Nebius is because Nebius is contracts are shorter term. And so with prices of GPU spiking, Nebius gets to take the upside better. Right? Like that's something that we've released in our research in a separate, separate group of people, not Jordan and the technical staff that are working on cluster max or, or,

42:11Hey, like Iris energy has this massive data center that's going to come online. We think it's going to get leased. And we would put a report out about that while at the same time saying you should never use Iris energies, um, iron's, uh, managed services. Um, and so like, it's like, it's, it's, it's, it's completely different things, right? You know, they sold that data center as a bare metal to Microsoft. And so we were, you know, our research on the institutional research side for someone who might be buying stocks is completely different from like, Hey, cluster max is really made for people like periodic labs or applied compute or companies like that who have publicly came

42:44out and said they look like, like cluster max and what we do there. Um, you know, even opening. I said they like cluster max and we have like a quote from them on, on that. Um, we have a variety of different buyers who said they really appreciate the methodology and that we are pushing the bar up for all the clouds. And so I think it's, it's, it's, you know, if you're trying to buy stocks based off cluster max, that's like completely wrong. It has nothing to do with stock performance. Yeah. It's a hundred percent, man. This is strictly for buyers. This is for helping people buy compute.

43:15It is not for people to pick what stock to Yolo all their savings into. I mean, it, it, it will correlate, uh, at, at some kind of limit. Um, I think like, I think there isn't a tendency to literally, it's literally intercorrelated for my body. You guys are so okay. All right. All right. All right. Jesus. Oh my God. Right. Like someone, someone is terrible at running a cloud, but has a gigawatt data center that they're building. And, and it's like actually credible, like a gigawatt data center. Well, they could be the worst ranked cluster max and they could still make shit loads of money.

43:48Um, it like, it like doesn't matter. At the current price, at the current price of the stock, many names are cheap. Many names are expensive. Like. Yeah. Trust me. Yeah. I do. I do. I do get it. Fundamentals versus, uh, you know, one offering, uh, that contributes to your overall fundamentals, that contributes to the overall, uh, valuation and direction. Sure. It's a data. I, I used to be a hedge fund guy. I get it. You know, I'm just saying like, this is like, you guys are the sub domain experts and people do value the rating on that elements. Uh, but obviously this is way more by side, uh, and bias and, uh, you know, the, the,

44:22the new lab. Yeah. I started to just keep stressing this, but we genuinely have a core research project where people report on stocks and give opinions about the names. And many of the guys that work on that research, look, they ask us questions, but then they come out and they say people in the underperforming or unavailable tier were positive on the name. Um, this is different than what we put out publicly and say, you should rent clusters from. And maybe I need another disclaimer on the future one. That's like, this is not a cosine of the current valuation of their stock. It could be, you know, it could be really interesting for like a two by two, uh, or like a

44:54two dimensional chart where like, here's the cluster max ranking. And here's the, uh, same analysis, like stock ranking and like it, it, it, the differences are gonna be interesting. Yeah. By the research. Um, maybe worth saying though, there is, uh, a lot more coming, which is that, uh, Dylan mentioned it, individual names and logos on here have businesses that are much larger than just managed clusters, which is what we're raking here. And so we are going to be addressing that on the newsletter for inference endpoints for bare metal, for our infrastructure, for sandboxes, for, you know, part of our L infrastructure,

45:28but kind of different, uh, we're gonna be writing about all of that and stay tuned for that because I think people will start to have a more holistic view of these businesses that compete in different market segments across inference managed clusters and bare metal. And we've got a lot of focus on managed clusters here, but frankly, so many people are reaching out to us for help deciding on which endpoint provider to go with, or which hosted RL training provider to go with, or which sandbox provider to go with. And that's a different question than who to go with for your managed cluster.

45:59Yeah. So, so is the solution more maxes, more benchmarks? Yeah. That's the point. Yeah. Okay. How many do you have? You have one on like, when you go shopping, when you go to the grocery store, when you're hungry, it's the same way. Right. We want to do everything. We'll see. We'll see. Hopefully we can execute on all of it. Yeah.

Expanding to Inference and Hardware

46:20Maybe to be clear though, inference X today, the name is because it's for inference performance, but it's about chips. It's about comparing chips and systems. And we're going to be releasing something called endpoint X, where we test the serverless inference endpoint providers and kind of dig into those details on. Because I think there's a very similar methodology. We know there's a similar methodology that we should apply where we go beyond just performance. We, you know, we look at cost. We look at reliability. We look at security. We look at monitoring infrastructure. There's all of this stuff that goes into an inference endpoint that says just because somebody hits a higher target on batch one, you know, contrived tests doesn't necessarily mean they're the right partner to scale up your inference requirements with when you need all that extra.

47:07Yeah, there's providers with better performance, like raw, better performance, better optimization engineers, but then they're like API is less stable. It's flakier. You know, there there's there's all sorts of reasons why it may not be as good for the endpoint and customer, even though it's like more tokens per second per user and lower cost. Yeah, amazing. Um, do you ever how will you cover the let's call it the the non Nvidia universe? Um, so which is right.

47:39I mean, you have some in here, right? Like tensor wave is AMD cloud. Yeah, that's another one where I explicitly have an investment, by the way, guys, and and we still rank them poorly. Um, you know, prime intellect had investment in the first cluster max, but we put them as underperform, right? I think like a lot of this bias against like it's it's yes, I might make an investment. I'm going to talk to them and try and help them. But at the end of the day, the ranking is the ranking. Um, so we we do want to extend beyond, uh, but ultimately, like, you know, fluid stack is not going to rent TPUs to us, right?

48:11Because they're renting everything to Anthropic, right? Maybe. We'll see. Maybe. I hope. Yeah, it's kind of an excusable that they're unavailable. Like, come on, guys. Like, no, I mean, it's it's it's valid at the same time, right? It's like, look, it's unavailable to us. So we can't we can't we can't rent it. Yeah, we'll be able to rent it either unless you're really, really freaking cool. Um, because they're selling everything like SpaceX is not going to rent to us. They're not going to rent to us. They're not going to rent to, you know, random companies. They're going to rent massive costs. And Dylan, I think, I think you need to work on your, uh, I think you need to if this is the first time I'm ever giving you this feedback, but I think you need to work on your self confidence, man.

48:48We, we, we can get it in there. I think we can get in there with these guys. Yeah. Well, I think over time, like today, a lot of these providers, um, even fluid stack and people that we know who are at SpaceX AI now, um, see value in reading the article and they see value in getting an outside opinion. Even if they're not necessarily going to provide the service publicly that they offered to us, they, they, you know, they, they enjoy a third, you know, a technical opinion that's outside the company. That's honest.

49:18That's going to give them real feedback. And we're, we're here to help for all of that stuff. Um, to answer the question about multi-silicon going to be huge next year. Like with the Vera Rubin generation within video, we're going to see it because they're going to bring the LPUs in from croc, the LPX systems, right? So even if we're just testing in video, we're going to have multi-silicon to test. AMD obviously has their own thing. Cerebrus is becoming a Neo cloud for opening. I are building their own data centers, right? Um, moving from just inference, end point stuff to like really managing clusters of their chips for everybody else.

49:50We see Salmonova coming to market. Lots of the chip startups, like Positron etched med X, like lots of people are launching stuff like this. Um, there's rumors, you know, obviously fluid stacks got TPU outside of GCP. There's more stuff coming there. Tranium lots, AWS can produce a lot of chips. Maybe they won't sell, build all of it in the future. They've got announcements on heterogeneous Tranium with Cerebrus and stuff like that. So I think there's going to be a lot more for that next year. And we're going to have to contend with multi-silicon as part of a comprehensive offering from the leading Neo clouds.

50:24They will need to be able to support heterogeneous compute in the future. And that's all focused on inference mostly, right? Like, uh, because of it's, you know, it's like... I mean, Anthropic, for example, uses TPUs for training primarily. Um, so in that case, it's sort of the opposite, right? They don't actually... Most of their inference is on Tranium and GPU. Most of their training is actually... Especially like, you know, Fable 5 was trained on TPUs, for example, for the pre-training. And then the post-training and inference is mostly Tranium and GPUs.

50:55So it's a multi-silicon... I would say in most cases, other silicon offerings are for inference, but there are some cases where people are using it for training. Yeah, but I think the key thing is inference versus training is not this hard to split now where there's lots of forward passes during RL when you're doing research and when you're actually post-training the model. And so... In fact, most of the workload is forward passes, right? And so you might call that classically inference, even though it's for, you know, forward passes for reinforcement learning. Yeah, inference is part of the loop.

51:27Yeah. There's a whole bunch of benefits of using, let's say, cheaper chips capable of high throughput for the forward pass in RL. But then there's a bunch of challenges with numerical correctness if you're using two different chips, even just two different generations of NVIDIA GPUs. B200 versus B300. You might... You have different amounts of memory, different patch sizes. There's all of these effects. So... I absolutely expect there to be heterogeneous silicon used across training and inference. Pre-training, probably a homogenous cluster.

51:57People probably not going to risk that at like 400,000 GPU scale, but we'll see. Yeah, but like, you know, RL is like as big or sometimes bigger than the pre-trained workload right now, right? And so that actually means that you need like maybe a new form of disaggregation where like you have different kinds of compute in the same data center. It doesn't even have to be the same data center, right? People are doing RL across many, many sites, right? This data center is doing token generation. Yeah. That data center is doing token generation.

52:27And it's actually fine to do different types of chips there. It's just a very complex infra problem because you're going to shard your model differently and weight updates happen at different times. And there's also... It's just speed of light. Async. For people listening to this right now are like, God, no, please, no. You have to. Multisilicon RL updates on the fly. God, please, no. You're fighting speed of light. How are you going to transfer all this data? I mean, there's interesting architecture stuff being done on disaggregated, like decentralized RL.

53:00The Mimo 2.6 that just came out, it's a good open paper on how they super scale up the RL side of things. But on that point as well, you mentioned a lot of the custom chips, anything on Huawei, Chinese chips? Do you guys look at them? Yeah, we're tracking them. The Huawei chips are really getting better much faster. For a while, they were like, you know, basically when they were banned by the US government, it took them like a few years to reset. You know, and the Ascend 910, B and C, but now they've launched the 940, 950 and 960 in pretty quick succession.

53:39They, you know, we're trying to sort of get access to them and test them along all bounds, right? Whether it be renting them so we can test the performance, whether it be, you know, looking at the specs and the manufacturing capacity and things like that. Also trying to access them so we can do a reverse engineering analysis out of our Oregon reverse engineering lab. We're trying to do work across all of these dimensions, you know. How's the availability? Are there clusters? Like are the labs okay without NVIDIA? Any early signs?

54:12In general? Yeah, I mean, people are super, I mean, the labs are using, you know, Anthropic uses four different. No, no, I mean like the Chinese labs using their own chips. Yeah, did you see that? Yeah, yeah, yeah. So I think it was like GLM said explicitly, right, that they had 50,000 chips that were non-NVIDIA. They were Chinese made AI accelerators for inference of their newest model. As we extend out to other firms, you know, CameraCon, Ilvatar, Inflame, and many others, there's a handful of different companies.

54:46They're all pushing up the curve, you know, obviously they're further behind NVIDIA. Huawei is definitely the best of them all. And then Baidu is spinning out their chip business as a completely separate company. So there's a lot of developments in the Chinese chip ecosystem. And especially for inference, I think, you know, the Chinese, you know, as frontier models become, and even now the open source models become better and better at writing kernels, it's easier and easier to port stuff. And so ultimately, you're going to have more diversity and heterogeneity,

55:18which is why NVIDIA has moved to this super fast pace of trying to release new chips every year, and new systems every year, because they know they have to run as fast as possible at the speed of light. Otherwise, they will get caught up to, because the CUDA mode is rapidly being deteriorated. But it's just volume, right? Dylan, like the, these chips do work today, but they can't produce enough of them to really serve all their domestic demand. So. Yeah, I mean, it's, it's, you know, you know, if you look at, if you do an H 100 equivalents,

55:49or let's say if you're doing GV 200 equivalents, NVIDIA is making, you know, millions and millions of chips. Google's making millions and millions of chips. Amazon's making millions and millions of chips. AMD's making millions of chips, right? But then if you look at the Chinese companies, really Huawei is the only one that even in gross chips, they're making a million plus. And then the Huawei chips are behind. So then you have to discount them for that. Everyone else is still in the hundreds of thousands of units and scaling production is very difficult, especially given export controls and stuff. I think China will ramp production rapidly across next year and the year after.

56:21And they will get to even tens of millions of chips produced by 2028. But it does take time. Yeah, I mean, let's not forget that they're the first to have a national plan around all this stuff. You know, when they get going, they get going in real scale, like, unlike we've ever seen in human history. Okay, lots to discuss here. Congratulations on ClusterMax and all the stuff that you guys are doing. Actually, I think we managed to hit on, like, many of the major topics of, like, the year so far, which is great. Got to catch up on, like, the hardware lab that you guys are building in Oregon.

56:53I'd actually love to see it. We should do, like, some kind of trip. Yes, fly out, fly out. We will give you a tour. Yeah, give me a heads up whenever you are going, like, we'll do a tour. Dylan will give you a ride in his orange Jeep. I want to go to Abilene. Like, let's just do all these things, you know? I mean, obviously, you guys have been there as well. Anyway, you guys got to go. It's great to catch up. Congrats on everything. And also, like, I think there's just a lot of more work to do. Jordan gave us a sneak peek at the kind of things you're doing.

57:24EndpointX sounds interesting. And, like, the market is really segmenting, right? Like, to me, that's what you guys are identified early and are raising the standard for everyone. So I think for, you know, on behalf of everybody, like, thank you for doing this. Thank you, man. Thank you. Yeah, good to talk to you guys. Bye.