The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

0:00okay we're here in the studio with uh philip uh old friend from from uh inference engineering the book as well as base 10 and uh everything that you've done you and i have done before as well as ali welcome pleasure to meet waterloo intern waterloo intern always when did you get waterloo intern as a as a handle oh i think the rebranding happened like mid-march when i saw it was open i was like i have to take it for grabs the problem is that ali is really good at his job it's not going to be an intern much longer so we have to figure out you know who's going to get

0:33the handle pass the torch over oh okay it can be like you just pass it to another waterloo it's another waterloo intern yeah yeah enter yeah and i know you gotta get an intern from yeah but it could come from base 10 so it's like whoever base 10 gets from waterloo right it has the title they have to pass you exactly halfway through you should you either get it or you're out you should also do like a big graduation ceremony where you change the handle just say i mean you guys are good at ceremonies clearly you know we had a nice launch of the book very successful but before we get into

1:07all that i want to start off with a fun question for you okay you're expert inference engineer what happens when i send a long query say 200 000 tokens into base 10's inference what's the process of query through gpu model routing balancing all that what is all the stuff that we don't think about with a long query specifically the first thing that i'm going to ask is have you sent me this query before or at least part of it and i really hope you have because it's going to be a lot easier for me and a lot cheaper for you so the first thing that we're going to look at is some kind of cash

1:41away routing where we're going to see we probably have a number of instances a number of replicas up serving whatever model you're hitting we want to send this one to something with number one available prefill workers and number two ideally some cached input already there so that we can skip prefill on at least part of these 200 000 tokens if you're doing 200 000 tokens it's probably coding or a multi-tone agent or something where you would expect to have that cached if you don't

2:12we're going to have to send it to a prefill worker we've at least on certain models disaggregated prefill and decode um so you're going to have one set of gpus that's solely going to process the input create that kv cache and get you your first token and then that's going to be passed over to a separate set of gpus which is going to run decode we're going to iteratively make those tokens um we're probably going to have some kind of speculator model in front of that i'm going to assume that you're doing coding and because of that our speculator model which assumes you're doing coding is going to have a

2:45high uh draft token acceptance rate if i'm wrong and you're asking me to summarize every harry potter book uh it's going to be slower and then we stream that output to you and uh account for it um charge you you know some number of couple of pennies and say uh hey would you like to send another one except base 10 doesn't charge by pennies well yeah we we charge we i'm assuming that we're talking about the public model apis if you are you know setting up a dedicated deployment then yeah it's not pennies

3:17yeah i mean one of the key differentiators when i was talking with uh base 10 initially was that uh actually people who want very very high volume just need to rent by the box because then it's up to you to figure out how to saturate the box and more often than not it's like way cheaper if you're pushing like millions of tokens per hour if you just pay per hour instead of paper token yeah they do i think that we've increasingly seen a lot of demand for the sort of paper token apis just because everyone wants to try open models and then once they find a use case that's really sticky uh then they they

3:50move over to dedicate it their best practice on when it's time to swap over a couple reasons yeah reliability that's a big one right like they have a very specific use case they want you to train something specifically for them like they want their own spec deck for instance for their own traffic spec deck is speculative decoding speculative decoding yeah um like just sorry like the way like basically if you have a huge model right and so the model is going to be generating one token at a time every single turn every single forward pass so we attach like this little like kind of parasite like this this layer that goes on top of the model and this model just has to predict it does three

4:23very fast ultra-aggressive forward passes and it will predict like you know three certain tokens and then you do one forward stage over the entire original model in order to see if those predictions were correct or not and then you accept them or you reject them now this draft model is traffic specific so if you like like um you know philip said if you're summarizing harry potter books i can train exclusively that draft model on harry potter books and i can guarantee you that i'm going to accept the three tokens every single time and so with that case i increase your decode speed i wouldn't be able to provide this too if you want to share an endpoint because i have no idea if you're

4:54doing harry potter if you're doing coding if you're doing english like we don't know um also there's a thing in the book that mentioned that they really cared about a specific threshold chapter four i think do you remember that yeah the the things that you can do is you you can set you know a specific like batch sizing a specific like parallelism strategy if you're trying to optimize for like throughput versus latency um you can you know maybe maybe a nvfp4 quant doesn't pass you benchmarks and you want to run a model at higher precision you could do that um there's just a bunch of reasons why you

5:28might want to have your own endpoint and and the the biggest one of course just being like you don't have to deal with someone else doing a hundred million of tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users yeah i think one thing that is that is a classic journey you know like it's basically vivo is asking the what happens when you type google into the browser um tool calling is that just you know you're generating json or is there more complication beyond that certain customers that we have they have their own post trained models and so

6:02they they demand uh tool calling that's not just like you know parse a file or you know go find the weather it's it's something that's very specific and you have to do post training on this and if the post training on the model is not good or if the quantization after the post training to get the inference to be fast the model will struggle reading the json file and reading the tool calling but it doesn't it doesn't require its own like sandbox it's not like it's going to use that tool calling to like like escape a sandbox or like it doesn't have to be contained it can just be a normal dedicated deployment the challenge with tool calling more and more seems to be that

6:34the companies want certain tool calling which which is a very sensitive thing to train and because you're dealing with all of the json outputs if it doesn't like close the end of the request in a very certain manner you end up with a model that did the tool calling and like the thinking and so as a result of that it didn't see the result and just hallucinated the result as it decoded that seems to be the most challenging thing with tool calling not really a sandbox as for one yeah that's a challenge on the training side and then on the inference side there's work that you can do to scope the possible output so we published this actually at this point close to

7:10two years ago um the the solution to this problem which is you basically make a state machine and you use that to constrain the output to a specific format so this is the structured output problem if you remember back in yeah the specific grammars yeah gml had this thing yeah so it's it's like the the old school like make sure this is only json return only json or my grandma's gonna die type bnf grammar at some point the opening i had released a thing that was like yeah if you want to constrain

7:44your output write bnf grammar uh beckes nor in our inference system it's just a yeah specified output format um and and you get the the guarantee that your output's going to be structured along that format um and so applying that to to tool calls can like help cut down on obviously you can still call the long tool or call no tool it doesn't solve the the certainty problem but it at least solves the output structuring problem within tool calls and mcp is just another form of tool right yeah

8:14exactly there's no special thing there the thing i'm always like explaining to people is the llm is actually not capable of doing anything it's only capable of making suggestions of what to do and then if those suggestions are formatted in a certain way and applied to a system that knows what to do with them then an action occurs yeah part of the fun stuff is you know this is solved outside of tool calling too like in an agent loop if the output is not correct or you're right like reasoning uh tool calling was done in the reasoning so you'll just be like oh i don't know what to do let me just

8:45try again and you know it might get there after a few tries um and on your point of training sometimes this is harder in smaller models so you don't have the same exact quality output when you just swap from a big model right yeah i will say that um before we i think we need to go back to inference engineering proper but um i had expected that something would replace json because it's hard to stream json because json must be complete and you must have open and close brackets and everything so it's hard to parse something or validate something while it's being streamed uh so people

9:19invented all sorts of things that are like i forget the the name of some of these alternatives but it's basically something like toml something like yaml um but json seems to be dominant still the json outputs aren't that long right like i guess you could have a long because tool calls also contain the arguments in them and perhaps for a certain tool you might pass like a very long argument but my impression of the sort of median tool call is that it's a relatively small number of tokens right so i would expect that speculators are generally fairly good at something as formatted

9:53as json um and so you would have like a pretty fast decode step there and and that the streaming wouldn't be as valuable but maybe i'm wrong about that you're also bounded by the software that the model is going to integrate with if the software is built with json for the tool calls or the company that you're you know if your customer says that this is how our software works and our tools are interfaced with json you can ask them to like you know change their their software and say like yeah this is going to be better for the model but like with the web training shouldn't be that much of a difference also more profitable if it outputs more tokens probably depends on your business model

10:26you know it really depends uh but i will say that you know as a writer would like experience a lot with ai generated output i do try to move from text to json text which is very long json right like like there's paragraphs in every field because i'm trying to structure it right i want you to first make factual statements then make opinions then make bullet point summaries have dates have entity references have your sources for references all these things anyway so so these are things that like i think people who really really experiment with structure output have to really care about but um let's

11:01let's sort of recurse up the stack a little bit before we started recording you actually mentioned something which is really cool which is that there's a lot of engineering inference engineering that goes on when a new model provider releases a new model right so let's call it glm 5.2 kimi k3 i had previously assumed especially if it's like well glm 5 to 5.1 to 5.2 like you know that you've supported them before is it that much work it's a lot of work yeah okay so like you know a lot of people all you guys right whenever a new model launch like people rush to say like oh hugging

11:34face supports this fireworks supports this base 10 supports this and i'm like yeah of course you support it but what goes into that i think it's more than just supported too right uh it benefits the consumer a lot like i think it was with uh kimi k2.5 or glm 5.2 the latest there was sort of an inference war right x provider is at 90 tokens a second the next day we're at 150 i kind of kicked that off with the glm 5.2 i wrote a twitter article about it got like half a million views um based on being number one yeah yeah which uh got everyone really excited about hey how can we you know

12:10bench max a little bit further and um there's a difference between support the model as in like i can make a token out of this model and support a model as in i have a production ready api from this model getting to the point of i can make a token out of this model is not that hard because generally the um open source inference engines your vlms sglangs of the world oftentimes even receive weights ahead of time maintainers do or the people making the model merge prs to ensure support

12:46so you generally can you know just kind of get it working on the standard open source stack without too much pain in most cases the challenge is you know every inference company is going to have our own proprietary stack um you know some open source components some some in-house stuff and for any arbitrary model there's going to be some new stuff sometimes you get lucky like k uh two five to two six was like pretty pretty similar yeah it was pure continued post training

13:18if i remember correctly even in those cases there's still stuff you have to do you have to redo the quantization work you're taking the model from generally these models are not released in nvfp4 um and we want them to be in nvfp4 for maximum blackwell compatibility so we have to perform that quantization um and you know calibrate the quantization to make sure that we're not causing any kind of regression in the model's intelligence and then we also have to train the speculator as we've talked about generally we have obviously we have zdr zero data retention on our

13:51model apis so we don't know exactly the traffic that people are sending us but we know what's popular we know that coding use cases are popular uh we know that agents agentic use cases are popular so we can get public data sets that are representative of that kind of traffic and train general speculators now with speculators today you need to train the speculator using the base model itself because you're getting hidden states out of the model from running inference on these specific prompts and that is the training data you use to create the speculator so there's that process which you

14:25need the the real model weights for uh and then there's of course just the process of you know standing up all the infrastructure behind it loading all this stuff in testing it um and then when there's a new model with a newer architecture i think that like obviously the deep seek models tend to be the most challenging as they have like the most novel architectural stuff going on uh model over model but every new model has something i mean kimmy k2 had oh sorry glm 5.2 had sparse yeah the the dsa which is brought from deep seek yeah yeah and and you know we can copy

15:00pasting you know i don't know how this works you know so like we had to like build support for that into our runtime um and and you're right like it actually is really interesting the way that all of these open source labs borrow from each other for example like glm 5.2 doesn't have vision so something that uh hailey a guy on our team if we could take a look at this um he like kind of grafted the kimmy um vision encoder on to glm 5.2 or retraining the projector exactly so if you think about like the encoder there's the encoder which is the part that looks at the image and turns it

15:34into latent information and then there's the projector which kind of like space it's okay and then there's the projector that that maps it on to uh the the model itself and then there's the model weights you don't want to mess with the model weights because you want a chance of making the model dumber at something else for the purpose of giving it vision so instead harry started with just a projector which is only a handful of millions of parameters yeah and can you show the the training one like the way it groks is yeah maybe you know maybe ali you should you should

16:08take it from here you've got a better understanding of this than i do yeah you can see like he the way he trained this is really really cool at the beginning he was training it using just like here's a picture of a mountain can you describe what's in this mountain and that caused it just like the first the first you know learning walls like here you can see this all we're trying to teach it is to translate the encoded like it's it's already taken the encoder from kimike it's taken the the image it's frozen frozen frozen frozen so the understanding the brain is frozen and the eyes are frozen it's just we're trying to try and interconnect between the eye and the brain right so the

16:39projector and so you take the tokens and then he's like oh can you describe what's in this image and he's like oh it's it's a mountain or it's a person or it's a human whatever the case is but that didn't cause complete understanding so he changed it such that every image was associated with a data set of questions like does this image have a white male does this image have birds in the top corner does this image have a scientist in it all that stuff and it would have to answer questions correctly and using not just training on describing an image but being able to answer question answer question answer question like question answer over time like you can see the grokking which is like genuinely insane that retrofitting vision into a large lm can learn to that

17:15extent and even for images that it doesn't perform well on for instance if you ask it a picture of like stephen hawking who is this maybe it doesn't get it but it will say something like this is albert einstein like it still understands close enough is a scientist who is a man who has you know done significant achievements all that stuff so that's like really really cool yeah so uh we've covered hao tian before who was the author of the lava paper that did this uh a while ago and i think that that's very foundational work for anyone who hasn't done vision work before same with the clip and meta clip where you go from just captioning to building out questions off the image and how much

17:49better you can get performance right right yeah but what's what's so exciting about this is if you look at a model like this now obviously this is a little bit more of a research project it's not you know it got to 56 on mmlu pro i think so not not quite frontier but if you're running this model you haven't suffered any loss on your glm 5.2 quality if you don't have an image it'll just behave exactly the way it used to and ultimately in the inference code you literally do not include the other part right yeah i mean you would just skip the encoder if you don't have an image input

18:22okay um just confirm it a lot on the overall inference side like you're not adding much you're adding a very small vision encoder these are typically like less than a billion right yeah it's i mean there's a little bit less standardization among vision encoders um so the sort of support matrix can be a little bit uh sparser but overall um yeah it's it's a pretty it's a pretty minor component of the overall system and and ultimately what you get out of the system is all of a sudden you have kimmy vision glm weights and deep seek attention all in one model and that's i think a lot of the power

18:58and beauty of open source is that you can take all of these different components and combine them together into a system that's better than anyone can be individually people used to say that you would also do franken merges where you would take like layers from each model no does anyone do that anymore well to your point previously when you were mentioning like um the work that goes into supporting a model when it first comes out like glm52 or minimax m3 or whatever the case is sometimes you do have to like you do have to switch out some things like for instance the minimax m3

19:28head uses full attention and with full attention you end up with this like insane bottleneck inspect it because you're doing auto-aggressive token generation for three tokens and you're doing this like like o n squared over all of the tokens that are in your sequence um your kv cache is like very large because it's not sparse it's not top k so we find it better to like okay we're going to replace this you know we're going to replace this layer with a layer from another model that's using like gqa for instance and then just for the right training you can get it to have the same acceptance right so it is it is very possible to to retrofit layers from other models and very much

20:04needed actually if a layer is like inefficient the training just becomes the challenge like how do you ensure that you train it properly which again to earlier points is like the mesh between training and inference as in like you need very good training in order to do fast inference that's like i feel like more and more becoming true yeah anything else on the support side when you when you say like get it to fully production ready yeah i think that there's also a question of just you know we can test a model to a pretty extensive degree but we're trying to get it out quickly and then you see a bunch of

20:40other people test it and you get interesting results there was a an issue with um glm briefly where we had some like mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures like once you expose an endpoint to to the real world there's going to be you know so many more varieties of of things given to it that that you're able to you know discover and patch things so it's not just a you know day zero process it's then like for the first week for the first

21:13month if a model remains popular like how do you both fix bugs and then continue to push the envelope on performance what do you mean you don't want your model outputting ssssss is there loop detection on that stuff by the way it still happens like quite a lot which is surprising we have like in our in our endpoint like if a model was output the same exact token like four plus times we just call the generation we say like oh sorry this should like try again or like we will re-process the request because we know then like if like if it's four times the same token it's probably collapsed yeah is

21:46there a way to opt out in case i really actually want that you're actually i think i think there's a way that we have to handle it i'm not exactly certain i feel like in certain models like when they output something like you can imagine a like a table for instance and so they want they want to draw like 12 dashes and 12 dashes yeah i think there's a way for that to happen i think we only do it on certain tokens like we exclude certain special characters yeah so we only do it on like certain like like s is the most common almost glm5.2 oh um and i think it was dsv4 as well like you'd just have like looping issues where like yeah is there special something special about s no just

22:21seems to be the one token yeah uh and it's and it's only temperature zero or even at like 0.9 or whatever it will still it will still collapse that's weird right it's an inference it's an inference problem to be honest like a software problem like oftentimes um the image you're on like video will release an image for instance and if we will upstream the changes from their latest trtlm image into our stack we'll find that it fixes it or oftentimes this will only happen in an inference engine that you're using like sglang but if you were to switch to vlm that isn't the case so it seems to be like an extremely like non-deterministic kind of software issue and not

22:54really a model issue it's not like a weights problem like i'll be able to say like oh it's a problem with the quantity we did ptq wrong right but that isn't that doesn't make sense because the same exact weights used with a different inference engine does not repeat the the problem and sometimes it's um the kernels that are being used in the back end have like these very subtle sometimes raised conditions where if you were to use this model hosted on one cluster you will never get this problem oh my god if you host it on a different cluster you will and the reason is the kv cache transfer from a node to node in that one cluster is using a slower

23:26interconnect than the node to node in another cluster so that exposes the race whereas in another cluster it doesn't so then you end up just like okay this model is not going to be hosted on this cluster we're going to host it on you know another cluster because that cluster exposed that problem but then it ends up with like okay is it the software is it the model weights or is it the hardware there is a thing about this with temperature zero still not being deterministic right mostly because of hardware even at temperature zero same model you won't always get the same output even but i'm surprised by the race condition one because uh i thought pytorch was a graph that like

23:57guarantees that you at least you know execute things in the right order well they're not true like like i guess i guess i'm not i'm not saying that this is like i guess well you have things like pdl optimizations where like you can you can start a kernel before the end of the previous kernel and that's like you want to do that exactly exactly exactly but it's but you don't do it cleanly like you overlap a little bit of the execution no i guess it is very possible that the kernel itself like that one block that is supposed to be running in this instance of time that kernel itself has a race condition for instance like a missing barrier like often if you're designing a kernel and you want it to be very fast if you don't test it extensively you'll you'll have

24:30certain threads access data points from registers before they've been written to by other threads yeah because like your barrier is wrong or your synchronization is wrong but yeah like the testing itself is is very very difficult in those like and there's no like borrow checker for you know like rust like that if you if you're trying to have like memory safety if it sounds like a comparable problem well i guess but you're working in code right and video gpu it's like you just need a higher level language like modular maybe that's what modular is supposed to do i don't know how do you see keeping quality of the model so you talked about all these steps of okay

25:04you gotta do quantization train your own speculative decoder run on different hardware um looking at other model providers okay you kicked off a inference speed race on the consumer end um what goes into keeping quality the same across them right sure you can run benchmarks but like how do you determine how much quantization are the standards what actually goes into there's a few things on quality most inference optimizations are lossless kv caching for example you are just recomputing or preventing

25:37recomputing the same values speculation of course if a draft token is wrong it gets rejected the main lossy optimization is quantization um and that really comes down to number one data format uh number two uh which parts of the model you choose to quantize which layers and number three like doing a lot of calibration on the quantized weights uh to ensure that you're sort of preserving all the outliers there's other sort of tricks that you can do though a big one is long context because one

26:11thing you asked right at the beginning is oh what's going to happen if i send a 200 000 token request in so obviously with a long input sequence you need to you know store a lot more information you need to process a lot more tokens and so even if a model has a context of a certain length you might as an inference writer choose to build an api with a shorter context length and of course a full length one as well because if someone doesn't need the full million token context for example you can get

26:47them better performance i don't know if that's exactly like quality of the model the way that i think about quality is to what degree are we faithfully serving the original model if you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform i think of quality as how close are we getting to that you know 100 fidelity of the model you can also of course think about quality from the training side and how do you push yourself past 100

27:17but when i think about purely inference optimizations it's getting faster while staying as close to that 100 fidelity mark as as possible and certainly our standard internally is that like you should not be able to tell the difference between our api and a you know sort of official api i think kimmy in particular does a good job of vendor benchmarking here yes they released an actual exactly because they accused uh some people amazon there was some provider that was not doing very well on kimmy's yeah so

27:51it would reflect it would reflect we this was a long time ago right no like like three four five months ago this also happened with i don't remember which model but they they pulled out quite a few and then they started a whole chart about this it might have been kimmy vendor verifier yeah yeah because you because you'd be pissed right like if you're yeah if like um like if i'm a consumer and i'm using like amazon's endpoint for instance and i i'm used kimmy i'm like oh my god like this is bad i'm not going to say amazon quantized the model in a bad way i'm going to say oh kimmy sucks right so it seems like that yeah they care they care justifiably yeah uh this is probably a stupid

28:24question but just checking has anything improved from being quantization like is quantization always strictly words uh well technically it's a lossy quantization is a lossy it's a lossy implementation speed improves it's been improves it obviously like i always look for inverse scaling laws this is something i learned from gnome brown where like things that normally act in one direction sometimes well technically when you run a benchmark because these models are non-deterministic sometimes you're you know mvfp4 quant is like uh you know two basis points higher than you're

28:56exactly yeah it's it's within that's why i always say within margin of error and i actually stopped saying that because everyone assumes that what i mean is well within some margin of error we're barely inside of that to the worst so we're saying but yeah sometimes it's just like uh gives you a higher output score but like like ali said that's noise um to my knowledge you're not necessarily making the results better you're just trying to again like keep your fidelity as close to 100 percent to the original model there is to your point research that we did on mp i don't know if you are able to pull

29:31um a tweet we did one of our research interns joshua um i think it's a tweet on on how we have 20 percent better quantized jlm52 than nvidia um essentially what we found throughout like this two-month research is okay quantization is is a lossy it's it's you're compressing the data from you know occupying 16 bits to occupying you know four bits for instance and so you're obviously losing some some information and you're trying to minimize that and so when i say that i'm going to quantize the model my my job becomes how do i find the layers that i can quantize and how to find the layers

30:05to not for instance with image models i don't quantize modulation layers and i don't quantize out projections because those two are like out projection is what you see as the user modulation is what the model sees or understands right exactly um and so i guess to his his paper do you have the i guess doesn't have the yeah it's a long paper i don't know if i can find if there's a part to search or it's probably in the thread it's probably in the thread yeah but basically the the long the short is it is very possible that quantizing more of the model makes the results like if i have a

30:37model that i quantize layers one five and ten and another model where i only quantize layers one and two it is possible that the model in which i quantized more information is going to perform better because the quantization errors have canceled out and so what joshua showed in his mathematical proof where he had like a verifier in is that you can predict which layers are going to have quantization errors that will cancel out with each other and you choose to quantize those layers and so the result of doing this mathematical quantization is you end up with a model that's 20 more quantized than another provider so you get 20 more throughput of it because there's more

31:11layers that running in vfe4 and your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out like one layer scoot to the right one layer scoot to the left one layer scoot to the right your final logist distribution is more similar to the original distribution of the model so you have better fidelity and so the way we proved this was with kl diversions so instead of just scoring on the benchmarks we scored the kl diversions between the logist distribution of the quantized model and the logist distribution of the original full precision model and we showed that with this technique we get if your if your you know if your probability distribution on the logist switch token it wants to select is more of the same as

31:42the original model you're probably going to end up staying true to the original model so yeah so it seems like like previously before this it seemed like the industry was well the more you quantize the worse it's going to be because the more loss you introduce that's not exactly not necessarily true so yeah doesn't improve it but can cancel out i think it might be this but reminds me a good bit about pruning actually where you can prune off certain layers but very interesting didn't know this was a whole paper you guys put out it's a fun fact it was originally 72 pages this paper and then we decided we can't uh we couldn't release it so it's our 45 still still 39 pages very very substantive

32:18we talked about evals and and all these things and uh like what's possible in terms of speed up i guess like it's like probably like the number one thing that people do want to care about and it's something that you wrote about in your post um like official api is 70 tokens per second and you push it up to 90 is that like a normal thing so what's cool about working in inference the reason that i think inference is going to be a useful place to do engineering for a long time is that if you look at highly optimized domains like say finance if you're in finance you measure how much better you

32:54got in basis points it's like oh i got five basis points better like one twentieth of one percent better that's huge news because everything is is so optimized when we publish optimizations it's it's twenty percent it's a hundred percent it's two hundred percent so there's still probably like a lot further to go honestly like you'll you'll know that inference is pretty much solved when researchers start publishing about how they got one percent faster at at something which by the way uh because i am from the finance background in the 70s that was the margin at the time when you did quantitative

33:26finance research you would find and like the twenty percent tens of percent yes yeah and now it's tiny for those for those people interested look up andrew lowe's paper uh he had a really interesting illustration of quant uh stat arb uh distribution uh narrowing down from like those kinds of twenty percent differences in the 70s uh down to nothing today um which is very cool exactly and we're at the beginning of of the same type of of of thing now benchmarking is hard um i think any anyone will

33:56tell you that and benchmarking provider speeds is hard because there's so many variables that go into it what hardware are you using how much load do you have on the system what's the exact nature of the prompts and input and output sequence links all that kind of stuff but overall when you start stacking these improvements you're looking at multiples um you can look at it the most common form of course is tps tokens per second which is bad naming by us in the industry because there's

34:27actually two tokens per second there's tokens per second the throughput number and the latency number um like total tokens per second out of the out of the gpu as a throughput number most people only care about tokens per second as the latency number which we should call itl into token latency but we don't um anyway so you can imagine a sort of standard api without many optimizations for one trillion parameter model operating somewhere in the 30 to 50 tokens per second range for a for a

35:01reasonable traffic profile um and we generally see the goal of you know pushing to 10x that um but you know not necessarily day zero but by stacking enough optimizations um if you have say like four optimizations each of which doubles performance or sorry three optimizations each of which doubles performance uh then you you stack that up that's an 8x gain that's kind of the the order of magnitude that we're working with in this in this space we're trying to make things substantially

35:36faster not just go from like 70 to 90 are you saying you you have done that so let's say you have as a reasonable baseline 30 or 40 tokens per second uh you can you can achieve 10x that so like on glm 5.2 um if you're running it unquantized uh perhaps on hoppers even um and you're just using an off-the-shelf inference engine with no particular optimizations no no speculator um nothing nothing extra around like kv routing no disaggregation uh you you're probably yeah looking at that like 30 to 40 you think that's

36:10like a reasonable baseline right right to get to something like 10x there's a lot of trade-offs that you're making if we're running at sort of more like a 300 400 tokens per second range um obviously you are using the best hardware possible uh you have a optimized speculator you have done all of your quantization work you are seeing a pretty high cash hit rate you are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput but it is

36:48possible um so these spreads that you see if you like go on artificial analysis or you go on open router and you look at you know the the worst provider to the best provider oftentimes can can hit that kind of range uh 10x is of course very aggressive it's oftentimes maybe more of a four to six times improvement but that's the the kind of performance that makes us really excited is when we can get these huge gains not just go from 70 to 90 tokens it's also like hardware dependent like

37:20if you obviously have a thing where you're serving it on just like a node of h100s and then you throw like you know you short the model across like four nodes of b200s like like you can definitely increase the speed with just throwing more hardware at it like normalizing for the same exact hardware and the same number of gpus yeah then you're looking at like a two to four x improvement depending on the influence optimizations so yeah it's some of it's you know what's the car and some of it's who's the driver if you break down the two to four x say say the example is run glm 5.2 on b200s single node right

37:54what's like the cost trade-off for effort to get like the last bit of juice out versus what should people just think of right spectac quantization yeah spectac quantization that's that's like 95% and how far does that get you and how how easy is that for the average person to do so say right now i want to throw the weights of glm 5.2 on a node of b200s how easy is it to find speculative decoder decoder model or already quantized model how much work goes into it if you're doing it up front it's

38:25quite a lot of work if you're doing it today there's going to be people who have published things that you can just you can just grab some nvfp4 weights you can grab a speculator uh yeah if we're thinking about like what are the two x's we're stacking going from uh bf16 to nvfp4 it's not quite a two x right it's like i think it's about like 30 30 to 40 percent um from 16 to 8 and then another 30 to 40 percent multiplied from um eight to four so that doesn't quite get you a two x but like roughly a

38:56two x speculator roughly a two x disag on top of that if you're able to get enough hardware and put enough traffic through it another roughly a two x and then you add in some you know double digit percent increase from having just a a better run time with you know the the latest kernels and stuff behind it um and that that's kind of how it stacks up um so building each of those like building the um quantized weights is for someone who really knows what they're doing hours to days of work

39:29um building the speculator again like hours to days of work and the um disag setup hours to days well okay once you have it getting disag working for the first time i'm saying of course is very difficult but the the marginal implementation is if you're just grabbing like if you are a person like just a normal consumer who has access to like a node of b200 and you're wondering how can i just host it myself you don't need to quantize the model yourself there's always going to be like an open source quantized checkpoint nvidia is going to push one out if no one else does you usually the

40:03providers will have their own spec tech that they've trained as well you don't need to train your own spec tech you can just use that as well yeah like kimmy uh glm 5.2 has its own mtp right right multi-token prediction i can do it for you in case i get it wrong you know uh yeah actually you should correct if we're wrong but their multi-token prediction can be used for self-speculative decoding i'm actually not sure okay i'm semi-confident but someone can check um but you know it's useful to paint the story of okay not just the average person but say a company wants to switch from

40:37serverless inference to i want to throw this up on you know i want to rent some gpus throw it up these are the steps you take to do significantly faster than just put it behind vlo right i was waiting for a mention on dynamo i feel like uh that's supposed to be the baseline that you measure against i would think of dynamo as less of a sort of out-of-box system and more of a toolkit for building with so when we talk about doing kv aware routing when we talk about doing kv out

41:07offloading when we talk about doing um pd disaggregation dynamo fundamentally is by the way dynamo is an open source library from nvidia we've done that part with okay cool so so then your your listeners know then that it supports all the different inference frameworks um and it actually is kind of multi-hardware which is interesting but it's just a router it's not like an optimizer there yeah all it does like what dynamo is good at it is a library for moving information around your cluster around your hardware

41:39so if you have you know kv cache on one place and you need it to be somewhere else dynamo coordinates nixel for you to to move that around that doesn't mean that like out of the box you just say you know pip install dynamo and then you get like a massive performance speed up um it's it's more of a developer toolkit yeah i would have said it would it comes with a set of defaults that you can then swap out it does um if the industry at large i think was like rolling out all of these deployments um standard

42:14then i think it would be like a credible baseline but um we've got it we've got a benchmark against like what we're seeing in the wild i did want to talk a little bit more about pd disag because that's probably like number three after quantize and speculative decoding in your book though i was just going to pull out the book yeah like section 522 on medusa 523 on eagle yeah 524 on it's it's 5255 um would be um would be disaggregation yeah well no i just wanted to dwell a little bit on the

42:44other like so what do you choose to include what do you choose to not to include because there was all these other techniques i guess yeah are these still relevant because i think they came out like a year and a half ago maybe medusa is quite old yeah medusa's old but is it in the book as a good here's the baseline like you should know this like i read the paper my god makes so much sense yeah so with the book i had a couple goals uh one was to give people just a working vocabulary for the

43:14space as a whole and the other was to give them some intuition about how each of these techniques works as i mentioned in my ai engineer talk which is kind of the first public addendum to this the speculation space has moved much faster than everything else so yeah even at the time that i wrote the book medusa i very much included as a way for people to understand how the how how the space evolved rather than what the most modern technique is and now of course there's d flash d spark um

43:46there's there's newer techniques even than than eagle although eagle is still very commonly used spec spec that yes speculative speculative decoding what can you it's it's based a paper by tree down and it's like it's basically doing speculative decoding uh-huh for the speculative oh my god it's it's literally just another it's like yeah it's most important to explain it and it seems like he got non-trivial speed ups there but it seems that the complexity with training it's almost like like in our mind at least it's almost as complex as training gans like it's like a very delicate balance

44:16and oftentimes you it's just additional but yeah it's literally speculative decoding on the speculative decoding yeah we saw this paper it's interesting right i wouldn't even expect it to be very particular to train now right the naive part of me is like okay trains like a decoder it makes sense like the whole idea of speculative decoding as you you it's like it's like almost like the iphone auto predict version but for a normal model right like you're just you're just you know generating three tokens and you're like okay do pre-fill on them and so you save those three turns for your original model now your speculative decoder is doing three turns

44:48of autoregression so why not just have an even smaller model i guess the the other question there is what are the size of speculators so say for glm right it's like a billion a billion parameters like for minimax it's it's yeah yeah it's like one layer it's like one sixteenth of the original model usually yeah actually i think we should do a paper when we get back to the office speculative speculative speculative coding no it's it does seem like like how when do you stop but then it also seems like kind of um like if you're able to train spec spec decode for instance right like if you're

45:23able to have a small model that is accurately predicts what the intermediate speculator is going to predict that is able to predict what the original target model is going to predict then why not just use that smallest model directly right yeah this is adjacent to the routing problem right yeah right the thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on there is a orchestration and resource competition problem inherent in that and that is one of the sort of

45:54constraints on speculation in general is that draft tokens cost resources to create and cost software complexity to manage and so if you have sort of like infinitely recursive speculators you're adding quite a bit of that complexity on the actual implementation within the inference engine as well not just in the training process i was gonna say i would wonder if you could do similar like distillation and pruning of you know it's the same thing it's just a model can we not just distill a lot of the weights quantize

46:26the speculator but out of my domain um i guess the the question that also comes up is this is all for big server workloads right how much of this applies to say i have this macbook i want to run gemma really efficiently um similar problems not the same pretty different i talked to sarah um about this on on his podcast a couple weeks ago the difference between inference engineering for the data center and for production workloads versus inference engineering for local ai is that we start with fundamentally like

47:01different constraints and different goals with local ai it's how do i fit this model onto my hardware and then make it less dumb and with data center inference it's how do i load this model and then make it less slow and obviously you know we care about less dumb and they care about less slow but the local ai inference engineering ecosystem i think actually has a lot for us to learn from in the data center space they are experts in various forms of quantization including dynamic quantization

47:34that we just kind of don't touch in the pruning in the distillation in the uh you know layer removal there's removal matters less yeah there's pruning really yeah but but but which is surprising right but just to fit something on the laptop right so yeah i mean it's it's a it's an interesting it's an interesting space not not necessarily that like their techniques make sense for us to do in the data center uh because obviously we have different resources and different goals but more that the

48:05the process as well as the openness of that field is is something to you know admire yeah like to your point like um certain optimizations that wouldn't like for instance turbo quantum sure you've heard like it made such huge hype on next and we we did like a whole deep dive on twitter and i said what is it how does it work why is it good or not and it took off and it was implemented on local devices because your your memory bandwidth is so slow on like a macbook for instance but try putting the same thing on like an nvidia gpu on a b200 turbo quantum would not be like it would not be used like

48:40nvidia like nvidia made it clear that this is not a good optimization and we've seen it firsthand where the overhead of doing dequantization quantization of you know in the kernel itself of turbo quantum kernel eats into and is actually much much slower than the time that you save from doing the bandwidth because on the b200 you have like 3.5 terabytes per second you don't need to you know decrease the the storage that much you don't need to do you know fb4 kb cache you don't need to use a requant there's there's there's better optimizations to be made but on on edge devices it's extremely important it's extremely useful so you know it seems to be like different

49:11optimizations there but then they're all uniquely combined with like oh you want to quantize the model you want the speculatives decoding like certain common prefixes principles yeah exactly exactly they also do a lot of work on um model parallelism especially over uh you know heterogeneity topology where you have you know some sparks and they're wired together with you know ethernet dgx sparks yeah this is the exo labs yeah you have you know a number of uh mac minis stacked up there's you know the inter they one thing that that i think we both have to deal with although

49:47they have to deal with a lot more is is the interconnect between machines um which is why like you know one thing that we do a lot is work with tensor parallelism and that's where uh you are using all of the you know all eight gpus um and and sharding the model across it uh tensor parallelism is not a good fit for local ai because it assumes a very high bandwidth interconnects like envy link uh was you know they might be forced to do something like pipeline parallelism which we're never going to do unless we're doing some kind of multi-node inference but since you mentioned it i actually wasn't sure if we're

50:20going to cover it but let's briefly explain tensor parallelism and expert parallelism uh since you have very nice images i just want to show off your images yeah uh shout out shout out to luke from base tense design team for making these these beautiful images oh that's a that's actually before we get into this just one other difference is we we talk a lot about the active parameters of a mixture of experts model and for local inference folks that matters a lot because if you have a batch size of one you're only activating that many parameters when we do yes i was gonna be

50:52in the diffusion conversation oh yeah yeah we uh when we go through like a uh moe model um and we host it um for an api we assume that all parameters are going to be active because you're batching your batch you're gonna you're gonna hit everything um cool so broadly tensor parallelism you can do with any model expert parallelism you can only do with moe models um effectively all models today are moe models um that are you know at least all models large enough that you would care to parallelize

51:24them across multiple gpus so that that's that that nuance is is less important now um with expert parallelism the idea is you put the entire expert on a gpu generally you have more experts than gpus so you might put like n experts per gpu like eight experts per gpu or whatever and then you uh you replicate the router where the router is very small across each of the gpus and then by moving the the generation from expert to expert with each expert being inside a gpu they're not competing for resources

51:58you massively increase the throughput that you're capable of doing and the uh gpu to gpu connection is is not as important because there's not as much communication uh tensor parallelism requires that you are able to do this like all gather all reduce so you basically shard the model across the gpus entirely um and then for each step you're combining the results of each of the gpus which is why the interconnect matters a lot and it is generally of course this is a this is a very high level

52:32generalization there's a lot of places where this is not correct but generally uh tp is is helpful for latency and in many cases you will use some combination of these two parallelisms um across the the model rather than just like picking one or the other you want to you want to add some color though like in a model it's not they're not mutually exclusive you do tensor parallelism and you'll do expert parallelism pipeline parallelism less solely it seems to me like we never use ppn yeah the only reason you would have to do pipeline parallelism um which is where you separate like different layers

53:04and you put like half the layers on one hardware and half on another is if you are forced to do multi-node inference um because a model is bigger than you have that like let's say let's say you're doing a deployment on h100s for whatever reason and you're putting a a trillion parameter model on there you have to use multiple nodes of h100 and so you the because the interconnect is so slow between the nodes the only viable way to parallelize there is is pipeline but then you would do expert and tensor

53:35within each node and the limiting factor for h100s is hbm yeah they just don't have enough how much what's the magic numbers that we need to like on a b200 is 180 gigabytes per gpu and then a node of eight you're talking like 180 times eight and the fp4 so each parameter takes half a byte so that's 800 gigabytes on a h100 it's like 140 it's 80 it's 80 yeah yeah i'm old i've been doing this a long time i actually remember h100 specs yeah so tell me about the t4s well let me tell you what it was

54:08like to want to model on a t4 back in the day um well one thing i was surprised to see that more people didn't do uh jamba i don't know if you guys remember jamba from ai21 they would actually specifically pick a hardware and then they they design the arc dimensions for the hardware and then it would obviously saturate the hardware like it makes sense and like somehow all these models don't do that don't they do this for the training side though uh i don't know the what sorry the training for training like deciding which dp which dp yeah yeah yeah they do and with training it's

54:42more like a math like you can run the math and see the flops and maximize it within inference it's more of like an auto tuning i don't know if you're familiar with like gpu current auto tuning but like it's basically like you you define that oh i have two gpus i can do tp1 tp2 ep1 ep2 for instance right and you so that gives you like total like two squared combinations and then you just like you shadow the same traffic like real pro traffic and you just see which configuration gives you the best tpm tps and just use that i don't like the fact that it's you cannot reason about which one's going to give you the best performance or that there isn't one specific configuration that's

55:13always best but it seems like auto tuning is just the way that you find the best one and with kernels and gpu kernels it's much of the same after you you design your kernel you design your configuration how many threads do you launch how many you know how much shared memory do you use you just just auto tune you just sweep the parameter space on the side and this is the best one empirically but yeah but they are they are combined they're not just in like separation there's a few bits of training that are kind of like hardware targeted if you look at for example nvidia nemotron models they run very very well on blackwell that's that's unsurprising um so there's some degree of that but i

55:48think that most open labs are trying to make models that can be run on as wide of hardware as possible rather than targeting just like a single chip i see for usefulness yeah uh okay one more thing while this chart is still up all gather all reduce is expensive one of the things that is a movement in silicon valley is mega kernels just keep fusing kernels i don't know is it that simple well i mean like a fused kernel can't save you like like here with the tensor parallelism you're the half the matrix

56:18is one gpu and the other half is another and if i need the entire matrix in order to do like a non-linear operation in the next step which is for instance like if i'm doing attention i need the softmax or i need to like like exponentiation i need to have the entire row so i need to know what the partial result was from gpu 2 and what the partial result was from gpu 1 in order to be able to do the softmax in the next stage so i like i have to make them communicate with each other even if i had a fused kernel because of the non-linearities within each one also with like mega kernels like honestly i'm very bearish on i'll be honest please please please no it's just like mega kernels it was a

56:53good research direction and it seems like a very like like intuitively theoretically it's nice like oh my like you have a lot of launch overhead from launching one kernel just fuse everything together but yeah but like like the the kernel complexity itself is is it's very difficult to write a very optimized mega kernel it's it's very very difficult to do so and even the like not to name any companies but like even the companies that have worked there's people that have spoken to who work at companies that do fused mega kernels they very very often don't end up running those in production because

57:25the trtl and modular kernels that launch are faster because you can optimize each individual component and you can just have them paralyzed with each other with the rubens i don't know if you guys saw the rubens twitter posts yesterday but they're also um he rubens like no no like the gpu yeah do you have a twitter account for rubens only no no no no i was like what are you talking one of one of the tech leads that nvidia like launched a twitter post like we're pulling the curtain on ruben and here's the here's the specs and and the third tweet showed like not to get

57:57too technical into it and i need to read it much more but the gpu is is is designed in such a way that it basically kills mega kernels you don't need to use mega kernels that much anymore so it seems like that entire research field goes into like won't be continued but yeah can i speculate about ruben for a minute please go um you know i've been through now we will and by the way they are covered in the book yeah but yeah well i mean they're covered in the book in the sense that like i am aware from the blog post yeah it's going to happen in the future and you even had the the name

58:30of the one fineman yeah it's like hey this is this is going to be this is very up to date i'm trying to future proof this thing okay i don't want to publish a new one until like next year or something uh anyway so we were discussing the degree to which i am old um and you know i've now been through three hardware launch cycles i've been through the ampere launch cycle the hopper launch cycle and the um blackwell launch cycle now when i say launch cycle i don't necessarily mean like the uh the actual shipping of the hardware like ampere's were racked up well before i got in this

59:04industry but there was a lot of time between hardware being racked up and hardware being sort of feasible for inference so if you look at like the original vllm and sg like vllm especially like that was written targeting ampere and then had to be updated for hopper updated for blackwell with each of these cycles it becomes faster and more urgent but also substantially more complicated when i look ahead to you know what's going to be new with with rubin i think that like dynamo gives

59:39me a lot of technical hints around like what kinds of work is going to be very valuable obviously we're continuing some trends from blackwell right nvfp4 is big the amount of compute that they have behind nvfp4 tensor cores is is massive um we're going to talk about video i think at some point and and that's the the big barrier there you've got you know much much faster memory bandwidth um but which was the same thing that that made blackwell so good um but the the big thing is more systems thinking you have

1:00:10more emphasis on the cpu to gpu interconnect more emphasis on the interconnect between gpus and when you look at dynamo it's a system entirely designed around how do i move the kv cache to where it needs to be when it needs to get there um so i think that themes around like kv cache offloading kv aware routing and and disaggregation are going to be substantially more important in the rubin era which means that inference engineering becomes not just a like cuda kernel problem but also like a very

1:00:43traditional hardware infrastructure problem which is something you know we've been building toward for for a long time and something that's like very exciting to me because we're going to see sort of multiple domains colliding and the ability to reason from the kernel level like up to the hardware level and back down is is going to be very valuable i will take what phil said one step further actually into that it's i think trending towards becoming exclusively an infrastructure problem where like problems of pd this like training spectac but try it in kernels is not going to be much of a problem

1:01:18because the gpu is moving more towards being an asic where it's you're just you're just trying to orchestrate what happens on the gpu but you're not actually controlling it thread by thread level and you see this with like qtl qtsl like you're you're just working at levels of like tiles of data but you're no longer working at controlling what each thread does on the gpu that's being taken care of for you um so i guess do you agree that a gpu and future gpus are trending more and more towards becoming asics that just need to be launched and then they do the data operation based on your

1:01:49conversations with other people oh i mean yeah no that that is a section of the market right and obviously asics can do a lot more performance for only their their workload right and the the g and gpu makes makes them continue to be very general yeah the the i think that there's like a spectrum it's graphics but yeah i keep saying this i have to correct myself in case you come at me for getting the g wrong yeah it's it's like it's like a spectrum right of of a very very general purpose compute to something like a talus where you've got the the hardware built for a specific

1:02:25set of model weights burn into the chip yeah no loading i don't i wouldn't say that like that that we're we're going all the way there it's it's more like along the spectrum it's a step in the direction of more specialization within the hardware yeah i'm curious i feel like he was driving towards something i guess my point is being bearish on like you say you say like everything else apart from burning the weights into the chip burning weights in the chip is like in fact because you want to fine-tune you want to optimize you want you want to quantize you want to release new checkpoints of the model if it's burned into the chip the chip's useless in like a

1:02:57month or two right i guess i guess my point is how can you not like seeing nvidia more and more specialized like take its gpus from a general programming paradigm where you're just it's a general computer that you can use to program threats and with every new generation you're putting more and more specialized instructions specialized sensor cores specialized you know umma instructions things that will allow you to just control it almost as an asic almost as a collection of asics how can you look at this trend and then still be bullish on companies that are coming up with asics for ai

1:03:29in the sense that in the sense yeah because they're they're sort of they're they're evolving towards that direction almost evolving towards and like like as in as in ruben i guess like compared to ampere or you know t4 ruben is is basically an asic it is it's basically just the thing that is is used programmable asic it's like yeah like you can program obviously obviously like like i i guess i like it's very controversial to call it asic it is it is a gpu it is a it is general it does have threads i can bright cuda to control it and changes operations but but it has the solid arrays and tensor cores and tmas and tensor memory and it has these things that

1:04:03are almost exclusively useful for loading model weights it has um you know tensor core instructions that are almost exclusively shaped around the head dimensions of models that exist in the market today if you say that you're going to come up with an asic and you're going to etch something into it but the next architecture is basically going to be useless yeah i don't know i don't know i think that the thing to remember is just how long these hardware cycles are so if uh chip is coming out today that means the design process for it was kicked off years ago and they've at nvidia they've done a

1:04:34very good job of predicting where the market is going to go and and you know i mean they have the most information for sure of course but if you if you look at you know there being public open source model architectures that look more or less like early versions of the one today ruben is honestly the first chip that was fully built in that world and so you can see a lot of the understanding of the shape of the workload that this chip is going to be asked to do in the way it's designed yeah okay so

1:05:05so i'm not going to be the best person to directly answer those questions i think these are very fair questions that are obviously the first one that's based on ruben that like i've heard articulated so well i do think that uh i will make a case for uh vertically integrated model lab asics so like the open ai broadcom whatever jalapeno chip which like totally makes sense like uh so i we first had this on the pod with uh martin casado where he was like look if you have a a trillion dollar or 500

1:05:37billion dollar training run then take 50 billion of that and make an asic like it's fine like like you will get more than 10 efficiency from from the asic and like that makes sense right so so like a model specific chip yes but asic companies the interesting thing is i feel like you are focused you're hyper focusing on like like you say like the tiles stuff they are doing a lot more sort of like uh surface area engineering or like like like the actual allocations of memory and hardware and like the uh the

1:06:07communication between between chips that um probably still won't be touched by ruben but i don't know the details i see i see they typically they often talk talk about things that i would expect to have bigger orders of magnitude than would be programmably accomplished by whatever ruben does but who don't who knows no i see i see i see yeah like i mean you know like think about what is what are the real blockers to 10x to 1000x faster inference it is not the stuff that can be rearranged uh just within

1:06:39the existing gpu design intercommunication yeah like these these guys are aiming for 300 000 tokens per second they're not fucking around like um might have to program some x6 maybe i think you know it is interesting to me that you're so bearish on so much of this kernel engineering work given how much of it you've been doing recently like the more i do it the more it just seems to me it's not mega i would also add like there's generations of models being out right i think on your guys's end you see a lot of okay one day it's glm kimmy deep seek uh minimax throw in the others some are doing completely

1:07:15different stuff right gemma no encoder the latest thinking machines is all from scratch but when you look at the other side like how long have we been on the gpt5 generation right right they've been serving that thing for quite a while sure there's maybe more pre-training there's there's different checkpoints but like you actually can squeeze quite a bit out in you do a multi-billion dollar train run if you can make it x percent more efficient they serve it for a while same with uh say the cloud five set uh family right like they release a new model like they release gpt6 now or whatever and

1:07:47they're releasing new model every year and we well we don't know about if we assume that they're changing some bits of the architecture and not just doing like post training like you're gonna be spending 50 billion dollars a year every single year coming out with new asics for the model and throwing out the asics of the previous year away yeah yeah easy so so i think okay um i would slightly disagree based on my again it's all secondhand uh on the longevity of a model there's still people out there using 4.0 yeah yeah llama not llama 2 but llama 3 i still see llama 3 workloads yeah because

1:08:18if it's done if it's trusted uh don't change it if it works which is one of the promises of open source right like the whole 4.0 save 4.0 movement like you don't gotta have a save llama 3 movement you just gotta have an 8.100 somewhere i think at some point there's also the question of if a model can do enough and use enough tool calls and be agentic enough uh can it just web search tool search write code do you really need to keep squeezing more we will because you guys will make it cheap and fast

1:08:49and smaller and i can swap it in but at some level like you give me 5.2 today or say whatever 120b model i can run with it for quite a while right this is assuming like you don't need intelligence i think there's a lot of you need reliability and predictability like i'm an enterprise like like this is tried and tested it is signed off by like my 5 000 stakeholders right like i'm not it runs a batch job every day and i like the results the results are predictable yeah yeah it doesn't make sense to keep using them like stuff gets sparser cheaper better right but that doesn't

1:09:24mean that old models glm 5.0 isn't usable right if we hit a stall say for whatever reason there's still a lot that can be squeezed out we're going to run out of time i i did want to also make sure yes actually we happen to have this diagram pull compare this versus any cerebrus diagram right um i don't think edge and uh medics have put out public uh charts yet uh but the complete the real estate is very different the size very different right this is not wafer scale right this is there's probably like i don't know a few hundred of these on on a wafer um i don't know i don't know

1:09:54how big the comparison is but like uh it is a it is a very like real estate allocation uh difference few few dozen i would say few dozen before we move from hardware i have two quick questions one the latest gimme which is really big uh three trillion yeah doesn't fit on most hardware on single node yes you need you need gb300 you need gb300 or a mt uh it's simple math um nvfp4 2.8 trillion parameters uh 1.4 terabytes um the gb300s have uh 288 gigabytes each um so

1:10:32across eight of those um you have enough room for the model and honestly like so the other thing with with gpu vram math is you have to leave space for the kv cache and that's going to depend on to some degree on the context length um so when a model is both has a very large number of parameters and a very long context length you're kind of like fighting over space um which is why you know the kv cache offloading uh would become like a more salient topic i think with these huge

1:11:05models uh because you you just you're very crunched for space with the rubens you now have what nvl 72 rack of 20 terabytes now you you still have nvl 72 on on uh blackwell as well but um you can't necessarily assume you're going to do inference on that um there's a whole lot more 8x racks in the world than there are nvl 72s yeah i guess my last quick question on hardware was do you notice anything with hardware generations for new pre-trained base models so one of the things

1:11:38you said for efficiency is you can swap hardware that's one of the 2x gains um when we see new stuff coming out training wise on rubens uh any changes on logs does this affect what type of models we will be seeing when these are more available and can you they get bigger like people understand the ceiling that you have in terms of how many parameters of a model you can run uh given the sort of latest inference hardware and that kind of forms a ceiling and so for example when deep seek

1:12:12r1 came out it was you know it was 671 billion parameters which at the time was really huge and i think did a lot to push us to really quickly adopt blackwell and get good at serving on blackwell um so yeah it's it's mostly in my mind about uh model size and then about matching the architecture and the native quantization to the the target hardware like we talked about with like you know all nemotron models of nvfp4 for example so we talked a lot about lms you have a lot more in the book

1:12:47what about audio video what's the other side of inference engineering ali you're pretty big in video diffusion video diffusions i think are like they're just shaped a lot of the stuff that you can think about reason about with lms being autoregressive with video diffusion it's it's it's not the case for instance you don't do you don't do batching you every request just comes in on one gpu and it serves on gpu you don't have to shard the models are a lot smaller like 1.2.2 for instance as a 20 billion parameter model um you don't need to worry about it's like orders of magnitude smaller

1:13:19than the best lms and it's one of those spaces where the the open source models are like with lms we see kimi k3 is almost comparable to you know mythos or like gpt 5.5 the difference between the best open source lm and best open closed source lm is very small like it used to be six months i don't think it's six months anymore i think it's like basically almost unparated video models are definitely not there's a huge gap if you look at the best video that you can generate today with an open source model like 1.2 versus something like with cling or video difference is night and day

1:13:51so it creates this disparity where media companies will choose to go most of the time to close source models for instance and i were to tell you hey i can generate an entire three hour movie for you with this model and i'll optimize it so that you only have to pay me ten dollars but if they were to do it on a closed source they'd have to pay a thousand dollars which is 100x like i'm 100x cheaper but it's still a thousand dollars they're still going to choose to do all of their cuts with video and cling so the it's like a chicken neck cycle where less demand causes less innovation in the field causes you know less open source checkpoints to be released and some of the labs that were

1:14:25releasing open source models like one will have closed source their latest models like one 2.7 is not open source we're still in one 2.2 the the challenge with with video models especially is the number of of tokens so so so so video models you want to generate a high quality model a high quality video so let's say you're doing 16 frames per second that's like the absolute minimum you'll do and let's say you'll do like 480p video so you can think about your like dimensions and i think have like a good um just like a diagram that shows the number the sheer number of tokens right let's say

1:14:56you're looking at like just one video of like you know sparta sparta 300 or whatever so let's say we're looking at like four frames right those four frames of that video if you go just if you're doing full attention if you go a little bit up like you're looking at uh 480p by 720 by 81 frames in just five seconds because 16 fps by five right and then you compress it down to latent space but you're still doing 30 by like 50 by 21 tokens yeah which means that for attention for just five seconds you're running attention on 35 000 tokens right so the attention becomes such a huge bottleneck

1:15:31and because it's open squared if you're doing like if you extend that to like 10 seconds well it's just squared 20 seconds 30 seconds so to generate a good cut scene of like one minute it's almost impossible to do within the same compute time and it's just it becomes unfeasible you can't do it and so you end up with moving towards two direction either you decide to do attention on the entire video at once in which case you are forced to displace attention so if you scroll back down to the version the video image like you can see whereas on the left for instance i would be doing full attention where every single token in that sparta 300 scene attends to every single other token as

1:16:03you can see the sheer number of like red patches on the right i'm only attending to each token only attends to like the top k top 12.5 percent that's important to it which can be like spatial so like you know the token that represents the crown attends to like the head the face and then the head on the other frame in the previous frame temporal locality spatial locality that kind of thing this results in terrible video quality and the whole point of the the post or the article here is to show like how you can train and you can do all these things but you will still suffer your quality a little bit so you end up with one of two things either you bite the bullet you have huge compute and you do full attention over like a million tokens because you're trying

1:16:38to generate like two minutes of video or you move towards autoregressive video autoregressive video seems to me like that is the bet that the future is going to be making but there are no good open source autoregressive video models out there today and that seems to be the way if you want to get like an hour movie if you want to see video models generating like an like you know hollywood level movies they have to be autoregressive in order to exceed the five second frame or there has to be some insane leap that happens in compute that allows us to do full attention over like millions of tokens at the same time in a in an efficient manner even millions of tokens it's like you're

1:17:10you're quadratic so you're going to get there really quick i think yeah can you explain the pros and cons trade-offs of autoregressive so one that comes to mind is you know the consistency across frames you will 10 minutes into generating autoregressive diffusion you're gonna forget but what what are pros and cons of this well like like autoregressive llms you can take a lot of your uh sorry autoregressive diffusion models you can take a lot of your optimizations that we discussed with llms like spectac and stuff like that and you can apply it there and you can if you have a very high quality scaled up model there is no reason why i can't stream the

1:17:43outputs as and i can show you the first frame and then i'm like like kind of like gpt back in like 2023 when you're like now it just almost like one shots the text back then you could read and it's generating as you read with video models you can watch and it's generating as you watch it generates the frames and and so token by token generation will allow us to to scale a lot up and apply the attention mechanisms there the downsides is every single autoregressive video model is shit it's just terrible quality if you like it's just if you put if you put the quality of any opens like 1.2.2 versus any other autoregressive model you can see like a like a video generated by 1.2.2 is like you

1:18:17know a cat and dog fighting autoregressive model will give you like degraded tom and jerry quality i don't know the the solution to generating long output then becomes okay we're not going to use autoregressive model we're gonna if you look at some of the things that like um like grok imagine or grok video does and they do it really really well is they'll they'll try to stitch these you know seven second chunks together and so you generate seven seconds and then you're like okay i'm gonna can you extend this video and they'll chunk two videos together open source doesn't seem to have the tricks that they have there and and by definition it's close source we don't know what they're doing

1:18:50but the closest you can get is taking the last frame of a video and feeding into like a text and image to video where it will take the text the prompt and it will take the image of the last frame and you'll ask it to generate the next five seconds and that's kind of like how you you can extend this level of the model to generate like a move where you're just you're constantly streaming frame by frame but you get drift so you start with like you take the image and then you generate a video and then that next five second video is like lower quality and the third chunk is like even lower and the fourth chunk is even lower and like sometimes you'll see things where like the the new video is like just

1:19:21ever so slightly darker than the first one and the next one is darker than the second one until like 25 seconds and you have black screen like it's just it's it's it's a we tried to have a demo that would show this but it was like it was it was extremely embarrassing to show like we just decided not to because it seemed to like but it is it is um i think models will get there they just need to in my mind scale up significantly and move towards being ultra aggressive but the training techniques don't seem to be clear there for those on who are interested in grok imagine we did a pod with ethan her from from that team right uh who dropped a little a few hints but not not enough that we can

1:19:56fully reconstruct everything specifically on this part that he explains a bit yeah so we talked about memory and uh longer context and all those things but as far as i know they're it's not all too aggressive even though like no one in industry that's all too aggressive yeah it seems to be yeah the key thing to understand between a auto aggressive model and a diffusion model is that diffusion attention goes in both directions while auto regression it only goes forward in the sequence so that's why you see this sort of like going off the rails behavior both in if you sort of naively construct a video generation model as simply generating a linear sequence of frames you can't then go back

1:20:32in that sequence and fix something to make the whole thing consistent well of course the reason that we need all this latent space for the video model is like you said we keep all the tokens in memory we iterate over that that full sequence and you can you can adjust the past in in order to make the future make sense so if we think about the architecture that's going to get us there to these longer richer sequences it's probably like you said going to be a mix of the the auto aggressive and the fusion um working together to do what each piece is good at well you get like you intuitively get

1:21:10right so like english for instance or just writing in languages like it's just left right you can stream your tokens you can stream your chain of thought just even as a human you write like you just you write and then you think about what's the next thing you're going to generate and then you write that and then you think about your ideas and you generate forward and sure you can argue that as you write you need to go back and you want to edit some things but you need to do that you know less soften than you think whereas with video there is no sequential you know the pixel in the top left corner of the video and the pixel in the bottom right corner of the video they both need to attend to each other to understand how the video quality is going to be almost as equally whereas

1:21:43with text you don't need that as much is there a parallel to audio like i'm not 100 confident on this but there was a point about a year ago where there was audio lm there's diffusion for audio and autoregressive and for the points you mentioned mostly on the inference side even though they're shorter clips you know most music is three to five minutes um we've basically swapped over to autoregressive yeah i can't speak to music but speech is autoregressive you uh effectively i mean this this was even back with like the orpheus architectural a year and a half ago you just

1:22:16add a bunch of waveforms to the vocabulary so that the lm can output tokens that represent those waveforms and then you construct speech and that's how you stream it that's it wow that's my aie talk from 2025 uh nice nice nice nice but it's not with audio it's not the same challenge does it because you like audio is sold with an lm that generates everything like like with audio it's still a transcript that you can generate an lm so your audio model just needs to like transcribe it next to speech for music there was a phase of a trade-off between diffusion for music and autoregressive and they

1:22:51were both pretty on par there's probably more pros and cons to either i just wanted to focus if you had takes yeah i don't know about music specifically you know what with what you said about editing your writing you know obviously i think my editor would tell me i actually need to do that more often and go back and fix things i can imagine music or poetry for example where you have a rhyming scheme and you might want to go back and make a change to make it to make it easier to set up a rhyme that

1:23:22you want to make later on uh there being some advantage to being able to attend in both directions uh but yeah to my knowledge you know i i very much bifurcate this this influence problem into the autoaggressive models which have a set of constraints and techniques and the diffusion models which have a set of constraints and techniques and um i think of text um embedding uh voice in and voice out as being in the autoaggressive side and then image and video being in the diffusion side there's some overlap between the two it's not a perfect split but that's the broad categorization i use

1:23:59i should point out i think it's confirmed right nano banana and gpt image are autoregressive image it's kind of this blended approach that we're talking about um but in the the image space uh it hasn't like made its way over to the video space at least in the open source world yeah but like i i assume that's not too far away if if that is possible on on the at least the quen image guys are trying it yeah yeah with uh yeah i'm really excited for quen image 3 i hope they open source it and then i'll also mention on the diffusion for text side there's been some movements not a lot

1:24:34yeah we've got mercury um you host mercury yeah nice nice there's gemma as well right diffusion gemma uh diffusion gemma is open source yeah um and then uh and we're on the science part we just have been releasing uh some uh virtual cell models that use diffusion as well yeah they they have built it's definitely still in the sort of cheap fast tokens um world yeah uh we're trying to it's i think it's the wrong marketing and i've told them this before i was like look like you're not going to

1:25:05beat the optimizations that you know the other lms are going to do uh but you can you can have different apis like you should be able to use it differently than chat response chat response how so because it's diffusion because you can do like what what is like context-free guidance for diffusion look like for for text like give me a give me a poem give me a plot structure that like diffuses into place exactly so that's where you know like i mentioned with with poetry for example where you might want to ensure consistency across you i've done a lot of lm sonnets it used to be one of kind of my go-to

1:25:41benchmarks and even models today yeah yeah they they don't get the syllables right and if you can attend across all of the different tokens uh you can you can get the syllables right yeah uh and uh david holtz from midjourney was uh investing in text diffusion i don't think anything came out of it but like the idea was that you can storyboard a long movie and then you can generate the scenes with video normal video gen but the the idea of like coherence across a thing that would just appear where like the the the end should attend to the start and you should not have this autoregressive

1:26:16path dependency does make sense in principle just the api should be different the marketing should be different none of the most heavily used open source or code source models use diffusion but isn't that like doesn't that point to almost like a it's it's chicken and egg because what if you just give it more scale what's the largest diffusion all along i don't think it's very big uh i don't know the parameter count on this one but like under 20b diffusion gemma is not i think it's a 20 something yeah yeah you know oh it's like you haven't actually tried you haven't

1:26:47given it a big one and you haven't so it's like very unfair if you should generalize a 25b and and that's what i'm saying is it's like foot size it it does pretty well um in terms of in terms of quality it's almost like the same challenge with video models to have the same size it's like you're comparing it to models that are much larger in scale yeah well unless you do the whole thing where you you have a text uh backbone and then you know you you like glom some kind of decoder thing that uh does that like you know she started off the podcast doing this for the uh for the inverse

1:27:18direction from image to text right uh and i think like it's it's it's roughly intuitive that you can do the opposite direction i agree i see it i see it yeah i mean we're speculating on on research in general um one part that we can end off with this is the the topic of your talk where uh inference engineering used to just be like let's take an open model make the gpu go burr and then that's it that's the job of phase 10 uh now it looks like people are using inference more and more in post training yes um and training and inference yes it's training for inference and inference for training

1:27:53both have become big topics well inference for training in the sense that like you obviously you just need you need to do you know you need to do rollouts when you're doing like oral training ones and so if if your rollouts are taking a long time if like you know you're using a vlm for instance as opposed to crtlm or if if the model that you're trying to train is not supported in crtlm and you have to fall back to an older inference engine your role is going to be slow and you don't want to do training on rollouts that are too off policy so you have to wait for them so you bottleneck your entire training pipeline and so like obviously the techniques that we we do inference optimizations

1:28:27for um will help them there the training for inference mostly comes down to like just the spec deck training eagle head training um and sometimes post training for instance if you want to quantize a model you'll quantize it down to like nvfb4 how do you like sometimes sometimes you get lucky and you can just do ptq and that works sometimes you quantize it down to nvfb4 and the model is is terrible like the quality is too bad and you have to do post training on the model in order to make it understand that it's going to now be in nvfb4 and let it still output the same

1:28:58logits you can do this with normal sft pt um you know quantization aware training all of that stuff but more and more so we're seeing techniques like nvidia released a quantization aware distillation paper where you you establish a version of the model that's in nvfb4 and a version of the model that's in full precision and then you'll do distillation training based on the logits of the two models in order to make the fb4 model understand and so more and more of the team the engineers like of the inference engineers that work on our team they have to be very familiar with like training techniques and and just being fine writing training pipelines for it yeah it just

1:29:31seems like uh they're meshing together in a sense well it's coming together yeah absolutely i mean if you think about the ultimate goal potentially of having a continuous improvement system you know yeah i mean it's it's it's kind of funny but at the same time it's also kind of happening and and i think within a few months to a couple years like a lot of leading agent builders are going to have these loops like really up and running in production where you are doing inference learning from the

1:30:06inference we obviously for a long time have been sort of like learning from inference as it's live and dynamically adjusting the system uh you know any kind of dynamic adjustment is going to beat a static configuration across you know your exact config across your speculator across that kind of thing and then the you know you can take the the traces that you're generating from your product continuously post train the model all those out a b test um get better signal get better model get

1:30:40better product that that loop is is really promising um the technologies and the infrastructure to build it are coming along quickly and so the sort of unification between training and influence i think is is only going to accelerate i actually was chuckling but i wasn't i didn't think it was funny like it's actually real like one of the big things for aie world's fair was that yeah you know we have rsi unto agi it's the rough tagline uh which like yeah i mean we have uh yeah i saw you pull a parameter golf like

1:31:15we have models training models and uh the next step is obviously models training uh optimizing their own inference which is kind of funny i wonder if uh models will be like on policy better at training themselves than training models that they are unfamiliar with this these are all like very interesting open areas of research one big part of my job a couple years ago was for any arbitrary model that came out on hugging face writing a config foot and kind of getting it up and running and now the get it up and running config is is one shotable um and so you know i don't have to do

1:31:49that anymore yeah i mean that's not exactly a model optimizing its own inference so much as a model like being able to read the sglang docs but uh yeah i mean well we do see it we do see it like with jlm5.2 friends and jlm5.2 is very very good at writing gpu kernels and so for like it was very funny internally we had a jlm5.2 endpoint that we were using to like that we plugged in in our cloud code harness so every engineer on the same user like our jlm5.2 and it will do a forward pass on the jlm5.2 instance of the you know the node and then it will get the profile trace and it will

1:32:25analyze it and it will find the kernels that are the bottlenecks in sglang and then it will write the new kernels and then we'll do another profiling trace and when it's done it uploads the image to our thing and then we can pull that image down and repeat the cycle and so for quite a bit of time we had like literally jlm5.2 and and like some of the gpu kernels that we're on jlm5.2 within our inference engine is written by jlm5.2 and the trace and the kernels were guided by jlm5.2 as the driver so it seems like i do see i do see that that circle being there i think a bit more time is needed

1:32:58there's definitely a lot of things that i can't do the models just aren't there yet even though they're like really really smart like they still try to like reward attack their way into like the the cheapest or like they're very like they're not good at like decision making almost it seems but yeah i do like yeah like a model optimizing its inference is is already a thing that that happens