Powering the AI Inference Wave with EPRI's Ben Sooter - Ep. 292

0:00Welcome to the NVIDIA AI podcast. I'm Noah Kravitz. Today we're talking micro data centers with Ben Souter, director of R&D at EPRI, the Electric Power Research Institute. The relationship between AI data centers and energy grids is an increasingly important one, to say the least. In a moment, we'll talk about how micro data centers can help strengthen that

0:31relationship. But first, a quick note about GTC San Jose. Join us at the world's premier AI conference. GTC San Jose is online and in person March 16th through the 19th. From physical AI and AI factories to agentic AI and inference, GTC 2026 will showcase the breakthrough shaping every industry. Learn more and register at nvidia.com slash GTC. Ben Souter, welcome. Thank you so much for taking the time to join the NVIDIA AI podcast. Really glad to have you here.

1:03Yeah, great to be here, Noah. Super excited. So, Ben, to kind of set the table before we dive in, for listeners who don't know EPRI, can you briefly explain, well, first, who you are and what you do? And as part of that, what EPRI is and what EPRI does? Yeah, absolutely. So, EPRI is sort of a unique organization. We're a 501c3 not-for-profit. It's an independent institute focusing on R&D, collaborates with more than 400 companies across more than 40 countries, and really drives innovation to ensure sort of the public has

1:36reliable and affordable energy. So, really awesome mission statement and been a really exciting place to work. You've been at EPRI for a couple of decades now. Yeah, I've been here a while. I just crossed over the 20-year mark, which I feel like is ancient times in the way the corporate world works now. Right. Well, congratulations. And you kind of, this is exactly why I asked, because I'm thinking about data centers and AI, but hearing you talk about things like nuclear and thinking like, man, you've like, must have seen some things and worked on some projects. And thinking back,

2:09you know, over 20 years and how technology and energy reliance and consumption must have evolved. I don't know if this is a fair question to ask, but can you kind of place our current moment in context to, you know, sort of what you've seen with how the world uses energy and stuff you've worked on over the years? Yeah. Yeah, that's a great question, a great way to frame it. And it gets to why I probably have stayed here 20 years, which is that there's just so much, you know, change and a lot of different

2:39exciting things that have evolved across the sector and the industry that have sort of landed us here today. So lots of stuff going on. It's been interesting as you come in and I'm sitting here in Knoxville, Tennessee, and behind me, we actually have a big laboratory that makes up the back half of the building. But pairing with that over the years here at EPRI, I've seen all kinds of technologies come through, whether it's, you know, solar energy, battery storage, electric vehicles, all kinds of things. And what's interesting is you see a lot of it several years, like a lot of times

3:16in advance of when it's cool and it's blown up and it's everywhere. Right, right. And so it's been interesting to just see all these technologies come in and evolve and the challenges that come with them, you know, whether it's, you know, how do we, how do we handle the loads of electric vehicles or how do we position the distribution system to handle all the solar capacity, all these different issues and then meeting those challenges. And so that's, it's been

3:47an exciting place. And I've had, I've gotten to have several lifetimes here because I've been here for 20 years. And so working through different areas and now kind of in this AI space, which is obviously just accelerated everything about 10x. Right, right. So let's, let's get into that then. Data center, AI, you know, when I say people, when I think of, you know, AI and energy consumption, data centers kind of popped to mind sort of immediately. There's more to that, obviously. But can you kind of set the stage a

4:18little bit for, you know, kind of explain what a data center is in this context, and then maybe that can get into what this idea of a micro data center is and how it differs, you know, how those differ from the kinds of things that, you know, people like me usually think of when I hear data center. Yeah. So, so good question. And, and, you know, I think there are, there, there are several flavors of data center at this point. And I think kind of the, the, the two I'm going to sort of hone in on today. One is the, the data centers that have been really in the news a lot lately. These multi-gigawatt

4:54behemoths that are being built with the objective of providing platforms to train these really exciting AI models. And so, you know, there's been an enormous push to build those, those types of capacity. Obviously, there's been a big crunch for power in order to meet that, that demand. And so a lot of lots of exciting research in that area, but all of that's really been directed at, at making the models that are going to potentially do exciting things for us in the future. Right. But the training

5:26of the models. Exactly. The training of the models. But, you know, and you mentioned this, you know, in the, the plug for GTC at the beginning, inference, I think, you know, people don't realize that while we're so focused on training the models, like there's this huge wave that's coming of once we actually get all these models and we move beyond just chatting with JADGPT and we're doing the, the real-time translation in our AirPods and we're doing the smart glasses and we're doing all the full self-driving and all these different, different applications that all that, all those

6:03applications all falling into inference, sort of using the models, it's going to sort of accelerate this second compute wave that comes along with all this in order to, to have the compute capacity to actually do all this stuff. And there's actually a, there's an interesting statistic out there that if you look at like the lifetime of a model, so if you look at a, you know, a GPT 5.1 or whatever, only about 20% of its like compute capacity and thus its power consumption is in the training side. 80% of it is in the inference side. 80%, okay. Yeah. And so the vast majority is actually

6:37in the inference side. So if you think about how much capacity we're building for training, we're going to need, you know, a couple of times that to meet the demand for, for all the inference. And so people start using these things or close to full max. Yeah. Yeah. And so, so that's going to create another challenge. Thinking about energy consumption is the distribution of energy consumption during inference as opposed to training. Is that just massively different and much more spread out? What does that look like from the perspective of, you know, energy load and consumption and

7:10figuring out how to try to balance things? There's a lot of great questions in there. And a lot of, I try to throw 12, 13 of Mechia. Yeah. And a lot of them are things that we're looking at as part of this micro data center project. Okay. So when, you know, when the, the world, when we got into these gigawatt scale training data loads, nobody really realized or thought about the fact that these, the, the way the compute and things would happen is, is the training loads would slam hundreds of megawatts of, of demand, you know, nearly instantly within milliseconds. And

7:44they can also fall off once that job is done. And so huge swings of power. And that created some consternation where, as you know, you had to solve sort of the technical challenge of meeting those demand peaks and, and, and spikes and things. You compare that to inference. And when, when we got into this, we started down this journey about midway through last year. And, you know, I was initially imagining this and I'm thinking about, okay, if inference is when I'm using, you know, one

8:15of these awesome models, I'm using ChatGPT, I'm using Grok, I'm using Gemini. And so it's being, the, the compute tasks are being generated by me. So that's going to give it what we call more low diversity. It's going to kind of smooth it out because it's being randomly generated. Sort of my initial hypothesis. Really interesting discussion last week with someone as we started bringing up just the whole agents and agentic AI that has taken over in just the last few weeks, you know,

8:46bringing open claw at the house, you know, to take over my world. And, and I started realizing like, oh man, like that's doing all this work at night now. Right. So, so when I originally thought this load was going to sort of look like a normal load curve for just people waking up during the day, putting on lights and, you know, air conditioners and stuff. Now, all of a sudden I'm like, well, that completely changes the paradigm because now it's running at night while I'm sleeping and, and is it going to do more? And, and so I, you know, to answer your question, this was a really

9:16long winded way to answer it. No, this is, I want to go deeper and ask you what you're doing with open claw, but maybe that's another podcast. So yeah, that may be another podcast because it's, yeah, that's, um, trying, trying to streamline the, how you actually survive in the 10X, uh, corporate environment. Right. But, uh, all that to say that that paradigm sort of evolving now, and I'm having to change my hypothesis. And so when we get, you know, when we actually start monitoring these data centers and things and actually building them out, um, and realizing

9:47and measuring them, I, it's going to be really interesting to see what they look like. And, and I have a feeling you're going to see lots of different loads because it's something that's very consumer centric may look different than, yeah, there were some, some great stories last week, um, of like some big financial institutions that were very AI forward and have, have invested a lot in models and don't have enough compute for their internal models. Yeah. Which is another, as a whole nother, you know, it fits very well into what we're looking at here,

10:18but it's a completely different, probably, you know, uh, shape and, and, uh, yeah. Yeah. Well, let, let's dive into what we can kind of grasp at the moment or, you know, as concrete, I should say at the moment, uh, and this idea of micro data centers, can you, you kind of alluded to it in talking a moment ago, but can you talk a little bit more about what they are and why now, and, and what are some of the problems and, and these may be some of the examples you're mentioning, uh, are you trying to solve for the power grid as well as for AI users with this idea of micro data centers? Yeah. So, so, so great question. So the, the real

10:53thing that, that we're looking at here, I mentioned everybody's focused on these big giant training data centers. Now we're thinking about how do we create these data centers for inference? And when you actually look at those data centers for inference, and one of the things you, you start to realize is that having the, the huge mega data centers that are centrally located don't necessarily make sense for the inference data centers because they are more consumer centric and user centric, uh, positioning

11:25them geographically around where, you know, the, the people are, um, uh, tends to make more sense, um, because they're, they can be more latency sensitive, um, et cetera. So you don't necessarily want to have them just in one place in the middle of nowhere, um, better to have it broken apart. I don't know. I may be way off here, but it reminds me of, um, when streaming media centers, you know, started popping up kind of in the, whatever period of the aughts, I guess, right. The first.com wave when

11:56multi-measster become a thing. And yeah, that kind of proximity, cause it affects performance, as you said. So. Yeah. Yeah, exactly. When, you know, the early years of Netflix where it started off very centrally and then they start, they realized, Hey, if we, if we, if we put a mirror onto the local, the local networks, it, it becomes a lot easier to distribute. So yeah. Yeah. It's another thing. And, and, and incidentally the, uh, the biggest user of geographically dispersed servers, game servers. Yeah. Right, right, right, right. Through this like journey, like learn that little tidbit.

12:30Can you walk through a little bit what happens in a micro data center in terms of sort of, you know, how do you design and build for an inference load as opposed to, um, a training load? And what does that mean in terms of both the energy usage, but then also like the ripple effect of not housing and everything in these central giant megawatt data centers that, as you said, at least for training, you know, they act differently than other big loads on the grid. They come up super

13:00quick and big, you know, and I imagine all kinds of other problems that are beyond my knowledge set, but just, can you talk a little bit about how they sort of work on that level? Yeah. So, so a couple of things, um, uh, it, that are kind of in, in that, that onion, uh, to unwrap. So, you know, I, I, the first one sort of on the, the really underlying needs, uh, underneath construction, it's somewhat similar. And the fact that, um, it's still very sort of GPU or TPU based compute need in order to actually run these models. Um, we're, you know, we're seeing,

13:33I think more chips like NVIDIA has more chips, more designed for inference and training now. Um, so there seems to be a little bit of diversification. What was now just sort of one, one chip initially. Right. And so we're, we're, we're seeing that. So there, there is some variability, I think maybe in the underlying chips, but, but traditionally it's been sort of the same chip that for training and inference. And so from that perspective, it looks similar. It's just smaller, um, because I, I don't, I don't need as much, but you know, I, it's

14:06as sort of the result of, you know, I don't need how, as much, it's sort of like, how much do I need? And that's, that's been one of the things that we've been looking at. And one of the interesting things that we've been working with technology partners like NVIDIA to really help us understand, you know, what the compute needs of, of the actually technology companies that are, are buying these data centers, using these data centers, looking in that, you know, is, is three megawatts enough? Is five megawatts enough? Do we need 20 megawatts? And there

14:36seems to be, we seem to be coalescing somewhere around, um, this idea of, of 20 megawatts. Um, but that's actually sort of, I, I hadn't gotten into some of the, the electrical aspects of all this, but as we're looking at where to place these micro data centers, 20 megawatts is, can be the not insignificant ask, um, of just dropping, dropping a load somewhere onto the grid. And so the, there's not a lot of opportunities to drop something of that size. Okay. And when EPRI was looking at, okay, you know, our partners are telling us

15:12about this, this coming compute wave. And we want to, we want to do what we can to help our utility members be proactive and get ahead of it. Where can we look at opportunities to find power for this type of data center? Right. One of the things we started looking at was, well, there's substations all over the United States and indeed all over the world. And there's a fair number of them that are actually underutilized. So they've got excess capacity available inside them. And so we started thinking like, well, is there, is there an opportunity there to partner with those substations that have that excess

15:48capacity and do something and, you know, put these inference data centers near it and, you know, maybe it directly adjacent is maybe ideal, um, but, but close by and, and make sure, you know, we've got everything that is needed in terms of, uh, fiber access and, uh, you know, right. All the infrastructure, all the, all the underlying infrastructure. And, and so look at all those things, um, and say, you know, does that work? And we thought, we thought that was a good, a good idea, but the answer is you're probably going to find three to five

16:20megawatts, maybe up to 10 megawatts of available capacity in a single substation. And so then we started thinking about, well, well, how's that going to work? Ben, just to interrupt you real quick, sorry, cause I, I keep having a picture in my head of, this is my own ignorance about our electrical grid of how big an exit, one of these existing substations is and where it might be. Is this kind of like suburban as opposed to metropolis? Is that, is that? So it could be both. Okay. So, so there's a couple of caveats in there. So you're right in thinking that your suburban

16:53substation may be more likely to have some of that excess capacity. Okay. That said, we have found that there's interest at the metropolis level too. In capacity. In capacity, because there is, there is need, you know, there's people there, so they want to get the compute close to it. And actually, if you see, you know, some of the metropolis environments, there's a lot of real estate that's available right now. Right. Okay. Which, which equates to load that's not there. So there's, there's opportunity to, to, to

17:25put load. Um, so that was another hypothesis going in. Yeah. There wasn't going to be interest, but actually it looks like there may be interest and opportunity at that level as well. Yeah. And so, you know, as, as, as you're looking at these data centers, um, and you start to say, well, does three megawatts make sense? And does it make sense for the person that wants to buy it? What we realized was maybe there's an opportunity and this is, this is the distributed part. We initially kind of called this project distributed inference truthfully. And while distributed inference seemed to be very technically accurate, it did

17:58a really poor job of giving anybody a visual image of, of like what it was we were talking about. And so what we realized was if we go to an opportunity, if we go to a regional area, we go to a city and we say, Hey, is there other five data centers that meet this criteria? And then, you know, each data center maybe has five megawatts of capacity. Now we've got five data centers at five megawatts. Um, and now we've got 25 megawatts of capacity. And so actually looking at it as, you know, instead of, of, of a single project, that's five megawatts,

18:32looking at it as a 25 megawatt project that just happens to be distributed across five sites. And so that, that helps meet the needs of like what the utility grid, you know, has available and sort of meet the economics of what the data center companies need in order to actually make it realistic and viable for them. Right. Right. How does this approach affect the way the grid functions for just, you know, people in general, the city, the region in general? So, so great question. And we've really sort of seen this as, as a win, uh, a general win for

19:06everyone. Uh, because the answer is if the existing substations are already kind of sunk cost, we've, we've invested that capital. Yeah. We've made the investment, we've built it. Um, and so if we can get, you know, extra capacity, if we can get extra usage out of existing assets, then that's sort of a win for everyone. Right. You know, if you're at a societal cost, if we're not having to put new steel on the ground, then that, yeah, that's helping, you know, keep rates lower, um, and,

19:36and things like that. So, so we really see this as a positive in terms of being able to leverage existing infrastructure speed to power, I think is also a big part of this where, you know, there's a huge scramble to, you know, for these, this, uh, capability and, and everything. And so it also means that you no longer have to deal with interconnection cues because you're off the transmission grid and, and, and all the things that go along with that. So it definitely speeds up the ability to get to a finished product that that's online and serving customers, um, uh, much

20:12faster as well. That's great. Are there, uh, clean energy implications? You know, you, that it's interesting you say that. So there's, you know, there's definitely, there's opportunities to, to layer all kinds of things on this. Okay. So, so there's opportunities, uh, to layer this, uh, with DER and, and, and, you know, solar, wind things. And there's, I think there's also a lot of opportunities for energy storage. One of the things we've been looking at, uh, getting sort of into the technical weeds. We've been looking at flexibility and, and, and how, uh, what you find is that you'll,

20:47you'll have a substation and it's, it's got excess capacity, but it's actually got quite a bit more capacity except for July 21st when you have the hottest day of the year. Right. I'm making July 21st up. That's not the hottest day of the year. Somebody fact check me. I was like, wait, what AI breakthrough happened on a July 21st? Right. Um, just, uh, been made up a date. Right. Super hot day. Yeah, yeah, yeah. So if you, if you're, if you have, if you can engineer it so that you can have flexibility to reduce your load, you can reduce your demand during those peaks, you actually have a lot more,

21:23you know, envelope that, that, that you could potentially use. Um, and so pairing it with energy storage, backup generators, just, uh, working with the technology partners. One of the, the other nice things about if you have sort of a distributed network, uh, of these loads is if there is, you know, possibly like a peak demand issue, I can run down my compute and wait at the center and route the, route the calls someplace else. Right. And move smooth things out that way. Um, so there's lots of possibilities. And so that's, that's another thing that sort of makes this exciting and a really

21:57neat way that a tool that the utilities could use as well. Yeah. Yeah. No, that's very cool. Continuing sort of along the lines of the applications of all of this, but kind of from the other side of it. And, and again, you talked about this, um, in reference to, you know, building the data centers, these smaller data centers close to where the users are, the consumers are and that performance aspect of it. Um, but there, are there other examples of real-time applications that, um, as this infrastructure rolls out, you know, you think will, will be, uh, you know,

22:30enabled or, or maybe just kind of accelerated, um, these applications that could directly benefit people? I, I, I think there's all kinds of things and I, and I am, I am certainly not going to claim to have a view into, into, into all of those options. You know, I, I mentioned some, you know, like the translation and, and, and, and drive, you know, self-driving and things. But yeah, I think especially as, as agents develop, as we get, you know, smart glasses that can analyze, um, just here at EPRI, you know, other exciting things, you know, we're looking at, um, and this, these are going to

23:05have applications for everybody, but, you know, can you use smart glasses to, to analyze your poles and transformers and, and things in a substation and make, you know, your line workers smarter, more efficient and safer, uh, all at the same time. And so I, you know, there's all these applications that everyone's looking at. Can we, can we, you know, on, on, again, grid focused, uh, but can we, can we make the control center of the future smarter and get smarter about restoration times and, and, and all these different things on and on. I, I think there's just

23:37internally at EPRI, there's a few hundred use cases and things we've identified and that's very grid centric. So, so, you know, obviously the, the audience is probably not all, uh, utility workers and things, but I can only imagine that if, if the electric industry has identified several hundred use cases that, you know, around the world, there's, there's gotta be just tens of thousands. We, we, we wouldn't, we wouldn't be here having this talk on, uh, on tape, so to speak if there weren't right. Kind of, uh, uh, I was just thinking about this as I was listening

24:11to you and you, and you spoke to it with examples of like smart glasses with people that workers in the field, you know, analyzing things, but are there ways that you've seen, you know, and, and whether they're, you're using them now or maybe things that you kind of see coming that you're excited about, ways that the energy industry has been using AI to, and I don't know if it's like to design better battery storage or to explore, you know, new forms of energy or to, you know,

24:45maybe something seemingly more mundane, but still really important, like reorganizing the way that, you know, companies approach different industries. I don't know what, but are there big examples that kind of jump out in, you know, your own work or what you've seen, um, of how AI is transforming the industry from the inside? Yeah. I mean, I, I think it's transforming it in all kinds of different ways. And it's, it's one of those things that I think has been, it's been really interesting because things do seem to, you know, there's lots of memes about how fast things are

25:17going. And I, I already made some, some comments about 10 X-ing and, uh, and things, but, but it's all sort of the, the proof is in the pudding and have we seen, where's that scaled demo? I think there's a lot of proof of concepts that we're seeing, uh, pop up around, uh, and, and really the, the thing everybody is waiting for is that scaled demo of where there's this application and it's measurable and, and, and we've scaled it out to the entire enterprise. So there's, you know, there's, there's definitely a lot of work to do, but I think there's lots of applications

25:51as well. Yeah. I'm trying to go through my head. There's just so many different things, but, but, you know, every, because everything from understanding, you know, in the utility industry, there's a lot of historical records and things and several of, a lot of them predate sort of the digital era. Yep. And so current models and things can make just ingesting all of that and structuring it into, to useful structured data sets that you can then use to create new models and, and, and create analysis and, and, and digital twins and all these things. So that,

26:28you know, I think that's, there's some of the places, you know, the existing work is already really useful. Right. Obviously all the, the, the things we do every day just to, to accelerate ourselves, you know, with understanding emails and, and, you know, figuring out how to, how to, you know, have that hard conversation with the problematic coworker. Um, and it's totally making, to making these up as well. No, no, no. But it's, it's relatable. It's that, well, it's that interesting sort of, there's two layers there. Well, there's many layers. The five layer of cake

26:58is, is the, the iconic layer at the moment. Uh, but there's kind of two layers when I'm thinking about, there's the layer of like the kinds of work that I don't want to call it knowledge work, but that kind of working with information you just described that is part and parcel of many roles in many industries. Right. And then there's kind of the, and AI is helping, you know, helps me day to day in ways you were just describing or, you know, kind of making up and I get you. And then there's that layer on top, which is specific to the kind of work and the industry that you're doing.

27:28And, um, the more people like you, I get to have these conversations with just the more in my mind, I see like, you know, it's both right. And one informs the other being able to go back and ingest all that old data. You know, we've had, um, there's a cardiologist or a radiologist on a while ago talking about how much hidden information there is in old analog film scans that, you know, AI image analysis is able to extract now and it's useful. Right. And that kind of stuff is, yeah. Did you see the guy with the microfiche like repository? Uh, it rings a bell, but I don't know that I did.

28:03This is, this is a few months ago now, which makes it ancient news, but, but yeah, there was, there was somebody that had access to this huge repository of microfiche and I'm, I'm old enough. Those of us that are old enough on here will remember looking at it under the little magnifying contraption. The machine in the library. Yeah. To see the news article from 1942. Right. But he had access to tons of this stuff and started using the models to ingest it all and, and, and just created a monster data set. And it's so cool. That's amazing. I love stories like that. All right,

28:35Ben, as we get to kind of wrapping up here so I can let you go, um, this is not to put you on the spot. Cause as you mentioned, these kinds of things are impossible to, it's always impossible to predict the future, but when things are moving as quickly as they are, it's, it's harder. Right. But if we look ahead to the next year or so, you know, loose timeframe, what does success look like, you know, with micro data centers and, and even more broadly, I guess that's from thinking about putting you on the spot, both for the grid and for everyday users of AI powered services? So great questions. I'll

29:07start with the micro data center, you know, part, uh, since we're, we're talking about it. And yeah, I think, you know, it, it, hopefully in a year or two years, we've got a pile of, of, of these, you know, micro inference data centers built out and we're monitoring and measuring them. And, and that's helping educate us on what we need to know so that we can continue to build about for all the wonderful things that the industry is going to create. So I think, you know, from the micro data center, uh, standpoint, you know, that, that I think is, is what I hope what success looks

29:40like. Yeah. And then, you know, I think just in general, you know, I have no idea that everything is so exciting. It's, you know, you mentioned GTC at the beginning, I learned something new, you know, from, from, from those types of conferences and stuff every year, there's new things that come out, completely change things. I mentioned agents, you know, which, which are just like weeks old, maybe a couple of months old that we've really sort of delved into that. It's changing the landscape again. So, you know, I don't know what it's going to look like, but I'm hopeful

30:11and, you know, it's going to be exciting. And I just, there's going to be compute needs. You mentioned, you know, at the very beginning, sort of the importance of power and stuff. You know, I think, you know, there's, there's still going to be challenges to solve, to make sure that we can provide all these awesome things to everybody, um, and really move society forward, uh, and everything. So exciting times. Excellent. Yeah. Well, I I'm with you, I'm rooting for you and, uh, I'm excited to see how it all unfolds. Ben, for folks who would like to learn more about the work

30:41you're doing, about the work EPRI is doing, where are, where's a good place for them to go online? Websites, social media accounts, uh, where should they start? Yeah, absolutely. So, uh, website, so you can go to EPRI.com, E-P-R-I.com. Um, it's our, our official website. So lots of great information there. Also very active on, on LinkedIn. There's, uh, lots of, if you're, you're interested into the latest news about exciting AI and data center updates, uh, sort of, uh, and their, their adjacentness to the electric sector, lots of, uh, good stuff going over there on LinkedIn.

31:15Um, so those are probably the, the two places to, to find us. Perfect. Ben Souter, thank you again for joining the AI podcast and, um, best of luck with everything you and everyone at EPRI is doing. Appreciate it. Great to be here. Great talking with you.

31:40Thank you.

32:10Bye. Thank you. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. You