One Brain, Any Robot: Skild AI's Skild Brain Explained - Ep. 295

Introduction to Robotics

0:00Robotics is a data problem. Unlike language or vision, there is not much data in robotics. There is no internet of robot data. So if that's the scenario, we cannot pick and choose which data we use. So we go, in a most general fashion, every single instance of our brain which we deploy for any kind of task or any form factor that contributes in making the brain better for the future scenarios. Welcome to the NVIDIA AI podcast. I'm

0:30Noah Kravitz. I'm here today with Deepak Pathak and Avanav Guptal from Skilled. Skilled is a robotics company that's building the OmniBrain, a universal brain that can power robots across any form factor to tackle any task. It's amazing stuff. Very excited to find out about it from the source. And so let's get into it. Deepak, Avanav, welcome. Thank you so much for joining

Skilled Company Overview

0:53the AI podcast. Thank you so much for having us. So Deepak, maybe you can start and tell us a little bit about the company, about Skilled, and then you can both talk a little bit about your roles. Yeah. So at Skilled, as you mentioned, we are building a general purpose brain. So we call this Omnibodied Intelligence. Any robot, any task, one brain. So think of like what ChatGPT is for language, we are building a general brain for any physical device or any kind of robot. So this is absurdly general, right? You can have a humanoid or a dog-like robot or a robotic arm

1:28on a conveyor belt, all being controlled by the same shared brain, shared intelligence behind the scene. So why do we go so general? And the reason is robotics is a data problem. Like unlike language or vision, there is not much data in robotics. There is no internet of robot data. So if that's the scenario, we cannot pick and choose which data we use. So we go in a most general fashion, every single instance of our brain, which we deploy for any kind of task or any form factor

1:59that contributes in making the brain better for the future scenarios. So this is the main goal behind this. And personally, in my role, like I have been, we both have been professors before this. So we are extremely technical. We have been involved in bringing up these technologies in the robot learning area for the last decade and more. So our role is both on the technical side to make sure that these things get built and they are super general, transferable. But our focus is also a lot

2:34on deployments. Right. Like we do not believe deployment to be a, it's not hindsight scenario. Like for, for instance, in case of chat GPT or language models, folks did research for several years, but once it was ready, you have million users in seven, seven days, maybe one day, I don't remember, maybe a hundred million users in one month. Right. Fastest growing product. Physical AI is not like that. The things takes time to deploy. So for us, deployment is our first priority from day one. Yeah, it makes sense. And you mentioned being a professor, you're at Carnegie

3:06Mellon. Yeah. And the company is based in Pittsburgh. So company has HQ in Pittsburgh, but we have offices in Pittsburgh. Now we are also in Bay Area, the San Mateo area. Oh, great. And one office in India, Bangalore. Fantastic. And Avanap? Yeah. I think one thing which I want to start from is like

Rethinking Robotics

3:22the reason we are actually so excited about this is because we are almost rethinking the way robotics is done traditionally. Like traditionally, robotics has been a very classic, like a vertically oriented field. Right. I mean, so what that means is if you think before this AI era, you first decide what vertical you want to place the robot in. So like, let's say I want to build a welding robot. Now you go and start making your hardware, which is very specific to welding. You start making your software, which is very specific to welding. Now the problem with these kinds of deployments has been is it's

3:57very easy to guess the first 80% or 90% of the performance. But then you hit this wall, which is called the corner cases in the physical world. Right. There are so many corner cases in the physical world. Like someone might lead a package in front of you and now it becomes a corner case and so on. And so that is why if there is a corner case now, because you are at 90% performance, you will still not be able to get it completely automated. Human still needs to be around to make sure the corner cases are handled and so on. And that is why it has not been traditionally robotics has not

4:27really gone big mainstream essentially. Now what however, things have changed when AI came in. Like if you think language also, before this whole came in, was very verticalized. There were some different companies building chatbots. There were different companies building search engines. But once LLM came in, they became the horizontal platform. And now everyone is building on top of that horizontal LLM platform. That is exactly how we are now thinking about robotics. We are building this horizontal

4:58general purpose brain. That will and this general purpose brain is can then be fine tuned for different verticals essentially. And our thesis is that if there's a corner case of one vertical becomes the central case of the other vertical. So now the data is from everywhere. And so now it will be able to handle these corner cases through the data play with the different verticals. In terms of what Deepak was talking about, I mean, we are definitely like very similar in that profile because both of us are professors. So we do not divide our work like, oh, I do business and you do this kind of stuff. We are more

5:30think of it as extension of each other's brain and like thinking about it, strategizing about it and the whole and really, really focusing on deployment. Like humans are limited in the sense we cannot enter each other's brain. We are fusing the omnibody intelligence in the human way. I have a feeling from talking to you guys for five minutes that you might be closer to fusing brains together than you realize. I don't know, you seem to be on

Inspiration for Omnibrain

5:53the same wavelength. So what was the inspiration? I mean, you discussed, you know, in some ways, the inspiration for Omnibrain building that horizontal platform. But were there deficiencies or gaps that you saw in existing robotics foundational models? Or what was really the impetus to say, hey, we need to go do this a different way. I think if you look at the current systems, I think I mean, I've already alluded to it. In a way, when the robots are currently deployed, they are they behave more like machines.

6:28Right. So everything is measured, everything like in factory setups, everything is. So for instance, if you look at a classical automation line, you will have a robot, but around the robot, you have a big cage, everything will be measured very precisely. The whole setup may cost several times more than the robot itself. Then if anything were to change, you have to redesign the whole setup. And then people talk about consumer

6:58applications, where things change, let's say your home, right, you don't, you can no matter how many sensors you put, you cannot measure everything single thing to point one millimeter accuracy. Sure. Right. So this, this whole paradigm of robotics has the main shift in robotics has happened going from this programming in the behaviors to learning the behaviors, which means you learn that from data. So now the engineering part has gone from, okay, how should my robot move? What failure may occur to

7:30those thinking where the data will come from, or how can I make it high quality? How can I get it at scale? And that's where the shift has come. So we saw the shift in academia, like we could be began seeing results one after another, like we could get a result today and demo live demo, you know, in a conference the next week. So for us, it was like, either we bring it to the, to the, to the masses or we are the ones who just get, uh, eventually, uh, replaced by it, uh, in some way. So it was just a no brainer for us that

8:03this is the future of robotics. And this is, I think this realization is also happening at the same time in the general field. You can see the excitement around physical AI in GTC. Uh, we are working with several major players in this space to bring this. Uh, so this is, this is not really, oh, this happened, hence this should happen. This is the way to scale. If you do not do this, it is almost impossible to scale the way how things have been in the robotic space. Um, I noticed on, uh, on your blog, on the website, um, I was reading an article about training on video data. Can you talk a little bit about, um,

Training on Video Data

8:40the benefits and why you're training on video data? And is that the primary way, the only way you're training your robots or, um, are you bringing data sources from other places as well? Yeah. Um, so, yeah. So, I mean, when it comes to robotics, we have multiple choices when it comes to data. Sure. So there are three, like three main sources of data. The first source of data is videos, or maybe let's start with the robot data itself. So now the way you will do it is you have to collect robot doing a task and that data itself can be used to train the robot. However, this is a very hard to

9:15scale because you are collecting data with robots. So for every data point, you need a robot. You need humans to control the robot because currently robot, and we call this teleoperation. So you have to collect data with teleoperation. The good thing about this data is it's the richest form of data because robot itself is doing the task. So you can read all the sensor values. You can read all the motor commands that are going in the robot and so on. The problem with this form of data is very hard to, very, very hard to scale. Um, and so when it becomes hard to scale, it's very hard to learn large scale AI models on top of it. The second form of data is like something like videos.

9:49Now in this case, you are, that there's huge diversity of the data because we are collecting videos in US, people are collecting videos in India, China, everywhere. So you can, you have huge diversity of the actions everywhere and so on. So this is a scalable form of data, highly diverse. But the problem with this form of data is that it's not rich enough, right? You do not know what exact actions, what exact forces people are applying to do it. And then there's a third form of data, which is where, uh, which is the simulation form of data. Now, in this case, it's highly scalable. Simulation is as scalable as it gets. And you can collect trillions of examples

10:23in a day, for example, and so on. Um, it is also, you can measure all the forces if in a simulator and so on. But the problem with simulator is there's always what people call sim to real gap. Like simulator cannot be exact replica of the real world. There's always some difference. And so now you have to bridge this sim to real gap either through algorithms or some other data and so on. And so for at scale, we use actually all three different forms of data. We believe every form of data is critical because every form of data is complemented to others. Like, I mean, if you think videos are

10:55scalable and diverse, simulation is scalable, but not diverse. And, uh, and then the third one is the robot data, which is the richest form of data. So every form data is useful, but some data has different, uh, metric videos is not as good quality for robot training, uh, as like, for example, the real world data. So what we do is we use the video data to pre-train our models. This is a data that is available in billions already. So we can pre-train our models, uh, to build a, build a model.

11:26However, the problem with videos is if we can learn everything from videos, uh, Deepak, this is gives us, this is a great example that if we can learn from videos, all of us would be Federer's because we will watch Federer and we'll start playing like Federer and so on. So that's never going to be sufficient. Just watching videos is not going to be sufficient. If it was sufficient, I could dunk a basketball, but I can't. Exactly. We cannot. Yeah. And so that is where for us, simulation comes into play. We get the idea of what the task is, what the action is from video, but then we practice it in simulation. We robustify it in simulation, but again,

12:00simulation is, there's still a gap. Remember sim to real gap still exists. And now we take this model, which has been pre-trained on videos and simulation, but before deployment, we post train it on the real world data on the small amount of real world data that we can collect in factories or whatever task we are trying to solve. And that makes it precise and help it solve. So you get the robustness from this pre-training data, like the corner cases. Remember, I was talking about these corner cases, those videos and that simulation helps you to robustify

12:30and to make it precise is where the post-training data comes in. Right. You can also find analogies with language. I think AI has been mainly successful right at a massive scale for language data. Right. But the same recipe is there. Like you have this when you are building this general model, like to go general first and then you go specialized model. The general model is training on all of internet data, like from different sources, different articles. But then let's say you are open AI, you build chat GPT and then Amazon comes and say, Oh, I will deploy a robot.

13:02So your model in my, uh, amazon.com website, then you will take that model and you will fine tune it. Right. Right. Uh, and then you deploy it. So then data from just amazon.com very high quality for Amazon, but very low in amount. Sure. So it's used for five post-training. Right. Internet data, maybe it's low quality because people are saying different things. And maybe there is junk texts, many, many, many places. So it's low quality, but at massive scale, uh, in pre-training time. So this separation of pre-training and post-training is how the current

13:35AI revolution is, is governed. Right. Even at Nvidia, right? You have chips, uh, for inference, you have chips for pre-training. And this is the same separation we are building to robotics. And which is why we are seeing this immediate, uh, access to variety of applications, which you would not have otherwise. Can you, you, you've talked about this a little bit, but maybe kind of to put a, um, I know a narrative around it for the viewers and listeners, can you talk about kind of what it takes the process of building, testing and deploying, bringing to market something like the

14:11Omni brain? Yeah. So it's a very complex question because it really depends on the, on the scenario, right? Like, uh, in, uh, in language, it's very easy because you can ask a question. It's just prompt, does everything. Oh, sure. So the general recipe, which we are going towards is that the behind the scene brain is shared. Okay. So any single action you will take will improve the brain. Now, how do we orchestrate the deployment of this brain? So the idea is, let's say if you have some task, if we have seen that task before,

14:42let's say if it's a task of moving around or walking or jumping over things, we can do that already very well. So in that case, you can just take the brain, put on the robot and we'll just work off the shelf, right? Then you can build applications on top. Like, okay, I want to use the robot for taking a selfie or security inspection. That's the second part. Right. Sure. But let's say now you go to a, a different task where the robot is, I don't know, like assembling a GPU, uh, on a, on a conveyor bed. Now it's a super different task compared to what people generally do. Even humans need training.

15:14Hmm. So in that scenario, what we do is on the, uh, on that robot, we may collect data for a few days. Okay. Either do that. Or if, or if you already have the assets, then we'll get it in simulation. Either way, then we use the data and we post train the model. And then that model takes over and it turns on the robot directly. Okay. So in this case, now what you have done, you have bridged the gap between what you saw before to a very different task by adding data from the, uh, actual task.

15:45So it's called domain specific data. Right. Now, as you deploy more and more of these robots, imagine you are getting a fleet of specialists, which all came from a generalist. Right. So it's very much like, you know, when you're in high school, you know, many subjects. Right. Right. I did PhD. I barely know any chemistry physics at this point. Uh, right. Right. Uh, but, but I needed that to get to, to, to get the knowledge right now. So then when you have this specialist,

16:16then the data can pull back from all of them and come to the same brain behind the scene, which is not how, what happens in humans, but we can do it in a computer. And now this happens. Now, when you have a next task to go to, you may need, you will need less data for the next task. Now this act as a, this is what we call in other words, a data flywheel. Right. Like you may have heard this term for self-driving like humans drive cars. So this data flywheel, now we orchestrate this across a vertical. So you start with factories, they act as data flywheel for semi-structured

16:48scenarios like hospitals, uh, grocery stores. Uh, I don't know, like hotels, you did a flywheel from there helps you get to the ultimate challenge, which is like homes. Right. Consumer robots. So this is basically how we are orchestrating the, uh, so self-sustaining data flywheel loop from every development. And this is why you can probably understand now why do we have omni-bodied brain? Because you want to take benefit of every single data point and use it for the next complex task. Right. And does the same concept apply to different form factors?

17:22Yeah. I mean, on, on factory, it's a robotic arm in home, probably some humanoid or some other form factor for security and inspection with dog, like robot delivery, a different form factor. So across all factors. So I want to ask you guys a little bit about how you're using Nvidia technology, um, and specifically around synthetic data and simulation, as you mentioned, but really just kind of open-ended. How are you, what Nvidia stuff are you using and how does it fit in? I mean, so, uh, our company is two and a half year old. Uh, uh, but we, I have been working

17:53personally with Nvidia, I think since, uh, 2018, uh, like, uh, not at Nvidia working with them. Uh, like, uh, so there is this whole, the suite of simulation, like Isaacson back in the day, there was physics and Isaac Jim. So we use that, the physics component of that to really create these gazillion scenarios on which we can try and practice like what Abhinah was describing, practicing and learning. So that's, that we are basically the OG user, uh, and, and we are now

18:26working with Nvidia on like Newton, uh, uh, as well. And in fact, we are co-developing better physics solvers. Oh, great. Yeah. Probably will open source them, uh, together. That's one, one, uh, collaboration on simulation side. Second side is the video models, like, uh, the Cosmos and other models. Uh, so we use them to data augmentation, like every data point, you can get that and you can create multiple variations with these generative AI models. So we leverage, we partner on that front. And I think the biggest of all is this, the whole compute platform.

18:59Sure. Yeah. Because robots are the next gen, next generation device. Right. And, uh, uh, this solution that worked for LLMs of big GPUs enough in like, uh, servers, it will look very different for a robot because robot doesn't have time to connect to a server if it's falling. Right. Right. Right. React immediately. Yeah. So on device edge compute, this is where we are partnering as well. Excellent. So when you're, when you're testing Omnibrain, when you're maybe when you're using it with a new partner or, you know, developing a new feature, do you have kind of a go-to test case, a go-to

19:31scenario that you put it through or, you know, walk us through what's, what that's like kind of

Testing and Deploying Omnibrain

19:36testing something before you're ready to deploy it. Yeah. I think that's a great question. I mean, although this is also very hard, uh, because that's a problem is something general purpose, right? Yeah. Yeah. And that's what Deepak was talking about a general purpose brain. Now, if you are fine tuning it for something specialized, like I'm bringing a special brain, should it forget the general part of it is, does it matter general part of it or not? It probably does not matter, but then it matters if there was a corner case that was coming in and so on. Right. Right. So those are the kinds of things that matter. So this is why we have been trying to develop a very

20:07specific strategy of like testing these out. So the first thing, of course, we have to test out is on the task itself. Right. Let's say we are putting, let's take the example of the GPU that we have been working with Nvidia as well as a partner as well, like the GPU, like putting a bus bar or on a, uh, GPU rack, uh, on a server. Right. Now there, there are two requirements. First, it has to be put properly. So that's the accuracy part of it. And then how many, how much time does it take you to put, if it takes you one day to put one bus bar, that's not good enough for any deployment and so on. So our testing has these KPIs that we first test

20:39on. These are the task driven KPIs that we are trying to match and so on. But just doing KPIs is not sufficient because that, that is where the whole idea that 90% is done through KPIs or 95% is done through KPIs, but the rest of the 5% is also what matters. And that's where we go and test for generalization. We say, okay, what if someone left a box here or what if somehow the lights were completely off or like we change these conditions and we have these set of conditions that we want to test in. Like even if these things happen, the robot will either continue to work,

21:12but still be safe. Safety is the third aspect of it as well. Like in all these conditions, we have to ensure that the robot is safe and it's not doing any unexpected behavior and so on. So we basically have this whole pipeline where we first start from task metrics, then generalization metrics. Like if things go wrong, I mean, this is something which you're not expecting, but you still want your robot to be robust to those kinds of things. And you, and we have like a whole list that we develop before we deploy that, okay, these are the things that we want to test on when it comes to generalization. And last is the safety that in no, no scenarios

21:47that you should, you should break the safety violations and so on. So we bought something called safety guardrails also before the deployments that ensures that let's say somehow, somehow someone broke the wire or some and cut the camera wire because now the robot is blind, it doesn't see anything. So that's a safety metric that we need to make sure that now the guardrails come in and say, okay, if I'm not seeing a camera, either I should stop or at least I should not cross the boundaries that I have been given by those things. So these are all the things that you have to test for. Again, the problem with the physical world is that it's not like an overnight sensation that

22:25you can become, you put it on a web page and now everyone can access it and so on. We have to go through very rigorous tests before we can put anything online for deployment. Absolutely. So this is one of my favorite questions I always ask as we start to wrap up. What do you think the future of robotics looks like? And, you know, we try to put a timeframe when next year, next two years, things are moving so quickly these days. And particularly as you were talking about with physical AI, you know, the embodiment of AI is really, you know, this year in particular, I think we're seeing so much more of it. But how do you

23:00see robotics developing in, you know, the next few years, five years, whatever the right timeframe is?

23:08I think in the longer timeline, we will be able to automate every single action that humans can take in the physical world, right? Because we are following the approach, which is very similar to how this actually thing things happened in nature. Now the timeline, and in some sense, the longer you go, the more you realize that this is the way to achieve general intelligence. Like currently, what we have so far, all the results in language models,

23:38vision models, it is all what people call digital intelligence. But digital world, if you think about this, it's not more than 50 years old. It's a good point. Yeah. But humans not intelligent before that. Right. So this is this is the longer term vision, right? Now, how does this orchestrate? Well, in our opinion, like, we, you will start to see already things getting automated with these kind of models in a very short horizon, but high complex repeatable, maybe less variable scenarios

24:12first. So it's like what we call unstructured, semi-structured, like industrial task warehouses, they act as a stepping stone, I was saying earlier, to get to more unstructured or semi-structured scenarios, more semi-structured scenarios. This is a spectrum structure is like everything is mapped, like a microwave. Inside microwave, you don't really care, you don't put your hand minutes running, it's a completely separate system, right? Other part is home, which is completely unstructured, it's a spectrum. So in this year itself, we'll start to see deployments in like factory,

24:46warehouse around people that bootstraps the next one, like hospitals, hotels, service industry, that bootstraps the ultimate consumer robots. It's very hard to break the timeline for the ultimate home robots, but you will start to see robots for sure. And you're already seeing that happening in this year or in the next couple of years. I think in the longer run, we all agree that robots are going to be everywhere, doing every task. And I think everyone agrees. And so shorter term also,

25:17we are like, at least in the company, we are all in agreement that this year, we are going to have like the structured places like factories and warehouses being more and more automated, like the penetration will start to happen by the end of this year, more and more penetration. And it's a middle which is unclear. And that's where we always have a betting pool inside a company also like gelato bets and all these kinds of bets that we keep going on. Then when will these things come into play? Everyone has a different view. Like some people believe that home robots might still come in two, three years, but then some people are arguing that two, three years is still

25:51very hard. I mean, we have to be honest and we have to say, okay, like the kind of uncertainty that can happen in the real world is very, very high. And while you're seeing so much hardware in humanoid space also, are these hardware reliable to be even put in homes today? Like no one has put them because safety again is a big issue. Like when you are putting them in home, what if it falls and there's a child around and something like that, right? So we have all these kind of within the company, all these pools going on and so on. And I think both of us are kind of like agree on the

26:24short term and the long term, but it's middle, no one knows. And we are just figuring it out. Okay. We are playing it as long. The interesting part is it's very surprising how it's playing out. I mean, because I mean, from the AI perspective, right, when I was doing my PhD in 2008, would have never guessed where we are in AI. And it's actually continues to surprise even more and more. Like if you ask me three years ago, where would we be today? That also is very unsurprising. And so the progress of compute and the hardware costs coming down has just made this all so surprising

27:00that I would say even the experts like us who have been working in this for 20 years are scared to say anything online. Probably you know this thing, right? Like this is a quote. I'm sure I'm not remembering from whom, but probably Bill Gates mentioned it somewhere. Humans are extremely optimistic in the short term and pessimistic in the long term. I think this applies. This is like a real world paradox. So my million dollar question is, when am I going to have a robot that can fold my laundry? That's, that's the task I want. Well, the thing is, you can have that robot this year,

27:31but if it does just that corner, you have to bring it close. You have to bring it like, would you really want it? That's the whole point, I think. No, absolutely. But if you can do the same thing and, and it's doing something maybe more complex in a factory where you have to run lights out every day, then would you want it? Of course, people are in line for that. So it's just the same thing, but different perspective. No, absolutely. And so what's next for skilled? What are you guys working on now? Are there new, um, new areas you're exploring on the technical side, uh, new industries

28:05or, you know, business avenues that you're breaking into? What's the company roadmap look like? One thing like, uh, in this, depending on when it gets released in the, in the, in the, in the, in these couple of months, we have been ultra focused on how do we take this general model and convert it into a specialized systems, which can be deployed at scale very quickly, right? Like get a new system up and running in a couple of days with a small amount of fine tuning and use that strategy to

28:35scale to as many scenarios as possible. And the reason behind that is to really get started on this general data flywheel, right? Flywheel takes time to set up, takes time to get momentum. And if these things are to happen in the timeline, we want them to happen. We have to start now. And this is made our one of our main focus, not that not saying that technologically we are, uh, there like everything is solved, but this is a big deployment in robotics is a technical challenge. Unlike language or other

29:09areas where you do, if you build the thing, it will get deployed because people will use it. I'll figure how to use it. But here deployment is in itself is a big, uh, uh, technical challenge. And how do you orchestrate that at scale? It has not been done before. Uh, so this is what we are focusing on a lot. It's amazing stuff. And, uh, as you know, to sort of paraphrase you, it's, um, it's not going to slow down. It's only going to get more and more amazing, at least in the short term, right? So, um, who knows what the long-term has to bring, but just fascinating stuff. Best of luck to both of you. And again, Deepak and Avanov,

29:40thank you so much for taking the time to join the podcast. Thank you so much for having us.