When I was at the Nvidia headquarters, I've heard about a robot that was going around handing out popcorn to people. I haven't seen it, but is that true? Yeah. Yeah. Too much pressure on the knee joints, I guess when kicking the ball. And yeah, and I was thinking like We're not 16 anymore. The less friction the better. Less friction the better. Yeah. The The goal is to get more work done. If we had another 1 million X in the next 10 years, what do you think could we do that we cannot do now? That's a great question. All right, two of the hardest questions. One. And you got me there. Good to see you, Spencer. Good to see you, Karoly. What did you announce today? Uh so today we announced a few things. Uh first we've got Isaac Lab with Newton updates. And one of the the best things about this update is that not only is it going to help with uh developers really being able to scale out, but it also enables them to uh fully take advantage of Newton. This is one of those things we've we've really been looking forward to. Um Newton as a physics framework came out uh last year as you know. And integrating it into Isaac Lab has been uh a bit of a step-by-step process. And the reason is because we're moving off of PhysX and and over to Newton. So uh these improvements are really going to bring some developers quality of life uh improvements. So we're pretty excited for that. The second is we have some DGX Station blueprints. And these are going to give people a really good place to start for um both training up the uh various policies as well as being able to leverage these DGX Stations, which are just fantastic workstations to have at your desk. Um so if you have one, I I think you'll have a lot of fun. And the third is Agile One SN. This is a reference platform that we built alongside with Agile. So we we worked with their fantastic Agile One platform. Uh and we've worked to ensure that we have everything from thorough integration all the way to optimizing all the the firmware and all the software stack on top. So that way when you open it out of the box, you already have Groot, it's ready to go for developers to get to just get off the ground. It's one of the the hardest things that we find with development on on these platforms is when you get things like the hands. So, with this one, it ships with the Uchi hands. Bringing the hands onto the robot, as well as just getting the software stack up can take months and you know, weeks depending on how how familiar you are with these platforms. And so, just providing it with with already quality assured, already something in that we're proud to ship, and something that we use ourself, um just gets developers getting going on on their actual workloads. We don't want them to have to fuss around with something that we can just solve for them if we can. The less friction, the better. Less friction, the better. Yeah. The the goal is to get more work done. So, we've had LLMs for years and years now, but physical AI robots, not so much. Why is physical AI so much harder than LLMs? Well, let's step back for a moment. LLMs had the advantage that humans have been writing down our history for as long as as we could come up with different modalities, whether it's writing or taking photos, or just recording just like this. And so, having natural language and just having language in general, we already had the basis for it. We already had a ton of content. So, training up these models really became an exercise of how do you take all of this unstructured data and how do you start extracting features, which then become feature engineering, and how are you training up these models to be more and more proficient as they go along. And then, we started adding different modalities to it, vision and audio and things like that. But for physical AI, you don't have any data set that already encapsulates all of the physics of the world, how you're interacting with the world. We talk about how you might act in the world, and so the actions, and we have the semantics for that, you know. We're very good at having VLMs be able to describe how people are how people, objects, agents are actually interacting with the world, but once they interact with the world, you don't have that physical data, and that's actually the most important part. It's the critical part. And the reason is because when you're working with things like robots, they actually have to manipulate the world and the world in turn manipulates back, right? In physics, you're going to call that coupling. If you have bidirectional coupling, when I make some when I make a an impact on an object, that object might push back, right? It's going to be if I have a soft object on a rigid body, there's going to be deformations on the soft on my my hand, which is a soft object on this rigid body. Um but that's really difficult. So, one, you have to have the physics solvers that can basically um emulate the reality the the physics of reality in such a way that we can then start applying it into the training of the these robotic policies. On the other side, you still have to have all of this data that's capturing what these dynamics are really supposed to be like. So, whether or not it's going to be manipulation interaction data, if it's going to be motion data, um even trajectories of, you know, we watch how robots um today we have to do imitation learning to start because it gives us somewhere, uh, it gives us a set of actions that you can then build off of. You can train robot policies that are more uh robust after taking imitation, you know, imitating human behavior. And so, what we'll do is we'll take teleop. You take a human and they're going to operate a robot, and because they're operating the robot, they can capture all the data on top of that embodiment. Now, over the last year, we've really started seeing improvements where you don't need to you don't necessarily have to only teleop. Teleop still has the same limitation that we have with humans. Anyone that thinks that teleop is easy has is obviously never teleoperated a robot. It's fundamentally different than you would expect because there is no force feedback for a lot of these. And because there's no force feedback, imagine numbing your hands. That's how you the brain is wired that you you're looking for that feedback and you stop when you feel something. it it stop when you feel something. Sometimes you're actually, you know, you're teleopping and you have the robot here and the person's here. And so, you actually have to kind of project in your head what the robot is seeing as you're doing this from the side, and that's just wildly difficult. So, trying to assemble um simple objects all of a sudden becomes a huge undertaking. And so, what we've seen over the over the last year is you can finally start taking human data, Right? You have egocentric data. I'm just going to watch from a camera what I'm doing with my hands, and that gives me some manipulation data. I'm going to take some data of the human body from the side, and I'm going to extract from that the human pose and the human object interaction. And from there, I can start getting data uh for how I can train these policies. Um so, it's it's really um it's really a challenge for us to not only define what is good data along with how to capture the data. And then on top of that, it's how best to use the data. And then really at the end, it becomes this co-training problem. I only have a limited amount of real data because it takes me so long and it costs, you know, X amount of money to get the environment. So, on the other side, we want to be able to use simulation and simulated data. While you can get vast amounts of it, it's not necessarily going to be extremely high quality compared to real data because we're still trying to figure out how do you do feature extraction, right? How are How do you identify what features of simulated data are really going to impact the model performance when you when you um uh pair it with real data. And so, there is this, you know, law of large numbers that we're that we're using basically for simulated data. Um and so, co-training them together, meaning you take real data and simulated data together as you're training these policies, um becomes a bit of an art. And so, physical AI, the reason it's so hard is because when you put all these problems together, and then you need to multiply it by n number of environments. We're not just trying to simulate this environment, right? We need to go and train a robot um or an agent to be able to operate in almost any environment. And right now, we're talking about terrestrial environments. Uh warehouses, you know, houses, uh data centers, even on the road for autonomous vehicles, in the sky for for um you know, aerial like they have these wonderful delivery drones, you know. Um it's great when you can uh you can order your lunch and it just comes to you in a park. But then we have to start going places where it's just infeasible for humans. It's just dangerous. We're going to go places deep down uh in the ocean. You need to go places deep into the earth. And all of these are are you know, extremely data sparse. Because how much data do we really have for all of that? So, it's um yeah, it's a multi-dimensional problem and a lot of it is is we're trying to solve it between how do we capture as much as we can and then how do we find the best ways to simulate that reality as best as we can. That was awesome. That was awesome. I've heard about so much like like it it sounds to me like there is some sort of maybe an ideal data diet for for training physical AI. Like like LLMs you have the textual data and we have the the multimodal image this and that. But for this kind of thing, you have like you said this teleop information, you have traditional handcrafted simulators really close to my heart. And you also have the neural world like like neural simulations, world models. So, how do you build the ideal diet out of that? So, what is what is each of them good at? What are the good proportions? because I I I bet that if you don't have the right proportion in the training data, then then you get something that doesn't generalize well. So, how it sounds devilishly difficult. So, how do you do that? Yeah, that's that's a really wonderful question because you have it is devilishly difficult. The it is um if anything the nuance of of data mixtures really is is kind of the right now I would say it's it's the art in of itself. So, let's take let's just kind of build our way up to up to the answer. The first thing that we can take advantage of is base VLMs. Right? We already have these wonderful VLMs, these wonderful world models that essentially are trained off of a lot of world data that it's already seen. Right? It already knows what it's looking at in the world. However, when you start working in with the neural simulators, basically, how do we take all of this data to train up a world model and then use the world model to simulate reality. What you start finding is that it starts hallucinating things like physics. And it hallucinates things like physics because we've trained it on on a ton of data um is not necessarily physically grounded. It's just observed. So, it's these learned physics. But, what you really want is to be able to ground it uh with high-fidelity physics for the types of things that matter. And a lot of things that matter are going to be, you know, the manipulation bits. If you're trying to walk through the world, what you're looking for is temporal consistency. How do I make sure that as I walk through a world, a simulated world, that it retains coherency, so that way it still retain you know, I turn left, and when I turn back, the world is still the same. And the other And the other hand, when you start to manipulate the world, the physics matter. So, if you imagine holding a ball, and you know, last year, I think the the models have gotten just amazing these days. But, last year, you remember these orange balls, you pick one up, and they drop it, and it just multiplies into four oranges. Mhm. And it it hallucinates not only the physics, but it has hard with, you know, object consistency. But, what we can do now is we can take these world models that we're now leveraging as neural simulators that give you great diversity, and give you great variety for all of the types of objects observations that you may have. And we're training them to be multimodal. So, can they uh can they output actions? Can they output uh touch? Right? How do you output tactile feedback, force feedback, acoustics, all these types of things? On the other hand, I can also use conventional simulators to retain that physics grounding. Because with conventional simulation hallucinate. It doesn't hallucinate, but it's extremely tedious to build an environment. It's extremely hard to create diversity because it relies on on human actors. Now, with uh the with uh the the recent advancements in in these uh in these models, especially things like Astra and and Opus 5.5, what we're starting to see is a lot of the challenges, the bottlenecks that we had before where a human would be the one that has to build up these scenes, has started to become, you know, relatively negligible. But, it What it means is now we can have a lot of conventional simulated data that we can then augment with these neural simulations. So, you can do a few things. Um you could use neural simulation for things like automatic domain randomization. So, one one way to think of this is as you're training a policy, a robot may be inside of the simulator world and it has a visual sensor. And this visual sensor is perceiving the world, but it's just going to look like a bare simulation because all you care about really is the physics. So, you're trying to randomize across an envelope of physics. That's really the goal for uh for you know, manipulative robots. You're not trying to get exactly the same physics that you have with this table. What you're trying to do is approximate across um uh uh across an envelope, you know, what are the the typical distribution of physical properties of this? That way, as it trains within this, you know, a few thousand worlds, a few thousand iterations, and you know, we have all these different clones of this one robot, it's basically learning across what every type of table may be like. But visually, it may not be very realistic because all you're worried about is physics, and and, you know, the main bottleneck that we run into is rendering pixels. But what if you could take that simple rendering, that simple observation, and you pass it over to a world model that then starts domain randomizing across things like materials and lighting and deformations and environment types, and then feed that back into the policy? So, the policy is actually learning both the physics side of it, you know, using conventional physics, but it's getting the observations coming from a neural simulator. And so, these are the types of ways that we can, you know, create these uh hybridized training environments. And it's just the start of it. So, we're we're pretty excited about this. Um, but we do need to leverage conventional and neural simulation together. And a way to think of it is they're they're siblings. So, a neural simulator, these world models, they're only as good as the data you give it, but there's not enough data in the world for it because of all this this physical data that we just talked about. So, you don't have all this physics-grounded data. And the better we get at, um, at generating physics data in conventional simulation and capturing of the real world, you can actually start building up the data sets that these neural simulators, such as Cosmos, are able to sim, you know, they're able to use, uh, they're able to simulate the world much more approximately to what we actually see. But, it's going to be dynamic by dynamic. So, take a subset of the dynamics that you're trying to model, whether it's, uh, you know, we'll have linear deformables. Things like ethernet cables are not the same as threads on on your shirt; they're not the same as spaghetti, which, you know, spaghetti is not the same as some other things that, that look like, uh, that look like wires like rope. So, trying to figure out how do you actually model each of these types of dynamics, and that's just one type of dynamic. You have cloth dynamics, you've got um, you know, uh, things like particle physics, like these all these things that you need to to model to get to the whole real world. You're trying to get the entirety um, of the real world. And so, it's going to be a bit of a conventional simulation and real world data, and they reinforce these neural simulates, these neural simulators then get to learn the physics based off of grounded physics. Now, they're doing better at simulation, and then we can start moving on to different you know, a different subset of of uh, dynamics to model. So, um, it'll be step-by-step, but we're we're making great progress, and these agents are helping us make even further progress much quicker. So, I'm not sure if that answered your question. Yeah, no, that answered 10 other questions, too. But, I I really love that, because essentially what is what is happening is that is that you imagine a robot that doesn't just learn in this world or a simulated world, but 10,000 other worlds, too. Like, what if we were on Mars? What if we were underwater? You know, what if hamburgers flew everywhere? That that that would be great. And and and neural networks like to memorize things if if you let them. And if you don't have enough training data, it it just wants to memorize. Yeah, yeah, and and over fitting and all all kinds of terrible things happen. But, but if if you have this kind of training data, you cannot memorize 10,000 different worlds. You can only understand. And and that's the only way that that it works. So, I I love that. I think that's that's really cool. Now, there is this uh, paper from the robotics lab called Sonic. And you you can upload it into a robot, and it will be able to walk and and do useful things without falling even crawling and dancing works for different kinds of body morphologies. That is insane and all this in 42 million parameters. That's like a grain of sand today like it's so small and it makes decisions in real time. Also open weights and free for all of us forever. So so first of all thank you for that. That's that's awesome and the other thing is how do you cram so much intelligence into such a small AI system? The best way to think of it is let's go back to the subset analogy right if you're trying to model a subset of things what Sonic is really working on is whole body control. It's not working on every part of the system right so you can actually have these smaller models that are able to have such great such great diversity and and capabilities. Now the challenge with Sonic and we're always making much more improvements but we you know we have to acknowledge that the challenge is that it's extremely data hungry. You need to have a lot of demonstration data for Sonic to work really well, but Sonic, once it works very well, you can apply it along with, you know, Groot or another another VLA of your choice to do the high level reasoning, and that gives you the ability to have whole body control along with the ability to do long horizon reasoning across tasks and motions and things like that. I think the the most important thing for us to to remember is Sonic is that first step right it basically proves that you can have whole body tracking and you're taking all of this data of how humans move and then compressing it down into this model that allows you to to give you whole body control. Now for those that aren't super familiar with whole body control one way to think of this is um when you're told to pick up a box a heavy box we're all told to lift with your knees. All right don't bend with your hips. The the thing that you're going to do is you actually have to articulate your entire body in order to reach that object. And the same thing happens for a lot of manipulative tasks, which is why whole body control is important. When VLAs first started, it was actually separating the lower body and the whole body or the the the upper body. So, basically, it couldn't walk and drink coffee at the same time. Right? And the reason that this isn't very, this isn't very useful, is because of the fact that you have to actually move up to static position and then you have to start manipulating; but the moment that you need to start reaching across, you're not able to articulate your body, cuz that's a whole body control maneuver. And so, that's actually what what these are trying to solve for. And so, these are the inherent lower lower level lower level models. And so, this could be you know, system zero. And then system one and system two are the VLA and and of course, as we move on to world action models, the nomenclature might change. I got my 10,000th reason why physical AI is so hard. Yeah, everything's Everything's just a little difficult. need you need system zero. So, you need like incredibly quick, rapid decision-making because like, do I take a step or how do I stay upright or something? You cannot just give it to a big LLM and it says, "Okay, I'll be back in an hour." yeah. You need to have reactions. Yeah. And the body there's two types of of ways that you interact with an environment, right? The first is you can you can react to the environment, which is there's some type of stimuli and you're just moving. So, if someone throws a ball, you you very quickly grab it. You're not actually thinking about it. The second is you're you're reasoning about what you're trying to do. And so, that's more of a response. Right? So, I'm looking at something, a challenge. It's long horizon task, and so, I need to respond to the challenge. So, how am I actually going to be leveraging the body? So, whole body control gives you both. It allows you to have reactions, and it also gives you the ability to leverage the the whole body as part of the the reasoning and and the solution itself within that that task. In an interview about a year ago, you talked about neural simulators with world models coming online soon. What's new there? Can you give me an update on that? Accountability. Accountability. I like it. Um Oh, actually, this that's a a wonderful That's a wonderful place to start because I'm super excited about this. Neural simulation, we're starting to use Cosmos for simulation. Just like I mentioned, where you're blending grounded physics simulation, a conventional simulator, like Omniverse. And so, Isaac Sim, Isaac Lab built on Omniverse. Um and Newton, which would have all of the uh it's a framework for allowing you to bring in different solvers and and different AI physics, you know, neural surrogates of of different um dynamics that you're trying to model. On the other On the other side, you've got Cosmos, which is that world model that we can then use for simulation. And we're starting to see it being applied for a few in a few ways. The first way is just like I mentioned before, you can use it for domain randomization, you can use it for data generation, right? So, that's the first first path. The second path is you can use it as a simulator itself. So, if I train a policy, I could use this neural simulator to actually evaluate that policy as well. The limitation of this is, of course, just like I mentioned, if the if the simulator itself is not able to properly simulate the the physics dynamics that you're trying to model against, then, you know, regardless of of how good your policy is, you know, you you're not really quite sure if it's if it's going to succeed in in real. And so, that's where we have to blend neural simulation and and conventional simulation. And as the neural simulators get better, uh we'll be able to, you know, test more and more and evaluate more and more policies. And then, the third one is actually the neural These uh world models are becoming the policy themselves, and that's where you get world action models. This is where we can leverage Cosmos as a backbone um for the the next variants of group. There is something incredible there. So, you said that essentially it can in a simulation run the verification. It can assign a score and and and train maybe a new policy. You can imagine it like um you'll have to have a few, right? The first is you have a policy. And then, the policy is enacting inside of this environment. So, we can use it Cosmos to be this environment. So, you have Groot that's doing something inside of a Cosmos world, and then you're about you're you're observing this, and you're trying to make some distinction as to whether or not it's successfully accomplishing the task that you're you're testing under evaluation. Um you have a reasoning model, another another VLM. This could also be Cosmos, but it's actually watching these, and what it's doing is it's trying to evaluate the metrics or give quantitative metrics based on what it's observing within that simulator. And so, we use uh right now we're we're using Isaac Lab for this, and we're we'll eventually bring in Cosmos as back end as well, but you can think of we call it sensitivity analysis. So, in Isaac Lab Arena, you take a policy, you put it inside of Isaac Lab, and Isaac Lab has a ton of environments. These environments have scenarios. And within those scenarios, you have defined metrics or criteria of success. And as the pri as the policy is enacting inside of this simulator, you're basically getting these metrics out saying is it how good is it and and you know, where is it failing? And that's extremely important. Based on the task, you decompose the task. So, just saying grab the bottle isn't enough. You decompose that into multiple steps, that way you can identify where in that task is it starting to falter, and where is it, you know, performing strongly or where is it not. And then within those subtasks, you're trying to identify based on a bunch of criteria, what are the things that are might be impacting? Is it the things like collision or things like, you know, is it lighting that's affecting it? You know, what is it that that's actually impacting these policies? And so, this becomes a sensitivity analysis. From there, you can start working on failure-based learning, which is just how humans learn. The more you fail and analyze why you failed, the more you can actually retrain yourself to likely succeed the next time. And so, we can use Cosmos as a as a back end a simulator for this as well. And you have your policy going in, and then we can evaluate it. This is is very new area using these neural simulators for it, and we're doing it for conventional simulators today, but you can imagine if you just step forward how this how this plays out. Having a hybrid simulation gives you that diversity and variety and then you have physics grounding da da da da. But then you're able to start evaluating is this policy going to work in real in reality? But it also gives us a little bit extra. As you start evaluating the policy inside of these simulators and then you deploy them and you see what you know how they're actually performing, you can start evaluating the simulator itself, too. This is why we actually see simulation more used for evaluation than it is for core training across a ton of different tasks. And the reason is is twofold. The first is conventional simulators, like we mentioned, are extremely hard to build. And so you can only simulate accurately a certain subset of dynamics. And so right now is mostly rigid bodies. And so we're we're getting now into rigid soft bodies. But when people started, they would do cloth folding in real. And the reason for that is because it's too hard to do that in simulation. Um originally. It's too hard for me, too. It's It's hard for Me, man. Yeah, but these these agents are getting pretty good at it. And and now we have really wonderful solvers that are actually able to simulate these dynamics. So now you have it in simulation. And the question then becomes how much do you trust your simulator? Which is that sim-to-real gap. And that's really what that next step is. You have the ability to have a neural simulator and a conventional simulator as your world. You're trying to test a policy across different environments, across a a bunch of different physics dynamics. And then the the big question that we all have to ask ourselves is, well, if you're in simulation, um how do I know that it's going to transfer to real? Or how confident am I that it's going to transfer to real? And I think that becomes incredible because I I visit robotics labs around the world and almost every single time, that's what I hear. That that in the simulation things work great. And then when they transfer into the real world, they are like in despair. Like nothing works. And and you're you're looking to close exactly that gap. Like that's the hardest part, right? Yeah. We like to say that you're doomed to succeed in simulation. You know, because no matter no matter what, you can always tune something to have it succeed in simulation. And just because you you succeed in simulation, there's no guarantee that it's going to, you know, the just uh transfer zero-shot to the the platform that you're targeting. Now, that's when you're using a generalized simulator. But as you watch people, they take our they take our platform and what they'll do is they'll tune it, and they'll make it extremely explicit for the types of platforms that they're going to be deploying it on. Because for them, the best thing to do is, if you build the whole robot vertically, you know, you're going top to bottom, you're actually understanding all of the different tunings and different, you know, uh, let's say it's the actual, uh, torque envelopes, and you've got all these different parameters that we may not get just off of a spec sheet. And so they're they're able to start testing, okay, I I train this policy in a simulator, I see these things, I go and transfer it to the real robot, I see, you know, this type of difference. Um how am I actually able to start tuning back and forth? And so tuning it gives you the ability um to start closing that sim-to-real gap. But the real question is, how can we do it in such a way where you could bring a new embodiment very quickly uh and train a policy across many embodiments, right? My My hope would be I would love to train a skill across many embodiments at a time, not just have one embodiment train a bunch of skills. And if you're able to do that, that generalized simulator then has to cross that sim-to-real gap quite quickly. And so you have to be able to identify or at least quantify um how accurate is my simulator, how much does it represent reality in in the the realistic system that I'm going to be putting this policy onto. Um and so it's a it's a difficult challenge, but it's that next challenge because now we finally have these physics solvers that are able to simulate the types of dynamics that we need, right? A lot of things right now are going to be in warehousing, industrial environments, data centers. Um these are known areas, and so you have known known set of physics that you need to start accomplishing. And then from there you can start working on that sim-to-real gap and and narrowly closing it. And sim-to-real, um you didn't ask this, but I'll I'll answer it. Um the Thank you. the sim-to-real gap is is actually not one phase. It's a mix of The reason why we do teleop data is because if you can teleop on the robot that you're going to deploy on, then the data you capture is already kinematically mapped to the robot that that's going. So, you don't have to do any kind of transfer learning. You don't have to retarget any of that data. The The difficulty you have with human data is you have to take a human embodiment and our embodiment, while it looks like, you know, these robots may look human, um, the joints are not always going to be in the same place. They may not have the same amount of joints. And so, kinematically it's it's different. So, you have to retarget those to this new embodiment. Um, so, sim-to-real starts with the data collection, goes all the way through data generation, how you train it, how you, you know, train and configure, uh, your your robot model as you train the policy, and then, of course, as you're evaluating, are you evaluating appropriately for, um, for the world that you're you're going to be deploying it on and and and the platform. And then, of course, how you deploy it and how you evaluate it, and then it becomes this loop. And so, sim-to-real is often thought of as this one stage, but it's actually, you know, end to end. Yeah. It's really interesting in the in the Google DeepMind Lab, I remember uh, seeing these these robots that did football, soccer. And they had a problem because they got injured over time. Like like this knee joint that there was too much pressure on the knee joints when kicking the ball. And yeah, and I was thinking like yeah, careful with And they actually had to be explicitly trained to, you know, not stand on one leg for too long and just ways to, you know, ease up on the on the knee pressure a bit. It's it's just like humans. Is that something that you can put in a simulation? So, so, I'm glad you you mentioned that. It's actually something that when you have enough training when you have enough training data and you've actually simulated across enough, right? Because it's one of those envelopes. So, an envelope, um, the the distribution of of actuation, right? Um, perfect wrist motion versus imperfect wrist motion. Or, you know, do you have a a knee that's been locked up or not? These are all things that you can absolutely simulate and and create by condition. But, what we've also seen is that these models over time, they have these emergent behaviors. And so, you'll have partners and you know, uh companies like Skilled AI. They have this wonderful video. They they effectively had a robot dog and then they locked up one of the knees. And by locking up the knee, it actually adapted, right? So, the emergent behavior actually learned to adapt on how to walk uh on three working uh legs and then, you know, how does how does it hobble on the other? And we're starting to see this with more complicated systems as well with these humanoids where an ankle um I was visiting one partner and we they couldn't quite figure out what was wrong with one of their units. And I was like, "Hey, something looks funny." And it turns out that the ankle joint was slightly broken. And the policy itself had already adapted to this. And so, it was it wasn't walking perfectly, but it was, you know, hobbling along. Yeah, it took into Yeah. And so, it it learns because ultimately what you're trying to do is, you know, uh you're you're trying to um train these things to to walk. You're trying to train them to do certain tasks uh regardless of what's happening with the rest. So, it's going to, you know, power through. It's a bit like a human, too, I guess. Um ideally, you don't have to because that also incurs, yeah, stress on the system itself. Now, let's dream a bit. And like, due to software and hardware optimization, we had a 1 million x ASB up in 10 years, which is just just insane. If we had another 1 million x in the next 10 years, what do you think could we do that we cannot do now? That's a great question. One of one of the things I like to think about is when you train with reinforcement learning, each of the when you you know, you've seen the the scene, you have um an environment and you got a bunch of clones of this robot and it's all doing different things, right? They're all trying to do the same task, but they're doing it slightly differently. And they're all learning a bit differently. You can kind of you can just think of that as experience. It's a little bit like life experience. So, for a human, we have we have to go at at 1 x, right? Every moment is a is an experience for us, but we can only step through a moment a second at a time. For these robots, if you're, if you're able to train these policies faster than real time which is what we're, you know, what we're achieving now but what we could do in the future with, you know, a 100 million x speed up, is you can train these policies to do extremely complicated behavior in a matter of seconds, and it, it may have lived for 100 years. All right, so this experience that we're giving it is really amplified with the more the more compute and the more compute efficiency that you have and the ability to take take advantage of these systems. The other thing that enables it will likely enable is when you have a million times speed up over a large system, it's another way of saying you have a million times speed up over a small system. Well, these small systems are going to allow you to adapt at the edge really quickly. One of the things that humans are are, you know, crazy good at is, if I teach you a certain set of skills and then you go into a novel environment, like a, you know, you've never seen this environment before and the task that I give you is not something you're familiar with, you know, I send you to Alaska and say, "Hey, you know, you got to go go on a fishing boat and you got to go learn how to how to carve up the fish, you know, like the fisherman." And you've only done it at home and you, you know, you're like, "Okay, well, I've only done it here and and maybe I just have no idea how You're going to do at edge learning, right? You have to do it out there. There's no there's no going to school, no internet. Um these are things I'm very excited for because what if you could actually do training on the robot itself instead of having to send it to the cloud, right? Right now, when we want to do large-scale batch training, you're sending it up to the cloud. Um as we get further along, not only are robots going to have the ability to do all the fundamental tasks, but they're going to start working on superhuman complicated tasks. And these tasks they could actually learn in situ. So, you would actually be able to have it running these simulations, you know, on on edge. Continuous learning. Continuous learning. You know, we we have to reason in the situation, Mhm. right? When you look at at um I'm not good at Rubik's Cubes, but I've I've seen people that are extremely good at them. Proud Hungarian product. Just saying. It is a, you know, that really changed the world and; and finally, have you seen some of the models are actually able to, you know, solve these things really quite quickly, in seconds? And as a even with agents just streaming it over, that's actually been really fascinating. But the the ability to think about the problem and then to imagine the problem is actually us putting simulation in our head, right? We're actually we're constantly simulating. If you're an athlete, you see them focusing and they're they're trying to imagine and project in their mind, okay, what am I going to do? How am I going to move my body? How am I going to do this? You know, you're in the Olympics and and before they do an event, they're thinking about what they're about to do. That's just simulation in your head, right? But how do we give that to the robot? The ability to simulate, you know, on edge in a sense. So I I think that's what I'm super excited for. X on you could do that too. Yeah. Yeah. Yeah, love that. Love that. When I was at the Nvidia headquarters, I heard about a robot that was going around handing out popcorn to people. I haven't seen it, but is that true? Yeah. Yeah, we have Well, we have them in in the gear lab and uh I have to say they they sure love their popcorn. This thing is going all the time. Um, I think what we're seeing is, uh, robots are now finally at the stage where the policies and the hardware are able to be deployed into environments where you could have it do useful things; and, and I think that handing popcorn to them is actually quite useful because, you know, a hungry engineer is a, is a hungry engineer. I know and I was I was always promised um that I would have uh a robot that could deliver me um popcorn to my cubicle and so I'm I'm pretty excited that we now have bipeds that can go up the stairs because when we had an AMR yeah. Yeah, this is uh something I've been I've been hoping for for a long time, but I'm not on the first floor when I first joined and so a wheeled robot was going to have trouble getting up the stairs. hope. I'd have to carry it up with the popcorn, you know. Um now we now we actually could have this going around and and I we don't have it yet, um but we're we're only a click away, you know, it's not the The the the amazing part of the age that we live in is there was this there was a time a few years ago where you thought wow, okay, it's going to take us 10 years to solve some of these problems. And with the advancements of AI and how quickly they've advanced and you know, over the last quarter. I mean, think of it this way, we didn't have agents in January of this year. And now it it's everywhere. Um and we're not even totally at the end of the year. Um we've now taken a 10-year horizon and just crunched it down into three to five, right? And really the bottleneck really becomes um are we going to have the hardware systems that are capable of enacting these policies? Do you have full body dexterity that can do all the types of things that that we can train it? And then second, do we have enough compute to actually train up the policies to be able to do the complicated tasks that we want? So, it's it's right there, man. We're we're so close. What kind of problems are still there that are easy for humans but difficult for machines? And how do you address them? So, this is you know, we're we're so close but somewhat far. It's Um I forget there's a paradox where things that are easy for humans are hard for machines. And things like um in-hand manipulation can be quite difficult. Things like um assembling LEGOs, you're going to start seeing that these are are tasks that we can start, you know, really accomplishing. But imagine some of the really fascinating ways that humans have learned to use our body to do micro-assembly. These are such hard tasks but they're actually simple to describe. I need you to take the screwdriver and you need to put this here and you got to certain locations but the dexterity required is so complicated for a robot. And that's that's something that we we need to start solving for but it requires a few things. First, you have to have um either real data uh or simulated data or a simulator that's able to actually um that is simulate this this complicated behavior. Second, you have to be able to train up a policy that's able to do all of the priors, you know, can it use a tool? Can it do something within sub-millimeter precision? Can it do all of this? And so you're building your way up a little bit like you're watching a kid grow and you have to do this on, you know, a pretty compressed amount of time. And then the last is do you have the hardware that has that precision? I think that that's somewhat overlooked. We've had automation, high-level precision automation for for decades. And if you look at the robots today, they're able to do gross manipulation and they're starting to get really good at fine-grain manipulation. However, when we think of extremely high precision, you know, 0.003 mm. Um humans can't do that. So our demonstration data is not going to be great for it. Um but we'll train a policy and you need the the hardware that's actually able to do this as well. And so the hardware is starting to come online. That's what we're starting to see is there's now options and there is plenty and there's wonderful ecosystem of of hardware builders and OEMs that are actually building systems that we can now start really, you know, taking this this next step. So I think those are the hard things that are easy for us um but are harder for robots. Um on the other hand, you know, if you ask it to do math or you know, try and generate millions of of uh sample data, uh it's going to be almost impossible for us, but you know, for these these systems it can do it in a minute. I wanted to ask your opinion about this. Um do you feel that like first let's think strictly LLMs. So not even physically I like would AI at some point become good enough? Or do we always want the best? And the question is deceptively difficult because my impression is that yes, at some point it's going to be good enough. Like it's not a nice way to measure things, but let's say that each LLM has an IQ. Like intelligence is a vector quantity and maybe even more complex, but just just for simplicity. And at some point you're going to have an LLM at your fingertips that's instant, that has an IQ of I don't know, 170. And there's a new one that's 220. And at some point you say like who cares? Like any of these will do. So for me it's easy to think that yes, at some point it's going to be good enough. But at the same time I I try to think whenever I ask something like which AI do I use? Do I use the thinking AI or the instant AI? And I I feel that I I reach out to the thinking AI a lot more than I think. Like if I have a contract to be reviewed, like like I understand that the instant is good, but I I kind of want the best for this one. Health questions questions about I don't know medication or something. Like LLMs should be maybe just one data point, like not a doctor, all that. But still if I'm asking something, I know the other one is smart, but I I probably want the best. So what what do you think? How do you find yourself using it? I'll answer it and I can in in in this answer I'll apply it to both LLMs and and physical AI models. When when you have a sufficiently Okay, let's I'll separate into two two buckets. The first is generalized models and the second are specialized models. If you're in a medical field or in our field, right? Um having a specialized model that gets you to, you know, 170 or 200 IQ, whatever we whatever we think is an amazing baseline that gets you across the line for, you know, 80 90%of the problems. I think that kind of good enough is actually what people are going to be using day-to-day. But we are always going to be pushing the frontier on the types of things that we're going to solve for. One of the things that humans are good at is we're we're great at inventing work. You know, we we've invented many means of automation and at no point have we gotten lazier. We've actually just found more ways to make ourselves, you know, push the the frontiers of of automation, we push the frontiers of the things that we can make, the things that we can create. The types of challenges that we want to solve for are going to be unending. You know, we're we're here on Earth, but at some point you want to, you know, go across the stars and you want to explore further. So the baseline intelligence will be great for us to be able to raise everybody up to be able to accomplish, you know, a uh a certain, let's say it's the predominant common or the the general amount of problems that you're going to run into which I think is is perfect. It's like going to get a master's degree is really fantastic for for most people, but getting a PhD is not necessarily for everybody, but it is for a very specific niche. And so that general Especially for our audience. especially for our audience. The fellow scholars. The fellow scholars. Always wanted to say that. And and the general model is once you hit that baseline, I think good is actually going to be what's in your pocket. But that frontier intelligence is what's always going to be having us push forward. So we're always going to have a need for both, but right now we're still just trying to get to the baseline of physical intelligence, and we're really starting to get to the point where these these frontier models are just becoming so excellent in a certain distribution of tasks that most people will be okay with, you know, what was frontier today might be just common tomorrow. But there's also going to be an immense amount of people that are going to require that frontier for for pushing everything else. All right, two of the hardest questions. One. Favorite Two Minute Papers episode. Come on. Ah, you know, I would have come prepared. I would have remembered the paper names. The, you know, I I think that, in general, I love your uh when you go through fluid, fluid and particle physics. Like, the fluid dynamics of parts: one there, they're just visually appealing, but two um it's not my specialty, and so the only way I understand it is actually when I'm watching it at lunch. I would say that those those are one It's always wonderful to see my colleagues'papers um in there. You know, there's especially when we're doing a motion generation and you've done some really wonderful coverage of of our motion generators and and what we can actually do with taking human human motions and applying it to to these avatars and and simulation. And in the future you're going to see how that immediately translates to the physical AI. Um And you got me there. No, that was that was really good. I'm very pleased with that. Um last question. This is the hardest. How do I get you to answer my emails? You know, I found um that at symbol works really well. The uh my I'm learning right now to have my agent actually help me undo uh let's just say I'm on a lot of automated systems and uh Tell your agent that Two Minute Papers is priority one. one. Honestly, I think you should the easiest is you call me. All right. All right, Spencer. I've learned a lot. And and when I go home and I cut this video together, I'm going to watch it, you know, five, six times more and I'm going to learn even more. So, this was amazing. Thank you so much. All right. We're done. Thank you. That was That was so much fun. That was That went by in a flash. Uh tricky tricky last question there. I use Lambda to reproduce AI research papers often in minutes. It's also great to train your own models or fine-tune an existing one. Run inference or text-to-image or video easy peasy. Running a deep seek chatbot or agent super fast, super reliable. Lambda gives you powerful Nvidia GPUs to run your own experiments. I test ideas from the papers I cover and moments later, results. Love it. Seriously, try it out now at lambda.ai/papers.
Generated algorithmically for Search Engine
Indexing.