00:31:52 · Jim Fan
So how it works is you take a pretrained video generation model, and then you can fine tune it on the robot data that you you you collected. And we found that because of all the pretraining on billions of Internet videos, this world model is actually able to learn the constraints, the physical constraints of the robot, all the mechanical kind of parameters of the robot, and then it's able to simulate how the robot will behave given a small amount of post training data. So here, on left and right, it's actually the exact same initial frame, and the only difference is you give it different props and then to a similar different futures. So on the left, it's pick up the apple and place on the pen, and on the right, pick up the can and place on the pen, and the model knows how to follow these languages, and all these pixels are generated. And also, the other good thing is, because it's an end to end learned model, it doesn't care how complex the scene is, and I think that's its strongest advantage over traditional graphics pipelines, where the more complex the scene is, the more effort artists need to put, and also the more compute you need to do ray tracing and so on. So here is a much more complex scene. The video world model doesn't care. It's gonna simulate just as well. And then we're able to run this on a lot more diverse scenes, and you can see, like, all the reflection, right, all the complex reflection on the silverware is able to do that without implementing a ray tracing algorithm explicitly. And then using this pipeline, we can generate lots and lots of synthetic data that we call neural trajectories. And, you know, we can add this as if they're real robot data and then train on those data together, and we'll be able to perform a lot better on the on the robot task. So I would say that work model is still at at its early stage. We have some initial ideas, early ideas to make it work, but a lot more research needs to be done.
00:34:09 · Molly Welch
[00:34:09] S0: Very cool. Thank you for sharing that. And and, again, to see... Amazing to see the demos. You mentioned robotic brains. I wanna zoom out and talk a little more broadly about robotic foundation models. What are the principles to build robotic foundation models, and where are we in the development of robotic foundation models today?
00:34:29 · Jim Fan
[00:34:29] S1: Absolutely. And I think I'm very much following the vibe that brought me you know, here at this moment in the first place. So I mentioned that I was basically choosing hard problems and then selecting for simple and elegant methods because they scale the best with compute and data. And it's the same principle that I'm applying to robotics, and we can summarize that in four words, data maximalist and model minimalist. I mentioned there are many different ways to collect the data. Right? We can collect through tally operation. We can generate data using simulators like Isaac Lab, and then we can also generate data using world models. So the data pipeline can be very complex, but the model should be as clean as possible, as end to end as possible, because in this way, you can leverage all the different data sources without building separate training recipes and loss functions for the models. So it takes, as input, pixels and then directly output the motor control signals that you saw, like the the high dimensional continuous signals. And earlier this year, we open sourced a model called GR00T n one that operationalizes this this approach. So the GR00T n one model, it's got system one and system two. Right? System two is a VOM that does reasoning. System one is actually a diffusion model that renders the actions at more than a 100 hertz. And despite it's got this kind of modular approach, it's actually an end to end fully trained model through backpropagation. So that's how that's how we approach it, and we're we're building an ecosystem, so we open source everything. We open source the training recipe, the model parameters, and if you search for isogrut online, you can download the model and try
00:36:19 · Molly Welch
[00:36:19] S0: it out today. It's a hugely valuable contribution to the ecosystem, so thank you for for sharing and encourage others to check it out as well. We've talked a little bit before, Jim, about the GPT four moment in robotics. Yes. What... What's your prediction? When will we get the GPT four moment in robotics, and what are the research and commercial problems that need to be solved before that happens?
00:36:44 · Jim Fan
[00:36:44] S1: Well, I get asked this question a lot, and thank you so much, Molly, for bringing this up. So I think the community tends to, in general, overestimate the short term and underestimate the long term. I will say I don't know for sure exactly when the GPT-4 moment for robotics will happen, but I know for sure when it will certainly happen. So here... Here's kind of my my reasoning. Right? So this year is 2025, and AlexNet was introduced in 2012. Actually, AlexNet was, you know, held as this milestone, right, huge revolutionary moment. But does the audience know how crappy AlexNet was? It's a horrible model. What AlexNet did is classifying objects, right, like one out of a thousand categories, like cats versus dogs versus airplanes, and its accuracy was only 62.5%, meaning that it confuses cats and dogs and airplanes more than a third of the time. It's completely unusable. Right? Even though it was a revolution, from a methodology point of view, performance was super bad. And then fast forward to 2025, what do we have today? We have LMs that pass the Turing test. We got the latest coding models and the recent models winning gold medals at IMO, and then AlphaFold just won a Nobel Prize, and that's what's happening in 2025. It's crazy what happened in the last twelve years, and then it so happens that technology doesn't progress linearly. It's exponential. So if we add... If we take 2025 as the middle point and we add thirteen years from now, that will be 2038, right, where we're in the middle between AlexNet and 2038, and then PhDs procrastinate, so let's round up. Give them two more years. By 2040, I believe the number of robots, intelligent robots, in the world will be more than the number of iPhones, and I say that with more than 95% certainty that it's gonna happen.
00:38:56 · Molly Welch
[00:38:56] S0: It's a provocative framing, and I want to think shorter term for a moment. When you think about models today and the next two to three years, many folks are thinking about what the next chapter of their journey looks like, whether that's starting a company or joining. Models' capabilities today, which application areas do you think have the right combination of market readiness and technical feasibility to be really tractable in the next two to three years?