00:34:09 · Molly Welch
[00:34:09] S0: Very cool. Thank you for sharing that. And and, again, to see... Amazing to see the demos. You mentioned robotic brains. I wanna zoom out and talk a little more broadly about robotic foundation models. What are the principles to build robotic foundation models, and where are we in the development of robotic foundation models today?
00:34:29 · Jim Fan
[00:34:29] S1: Absolutely. And I think I'm very much following the vibe that brought me you know, here at this moment in the first place. So I mentioned that I was basically choosing hard problems and then selecting for simple and elegant methods because they scale the best with compute and data. And it's the same principle that I'm applying to robotics, and we can summarize that in four words, data maximalist and model minimalist. I mentioned there are many different ways to collect the data. Right? We can collect through tally operation. We can generate data using simulators like Isaac Lab, and then we can also generate data using world models. So the data pipeline can be very complex, but the model should be as clean as possible, as end to end as possible, because in this way, you can leverage all the different data sources without building separate training recipes and loss functions for the models. So it takes, as input, pixels and then directly output the motor control signals that you saw, like the the high dimensional continuous signals. And earlier this year, we open sourced a model called GR00T n one that operationalizes this this approach. So the GR00T n one model, it's got system one and system two. Right? System two is a VOM that does reasoning. System one is actually a diffusion model that renders the actions at more than a 100 hertz. And despite it's got this kind of modular approach, it's actually an end to end fully trained model through backpropagation. So that's how that's how we approach it, and we're we're building an ecosystem, so we open source everything. We open source the training recipe, the model parameters, and if you search for isogrut online, you can download the model and try
00:36:19 · Molly Welch
[00:36:19] S0: it out today. It's a hugely valuable contribution to the ecosystem, so thank you for for sharing and encourage others to check it out as well. We've talked a little bit before, Jim, about the GPT four moment in robotics. Yes. What... What's your prediction? When will we get the GPT four moment in robotics, and what are the research and commercial problems that need to be solved before that happens?
00:36:44 · Jim Fan
[00:36:44] S1: Well, I get asked this question a lot, and thank you so much, Molly, for bringing this up. So I think the community tends to, in general, overestimate the short term and underestimate the long term. I will say I don't know for sure exactly when the GPT-4 moment for robotics will happen, but I know for sure when it will certainly happen. So here... Here's kind of my my reasoning. Right? So this year is 2025, and AlexNet was introduced in 2012. Actually, AlexNet was, you know, held as this milestone, right, huge revolutionary moment. But does the audience know how crappy AlexNet was? It's a horrible model. What AlexNet did is classifying objects, right, like one out of a thousand categories, like cats versus dogs versus airplanes, and its accuracy was only 62.5%, meaning that it confuses cats and dogs and airplanes more than a third of the time. It's completely unusable. Right? Even though it was a revolution, from a methodology point of view, performance was super bad. And then fast forward to 2025, what do we have today? We have LMs that pass the Turing test. We got the latest coding models and the recent models winning gold medals at IMO, and then AlphaFold just won a Nobel Prize, and that's what's happening in 2025. It's crazy what happened in the last twelve years, and then it so happens that technology doesn't progress linearly. It's exponential. So if we add... If we take 2025 as the middle point and we add thirteen years from now, that will be 2038, right, where we're in the middle between AlexNet and 2038, and then PhDs procrastinate, so let's round up. Give them two more years. By 2040, I believe the number of robots, intelligent robots, in the world will be more than the number of iPhones, and I say that with more than 95% certainty that it's gonna happen.
00:38:56 · Molly Welch
[00:38:56] S0: It's a provocative framing, and I want to think shorter term for a moment. When you think about models today and the next two to three years, many folks are thinking about what the next chapter of their journey looks like, whether that's starting a company or joining. Models' capabilities today, which application areas do you think have the right combination of market readiness and technical feasibility to be really tractable in the next two to three years?
00:39:26 · Jim Fan
[00:39:26] S1: That is a that is a great question. And let me also zoom in a little bit on the on the shorter time horizon. And, again, this year is 2025. And if we look back to May 2020, that was when GPT-three was first introduced. And from GPT-three to winning a gold medal at IMO, I think that roughly takes three roughly three iterations. Right? So there is SFT, and then there is ROHF, right, reinforcement learning from human preference, as kind of two big breakthroughs, and then reinforcement learning, like the the OpenAI O one and O three style, like RO for reasoning. So roughly three major model capability breakthroughs carried us from 2020 to 2025. So if we add five more year... Five more years, that will be 2030. So I think in terms of methodology, right, like the the breakthrough in robot foundation model, probably three takes five years will will get us there. So now what do we do in the next five years horizon in terms of applications if we have this moment of GPT-three for robot foundation models? I think the following application areas I'm most excited about. The first is programmable factories. I think factories and industrial use cases are probably the first to be tackled because they are more regular and also more structured and less demanding on safety. You can... You don't need humans to be around robots inside a factory. You can have the factory basically wall off, and then the robots operate themselves, so it doesn't need very strict safety criterion, And also, the manufacturing process is quite well understood with very good specifications, so it doesn't need too much generalization. It does need some generalization, but not too much. And if we have a GPT-three or GPT-four level robot foundation model, then we can quickly reprogram a set of intelligent robots so they can manufacture a new product almost overnight, and you can basically show the robots, let's say, a few demonstrations, and then you can program them. You program by demonstration, by data, instead of programming by hand, and that's why robots are very difficult to to be reprogrammed these days because people do it by hand, and then big factories are really hard to repivot. You need months, if not years, of expertise to reprogram something. But once we have these foundation models, the factories will be a lot more programmable, so I think that's one of the first areas.