00:17:49 · Molly Welch, Jim Fan
[00:17:49] S0: Very cool and incredibly clever, the approach. Remains incredibly clever even as the technology has progressed. So thank you for talking us through it. I... I'm curious. You know, you... MineDojo and and Voyager represent this progression towards agents that learn autonomously or more autonomously through exploration. What do you think are the key remaining bottlenecks for creating truly self improving agents? [00:18:16] S1: I think two things. So there's a popular word called thinking fast and slow from psychology. Basically, it says that all the intelligent beings, like humans, have two systems. So system two is slow and deliberate and does reasoning, and system one is fast, reactive, and intuitive, so the two system approach to agentic AI. I think right now, the system two is progressing quite well. It's not perfect, but I think there are, like, many ways to improve system two, right, like more data, more environment that we can plug it... Plug into the frontier LLMs so they can do reinforcement learning. These days, the frontier LLMs are really bottlenecked by the diversity of the environment, not just the pretraining data. We are kind of in this new era of RL post training, so that's on system two. And system one is... It still remains challenging because computer vision is not soft. It's also much harder to collect data for system one, and I think that's also a good segue to maybe talk about Project GR00T and robotics.
00:18:16 · Jim Fan, Molly Welch
[00:18:16] S1: I think two things. So there's a popular word called thinking fast and slow from psychology. Basically, it says that all the intelligent beings, like humans, have two systems. So system two is slow and deliberate and does reasoning, and system one is fast, reactive, and intuitive, so the two system approach to agentic AI. I think right now, the system two is progressing quite well. It's not perfect, but I think there are, like, many ways to improve system two, right, like more data, more environment that we can plug it... Plug into the frontier LLMs so they can do reinforcement learning. These days, the frontier LLMs are really bottlenecked by the diversity of the environment, not just the pretraining data. We are kind of in this new era of RL post training, so that's on system two. And system one is... It still remains challenging because computer vision is not soft. It's also much harder to collect data for system one, and I think that's also a good segue to maybe talk about Project GR00T and robotics. [00:19:27] S0: It's a perfect segue, which is what I wanna talk about next. So, you're leading physical AI research through the Gear team, Project GR00T for general purpose robotics as you described. What do you think is the grand challenge for robotics? Why is it so hard? Why is system one, which is so easy for us, system two is the hard piece, why is system one so hard in robotics?
00:19:48 · Jim Fan
[00:19:48] S1: That's a great question, Molly. Well, first, to... For the audience who's not familiar, Project GR00T is NVIDIA's moonshot initiative to solve general purpose robot AI. And I think to solve this, first, we need to define a milestone. Right? Like, what is this one challenge? So here's my proposal. Suppose you, you know, on on a Sunday on a Sunday night, you hosted a big hackathon party, and you wrecked a house, right, like pizza everywhere. It's a complete mess. Now you go to work on Monday, and you come back home to a clean house and a candlelit dinner, and you couldn't tell if a human or robot had been there. And I'm calling this the physical Turing test, and it's really a shout out to the actual Turing test, right, like that Turing proposed, and we as a community more or less solved the Turing test. When you're chatting to a chatbot, you can tell. It's very easy to fool a human now, But what about the physical Turing test? I think it's deceptively simple. Right? It sounds so simple, so intuitive, but it's perhaps the next or the last grand challenge in AI. And, Molly, as you hinted, right, like, it's so easy for humans, but, really, we're at the sake of the more of its paradox where since... Things that are easy for humans are actually incredibly hard for the machine. And why is that? I think the main problem is the data problem. You know, I I listened to Ilya talk, and he famously said that ROMs pre training is hitting a wall because we're running out of data. Right? Like, the Internet, we only have one Internet, and the Internet is the fossil fuel of AI. But I will say that you guys, LLM researchers, are so spoiled because you... At least you have the Internet to train on, and we roboticists, unfortunately, we don't even have that. So let me me share a screen. I would love to show the audience. Robotics is a very visual thing. I think one video is worth 10,000 words, So let me share. And, Molly, tell me if you can see it.
00:22:07 · Molly Welch
[00:22:07] S0: We can see it. We see the grocery in NVIDIA HQ.
00:22:11 · Jim Fan
[00:22:11] S1: Grocery in NVIDIA HQ. So this is, like, a typical data episode that we collect for for robotics, and this is the input to the robot. And, actually, this is the output. That's what the robot joint control or the or the motor control looks like. It's a high degree of freedom, high dimensional continuous signal, and unfortunately, you cannot scrape this signal from Wikipedia. So we have to collect the data in in some ways. So what we do is through a process that we call teleoperation. So here, this researcher is wearing a VR device where it's streaming her hand pose in real time to the actual robot. So the robot is physically mirroring her movements. And then in this way, we can control the robot to do a lot of tasks, like here to pour honey over over a toast, and the researcher basically sees the the robot actions, right, through this egocentric camera. So unfortunately, you know, this process is very manual. Right? You can see it's very tedious, and it hits the physical limit of twenty four hours per robot per day. But actually that's the upper bound. The robots are fragile. We babysit them all the time. They throw a tantrum, and if the robot god is merciful on that day, we may collect up to four hours of high quality data, Right? So it never hits twenty four hours. So really, it's it's a kind of human fuel that we are collecting for the for the robot, and then that's compared to the fossil fuel, right, that that Ilya said, you know, all the ROMs are pretraining on. So the question is, how can we get the nuclear fuel for robotics? Right? What is the clean energy? What's a way to generate almost infinite amount of data, right, to feed our our pipelines. So so here is a is a quick pander... Pyramid. Right? The real data is on top. It's the smallest amount. The web data is huge, and then synthetic data, right, will be the will be the future. So it's infinite in principle, but it's limited by PhD brain cycles because we need to come up with novel methods to actually generate the data, and also by the compute. So the more you buy, the more you save. This message has been approved by my boss. And I want to show some new research that we're doing on synthetic data. So, Molly, at the beginning, you mentioned Eureka. Yeah, I want to show this.
00:24:46 · Jim Fan
So it's a technique that allows us to train a high degree of freedom, robot hands, to do pen spinning at human level, and I want to admit that I am a subpar human. I can never really do pen spinning, and I gave up a long time ago in my childhood. So it's really glad to see my AI avenging my poor skills. Right? How do we get the data to train this? The answer is there is actually no supervised dataset. It's all done by reinforcement learning in massively parallel simulations on the GPU because, luckily, all the physical equations are mostly just matrix modifications. So you can run this on CUDA, simulate reality at 10,000 times faster than real time. And then you can transfer this to the real world. And how do we do that? Through a process called domain randomization, where for the 10,000 different environments, you can vary the gravity, friction, and weights, and then you follow the simulation principle. So suppose an AI model is able to master a million realities with different physical parameters, then it's highly likely that it's gonna zero shot the one million and first reality, which happens to our own physical world. And just to kind of show some quick videos, we can run like a robot parking simulation, and then it transfers to real world. We can control a four finger hand to manipulate a cube, and this one is most interesting. We have a robot dog walking on a yoga ball and trying to stay balanced, and actually, the yoga ball is very hard to simulate. It's bouncy, right? It's soft, but actually, we don't even simulate that. It's just a hard sphere that we do, but we do a lot of domain randomization, and then, you know, we're roboticists, we're weird, so we just transfer and run it in the streets. Yeah. It's an actual robot dog running a yoga ball in the streets. You know, sometimes I feel like I'm the director of a Black Mirror episode. Yeah. It's it's just so weird. Actually, my friend, after seeing this video, he told me that he tried a yoga ball on his own dog, and it couldn't do it. So in a sense, we have achieved super dog performance. And of course, we can scale this to a lot more complex embodiment, like an army of humanoid in sim, and these humanoids are going through ten years worth of intense training in only two hours of simulation time. And then with that much data, we can train a 3,000,000 parameter neural network. No.