00:13:31 · Jim Fan
[00:13:31] S1: It's it's all for AI. It's all for pushing the frontier of AI. So I think what was fun about that project was Minecraft is this open ended environment. It's also a simulation, meaning that you can basically generate infinite data of almost infinite diversity in Minecraft. Minecraft. I've seen people doing crazy things inside this sandbox game, right, like building the Hogwarts Castle brick by brick or even building a functioning CPU circuit using those, like, fundamental building blocks in Minecraft that actually makes the game Turing complete. So this game has infinite diversity, yet humans played it very well, but for AI, it was lagging way behind. It couldn't even do some basic things. So at a time when we started the Voyager project, that coincides with the introduction of g d four. That was the first foundation model that can do coding extremely well. So how we apply g d four to Minecraft is that we give it an API so that g d four can read the instructions of the Minecraft API and then write code, and then we execute that code in the Minecraft simulator to drive the agent to mine blocks, right, to explore, to travel, right, to dig, to craft new items. So, basically, we we we turned the game into a text environment, and it communicates by code. And then we found that g d four is not only good at in context learning, it's also very good at a capability called self reflection, so it can run through a box. No one can write perfect program at the beginning, like in one shot, not even experienced programmers. You've got to iterate, and that's what we observed with gmd4. It writes code, tests it, and the code breaks, it sees bug, it debugs, it tries again, and once it writes a good code, a good piece of code, it would commit to a memory, so we call this kind of an on the fly code repo. So g m d four will save some of the the code it wrote to this repo, so later it can retrieve it if it sees a similar situation. And in this way, we we couldn't actually fine tune the parameters of a GD four, but it still learns over time because it's committing code, retrieving code, and doing trial and error, and it accumulates the context as it goes. So it's a kind of in context lifelong learning.
00:15:59 · Jim Fan
And I find that idea really fascinating because at that time, people were really trying to experiment and see what are the capabilities of these coding frontier models, and Voyager provided one way to build a very complex agentic system on top. And then two years later, these days, the the agents are a lot more powerful. Like, self reflection, people don't even talk about it. It's kind of a given. Right? Like, all the cloud and cursor of the world, they can do debugging super well. They can go on, like, a thirty minute session to to implement a pull request, and then it just... It goes on autopilot, and it it typically does the job pretty well. And it's also able to call all kinds of tools, web browsing, connecting to MCP. So I think the field has gone way beyond Voyager, way beyond Minecraft, so that was that was really exciting to see. Now what was lacking a bit from Voyager was actually that we didn't solve the motor control problem, and we didn't solve the perception problem. We essentially simplified the problem by turning Minecraft into a set of code API and code execution environment, but later, I have come to understand that really the most difficult part may not necessarily be the reasoning side. I think the field is already solving reasoning quite well. The most difficult part of Embodied Agent is actually the low level control, the part where you interact with the low level physics. In Minecraft, it's the game physics, but in the real world, it's the actual physical reality that we're touching, and that makes the robotics problem a lot harder.
00:17:49 · Molly Welch, Jim Fan
[00:17:49] S0: Very cool and incredibly clever, the approach. Remains incredibly clever even as the technology has progressed. So thank you for talking us through it. I... I'm curious. You know, you... MineDojo and and Voyager represent this progression towards agents that learn autonomously or more autonomously through exploration. What do you think are the key remaining bottlenecks for creating truly self improving agents? [00:18:16] S1: I think two things. So there's a popular word called thinking fast and slow from psychology. Basically, it says that all the intelligent beings, like humans, have two systems. So system two is slow and deliberate and does reasoning, and system one is fast, reactive, and intuitive, so the two system approach to agentic AI. I think right now, the system two is progressing quite well. It's not perfect, but I think there are, like, many ways to improve system two, right, like more data, more environment that we can plug it... Plug into the frontier LLMs so they can do reinforcement learning. These days, the frontier LLMs are really bottlenecked by the diversity of the environment, not just the pretraining data. We are kind of in this new era of RL post training, so that's on system two. And system one is... It still remains challenging because computer vision is not soft. It's also much harder to collect data for system one, and I think that's also a good segue to maybe talk about Project GR00T and robotics.
00:18:16 · Jim Fan, Molly Welch
[00:18:16] S1: I think two things. So there's a popular word called thinking fast and slow from psychology. Basically, it says that all the intelligent beings, like humans, have two systems. So system two is slow and deliberate and does reasoning, and system one is fast, reactive, and intuitive, so the two system approach to agentic AI. I think right now, the system two is progressing quite well. It's not perfect, but I think there are, like, many ways to improve system two, right, like more data, more environment that we can plug it... Plug into the frontier LLMs so they can do reinforcement learning. These days, the frontier LLMs are really bottlenecked by the diversity of the environment, not just the pretraining data. We are kind of in this new era of RL post training, so that's on system two. And system one is... It still remains challenging because computer vision is not soft. It's also much harder to collect data for system one, and I think that's also a good segue to maybe talk about Project GR00T and robotics. [00:19:27] S0: It's a perfect segue, which is what I wanna talk about next. So, you're leading physical AI research through the Gear team, Project GR00T for general purpose robotics as you described. What do you think is the grand challenge for robotics? Why is it so hard? Why is system one, which is so easy for us, system two is the hard piece, why is system one so hard in robotics?
00:19:48 · Jim Fan
[00:19:48] S1: That's a great question, Molly. Well, first, to... For the audience who's not familiar, Project GR00T is NVIDIA's moonshot initiative to solve general purpose robot AI. And I think to solve this, first, we need to define a milestone. Right? Like, what is this one challenge? So here's my proposal. Suppose you, you know, on on a Sunday on a Sunday night, you hosted a big hackathon party, and you wrecked a house, right, like pizza everywhere. It's a complete mess. Now you go to work on Monday, and you come back home to a clean house and a candlelit dinner, and you couldn't tell if a human or robot had been there. And I'm calling this the physical Turing test, and it's really a shout out to the actual Turing test, right, like that Turing proposed, and we as a community more or less solved the Turing test. When you're chatting to a chatbot, you can tell. It's very easy to fool a human now, But what about the physical Turing test? I think it's deceptively simple. Right? It sounds so simple, so intuitive, but it's perhaps the next or the last grand challenge in AI. And, Molly, as you hinted, right, like, it's so easy for humans, but, really, we're at the sake of the more of its paradox where since... Things that are easy for humans are actually incredibly hard for the machine. And why is that? I think the main problem is the data problem. You know, I I listened to Ilya talk, and he famously said that ROMs pre training is hitting a wall because we're running out of data. Right? Like, the Internet, we only have one Internet, and the Internet is the fossil fuel of AI. But I will say that you guys, LLM researchers, are so spoiled because you... At least you have the Internet to train on, and we roboticists, unfortunately, we don't even have that. So let me me share a screen. I would love to show the audience. Robotics is a very visual thing. I think one video is worth 10,000 words, So let me share. And, Molly, tell me if you can see it.
00:22:07 · Molly Welch
[00:22:07] S0: We can see it. We see the grocery in NVIDIA HQ.