00:23:32 · Aditya Agarwal, Sergey Levine
[00:23:32] S1: Correct. [00:23:32] S0: So then we have a bootstrapping problem. Like, how do we get there? [00:23:35] S1: Yep. [00:23:35] S0: Well, a bootstrapping problem is is... That's not like a good problem to have. If you know the asymptotic thing looks good, and you just have to somehow lift yourself up there. Okay. [00:23:42] S1: Do the You're an optimist. You're... This is why you're startup founder. This is great. [00:23:45] S0: Engineering stuff, do the industrial stuff, and you'll get there. And you already know that, like, that place you're gonna get to is a really good place. [00:23:52] S1: So, you know, it's interesting. So we had Dimitri from Waymo. He was the co CEO of Waymo here a couple of weeks ago. You know, a lot of what we were talking about is that, yeah, it's great to have Waymo operate, I think, I don't know, in 15 cities today. But I mean, I think there's probably both data to support your argument, because the number of cars that Waymo is running is still not that large. It's like maybe like 5,000, 6,000 across all of its cities. Right? So it's not... It's actually, like, low thousands, if you are... Mid thousands, I guess. But on the flip side, their bootstrapping took them seventeen years to kinda get to that point. So I... You know, it's kind of, I guess, making an argument for you. Asymptotically, you don't need a high end count to be able to collect the amount of data. And, you know, cars are operating in the most, in some ways, high risk environments alongside humans. So that's good. But how do you kinda cut down the bootstrap time?
00:24:43 · Sergey Levine, Aditya Agarwal
[00:24:43] S0: It's because cars are too specialized. So if if you were a person K. And you're, you know, maybe you're a child, you're growing up, you're learning all your physical skills. If the only thing you were allowed to do in life is drive cars, it'd be very difficult to acquire all the physical skills you need. If you're, you know... Not not even a a baby. If you're like a five year old, a seven year old, and you get put behind the wheel of a car, okay, we all know what's gonna happen. Like, that's not a good that's not a good, you know, child development strategy. The good thing about robots is that they are less specialized because, like, it seems counterintuitive. Like, a robot needs to be able to do more stuff than a car, but it's also a virtue because it can do more stuff than a car. Therefore, it has more things to learn from, some that are higher stakes, some that are lower stakes. And those lower stakes things will bootstrap the higher stakes things just like they do for people. Like, that is not at all a radical idea. So the problem with cars is that they're too specialized. [00:25:39] S1: It's actually a good answer because you're right that, like, actually, certainly operating a car as a human is pretty unnatural to kind of, like, have the interplay of using your hands and your legs to kinda do something. There's not much prior art that you can kind of... In your own personal experience that you can draw from. Whereas for a robot, you can kind of reuse and kinda recombine a lot of the the core underlying concepts. So that makes sense.
00:25:39 · Aditya Agarwal, Sergey Levine
[00:25:39] S1: It's actually a good answer because you're right that, like, actually, certainly operating a car as a human is pretty unnatural to kind of, like, have the interplay of using your hands and your legs to kinda do something. There's not much prior art that you can kind of... In your own personal experience that you can draw from. Whereas for a robot, you can kind of reuse and kinda recombine a lot of the the core underlying concepts. So that makes sense. [00:26:01] S0: Yeah. Also, a car can't, like, learn about the world by playing around because you're gonna get stopped by the cops or crash by something. [00:26:06] S1: Yeah. RL is hard. Well, yeah, walk us through, like, some of the accelerating feedback loops that you have seen when you are able to deploy your models on kind of, like, hardware into kind of, like, settings that then allows them to kind of, like, start collecting data in a particular setting, which then kind of, like, in some ways accelerates the capabilities of those models?
00:26:31 · Sergey Levine
[00:26:31] S0: So both deployments that we've done ourselves and deployments that other companies using our models have done, you know, oftentimes, the the theme there is actually pretty clear cut that you get to a level of autonomy that's sufficient that you can deploy something under human oversight and human interventions, and the interventions that fix the mistakes provide exactly the data that you need to get the model to very high performance. That's like... That... That's not a a novel strategy at this point. This is something that a lot of folks working on robotics robotic learning understand very well. I think, however, in the in the story arc of this stuff, that's actually just the beginning. I think eventually, what we'll get to is a situation where not only can you deploy and use human interventions to fix up the mistakes and provide exactly the right data, but the robot's own experience and also provides additional data to improve over time. And this is something that is not yet widely used in deployment, but it is working in laboratory experiments that we're doing. And it actually make... It's a very natural transition because when you have a robot deployed under partial autonomy with interventions, what the interventions do is they correct mistakes. So the robot is maybe sorting something at a logistics facility. It drops an item. The intervention is to pick up the item and put it back. So now the robot knows how to fix the mistake. But it doesn't know how to avoid that mistake in the first place, And that's where the reinforcement learning will come in and address those issues. So initially, the interventions give you a way to fix mistakes. That makes you more autonomous. So now you can be clumsy but autonomous, and then the RL comes in and fixes the clumsiness. And that's a slower process and is enabled by greater autonomy.
00:28:04 · Aditya Agarwal
[00:28:04] S1: So maybe to zoom out for a second. Right? We obviously have a number of companies that are really doing full stack kind of robotics development. Let's let's talk maybe, as an example, Optimus, right, at Tesla. And then we also have folks who are working much more on, like, broad based horizontal capabilities with then kind of essentially integrators or kind of like folks who are deploying, figuring out how to combine them. How do you think this plays out over the next five years? Right? Like, obviously, pros and cons of each approach, and you have a very strong, I guess, know, horse in the race slash point of view. But walk us through kind of, like, how you think about the vertical versus horizontal landscape for robotics today.
00:28:48 · Sergey Levine
[00:28:48] S0: Yeah. So I think that the... To me, the most important thing here is that the robotic problem is really difficult. And one of the really important lessons of, you know, the... All of the recent advances we've seen in AI is that when you're faced with a really difficult intelligence problem, somewhat counterintuitively, making the problem as big as possible actually makes it easier. So if your goal is to solve machine translation, you know, naively, we might think that the best way to do it is to build a machine translation model. Like, get lots of paired examples of, like, English sentences and Chinese sentences, train a model that maps one to the other. That's what the the early sequence to sequence models did, and that's your method. Turns out that's the wrong answer. The right answer is to solve all NLP problems all at once, and then you get a better machine translation system. And this is like... At this point, this is not a controversial statement. Like, every single NLP researcher will tell you this is exactly what you do. So what does that tell us about robotics? That if you want your humanoid to be really good, do you get lots of examples of a humanoid, or do you solve all of the robotics problems? I think the answer there is very clear, and I think that there's a good reason why AI systems have this blessing of generality because when you have generality, you get more data. You see more of the unusual situations, more of the exceptions, more of the edge cases. And in robotics, that's even more important than it is in natural language. So I think this lesson actually applies more to robotics than it does to NLP.