00:48:01 · Karol Hausman, Mario Gabriele
For every single one of of of tasks or objects, it takes really long to get it exactly right, to get all the friction parameters right, to get the simulation behaviors right, and it's just not as scalable. That's been our finding so far. [00:48:14] S1: And so it's sort of the case to to try and distill that that, you know, with simulation, you can do some of these... I don't know. Maybe this isn't the perfect word, but sort of coarser, larger actions that are more self contained. But once you start trying to, yeah, implement picking up the coffee cup or the towel or whatever it might be, you're starting to rely on a simulation that would have to be so good, you'd effectively have to, you know, create a a simulation that is perfectly similar to reality, and that's where the the real world data starts to become so important. Yeah. You would need to simulate all of the external world Yeah. With any one of those tasks. And that's just too too costly, too difficult to do. But as long as you can get away with [00:49:00] S0: just simulating your own body, I think that works perfectly fine, and that's what we've seen in locomotion or, you know, backflips or dances or things like that.
00:49:00 · Karol Hausman, Mario Gabriele
[00:49:00] S0: just simulating your own body, I think that works perfectly fine, and that's what we've seen in locomotion or, you know, backflips or dances or things like that. [00:49:08] S1: There was a moment, I think, about a year and a little bit ago where you talked about reinforcement learning making a comeback, and that has since become a really interesting part of the way that physical intelligence seems to work with recap. What were you seeing at the time that made you think, you know, this this technique might have sort of a second or additional wind here? And why has that been so useful for you? Yeah. There's been a long history of applying reinforcement learning to robots. And the problem
00:49:42 · Karol Hausman
[00:49:42] S0: of reinforcement learning is such that you need to be learning from your own experience, and while gathering that experience you need to encounter some successes. And then you want to increase the probability of good actions that lead to those successes and decrease probability of the actions that lead to failures. What this means is that you need to have... As you as you explore the world, as you collect your own experiences, you need to have some of these successes because if you don't see any of them, you basically don't really know where to go. You're you're kind of lost. And if you start from scratch, if you just try to command random commands to a robot, it's very very unlikely that you'll encounter a success based on that. So let's say you're trying to teach a robot how to grasp an object, and you command basically random commands to all of the motors of the arms. It's very, very unlikely that some of these commands are going to lead to a single success of you grasping an object successfully. And that's what we often refer to as the exploration problem. But we don't really have a way of guiding the robot on how to explore so that you can encounter a success. And as soon as you encounter it, then you're on the flywheel. You can now increase probabilities of those actions. So just to get to the very first one, very, very hard. So... And that's been the problem of reinforcement learning for a very long time. But now with the models that we've been putting together, this exploration problem became much easier because robots now don't start from scratch. They start from a foundational like PI zero, PI zero five, or PI zero six, where they already have some intuitive understanding of how motions work and how you can explore the world around you. So even if you put a new object in front of the robot, the chances are that if you ask it to grasp it, it will either grasp it or it will it will fail in an interesting way. So the probability of you actually doing something useful starts to become very, very high, or at least much higher than it was before. So now you have a way to explore the world, and that allows you to apply reinforcement learning methods much more easily because they can encounter successes much quicker.
00:51:50 · Karol Hausman
So this is kind of how reinforcement learning has evolved over time, and that's what allowed us to actually kind of start applying it again and seeing if we can see the successes of it. Now from the other perspective, the reason why we thought reinforcement learning would be important is that the way most of these models are trained today is mostly based on imitation. So you collect a lot of data, whether it's in simulation or from human videos or from real teleoperation data. And then the objective for these models is try to replicate the actions that you've seen in the dataset. And the important caveat there is that it's not actually the objective that we care about. We don't care about executing every single action exactly the same as it was presented to you. What you really care about is the success of the task, whether you accomplished the task or not. And we don't really have a way of codifying this objective with these models. Right? They're all optimized for a mutation, and therefore it's very difficult to drive the success rate of those models because it's not the objective that they care about. They only care about imitating the actions as closely as possible. And reinforcement learning provides you a way to to basically codify this objective, to have them now be optimizing for actually accomplishing the task successfully. Both of these things combined kind of led us to believe that we should we should start applying reinforcement learning so that we can improve reliability of these methods and have models that work not 70% of the time, but 99.99% of the time because they actually optimize for the right objective. And at the same time, it became possible because now these models can explore in ways that are more intelligent than it was possible before.
00:53:32 · Mario Gabriele
[00:53:32] S1: To circle back to a topic you alluded to earlier, you mentioned that when you, you know, first started chatting with with Lachy, he understood the the need for patience on commercialization. And you sort of referenced the fact that, you know, there's a version of this company that if you're too impatient, maybe you can short circuit the the sort of long term vision. What are the ways that that happens and how do you have to be alert to them?
00:53:57 · Karol Hausman
[00:53:57] S0: I think there's a long history of robotics companies doing this or this happening to them. We're not the first company that starts with a big vision and broad vision for robotics. But if analyze what happened in the past, what often happens is you start with that vision, you start developing the technology, and then because there's usually some external pressure, you try to commercialize this technology at that point. When it's not fully ready yet. And you dive into a particular application, maybe the application that has the biggest TAM or something like that. As soon as you do that, you start cutting corners on the technology itself. And you short circuit that vision of it being a very generalist, general purpose technology that would actually deliver the most value, but now you're just trying to deliver the most value for that customer or for that particular vertical that you chose. And that's perfectly reasonable, Like all of your incentives are tied to that. Your revenue is tied to that, the customer satisfaction is tied to that, your valuation, how employees think about this. So you have all incentives in the world to start cutting corners and make a more special purpose solution. That's usually what happens. So you start with this broad vision of how general purpose robots are going to be solved, but you very quickly end up becoming an application company, like a warehouse pick and place company. And this would be a heartbreaking outcome for me. I think we really have a chance to solve the big problem, the problem of general physical intelligence. And if we do that, you won't just enable that one application that you could have focused on initially, but I believe you'll be able to to to solve all of it. You'd be able to solve it for any robot to do any task.