00:49:42 · Karol Hausman
[00:49:42] S0: of reinforcement learning is such that you need to be learning from your own experience, and while gathering that experience you need to encounter some successes. And then you want to increase the probability of good actions that lead to those successes and decrease probability of the actions that lead to failures. What this means is that you need to have... As you as you explore the world, as you collect your own experiences, you need to have some of these successes because if you don't see any of them, you basically don't really know where to go. You're you're kind of lost. And if you start from scratch, if you just try to command random commands to a robot, it's very very unlikely that you'll encounter a success based on that. So let's say you're trying to teach a robot how to grasp an object, and you command basically random commands to all of the motors of the arms. It's very, very unlikely that some of these commands are going to lead to a single success of you grasping an object successfully. And that's what we often refer to as the exploration problem. But we don't really have a way of guiding the robot on how to explore so that you can encounter a success. And as soon as you encounter it, then you're on the flywheel. You can now increase probabilities of those actions. So just to get to the very first one, very, very hard. So... And that's been the problem of reinforcement learning for a very long time. But now with the models that we've been putting together, this exploration problem became much easier because robots now don't start from scratch. They start from a foundational like PI zero, PI zero five, or PI zero six, where they already have some intuitive understanding of how motions work and how you can explore the world around you. So even if you put a new object in front of the robot, the chances are that if you ask it to grasp it, it will either grasp it or it will it will fail in an interesting way. So the probability of you actually doing something useful starts to become very, very high, or at least much higher than it was before. So now you have a way to explore the world, and that allows you to apply reinforcement learning methods much more easily because they can encounter successes much quicker.
00:51:50 · Karol Hausman
So this is kind of how reinforcement learning has evolved over time, and that's what allowed us to actually kind of start applying it again and seeing if we can see the successes of it. Now from the other perspective, the reason why we thought reinforcement learning would be important is that the way most of these models are trained today is mostly based on imitation. So you collect a lot of data, whether it's in simulation or from human videos or from real teleoperation data. And then the objective for these models is try to replicate the actions that you've seen in the dataset. And the important caveat there is that it's not actually the objective that we care about. We don't care about executing every single action exactly the same as it was presented to you. What you really care about is the success of the task, whether you accomplished the task or not. And we don't really have a way of codifying this objective with these models. Right? They're all optimized for a mutation, and therefore it's very difficult to drive the success rate of those models because it's not the objective that they care about. They only care about imitating the actions as closely as possible. And reinforcement learning provides you a way to to basically codify this objective, to have them now be optimizing for actually accomplishing the task successfully. Both of these things combined kind of led us to believe that we should we should start applying reinforcement learning so that we can improve reliability of these methods and have models that work not 70% of the time, but 99.99% of the time because they actually optimize for the right objective. And at the same time, it became possible because now these models can explore in ways that are more intelligent than it was possible before.
00:53:32 · Mario Gabriele
[00:53:32] S1: To circle back to a topic you alluded to earlier, you mentioned that when you, you know, first started chatting with with Lachy, he understood the the need for patience on commercialization. And you sort of referenced the fact that, you know, there's a version of this company that if you're too impatient, maybe you can short circuit the the sort of long term vision. What are the ways that that happens and how do you have to be alert to them?
00:53:57 · Karol Hausman
[00:53:57] S0: I think there's a long history of robotics companies doing this or this happening to them. We're not the first company that starts with a big vision and broad vision for robotics. But if analyze what happened in the past, what often happens is you start with that vision, you start developing the technology, and then because there's usually some external pressure, you try to commercialize this technology at that point. When it's not fully ready yet. And you dive into a particular application, maybe the application that has the biggest TAM or something like that. As soon as you do that, you start cutting corners on the technology itself. And you short circuit that vision of it being a very generalist, general purpose technology that would actually deliver the most value, but now you're just trying to deliver the most value for that customer or for that particular vertical that you chose. And that's perfectly reasonable, Like all of your incentives are tied to that. Your revenue is tied to that, the customer satisfaction is tied to that, your valuation, how employees think about this. So you have all incentives in the world to start cutting corners and make a more special purpose solution. That's usually what happens. So you start with this broad vision of how general purpose robots are going to be solved, but you very quickly end up becoming an application company, like a warehouse pick and place company. And this would be a heartbreaking outcome for me. I think we really have a chance to solve the big problem, the problem of general physical intelligence. And if we do that, you won't just enable that one application that you could have focused on initially, but I believe you'll be able to to to solve all of it. You'd be able to solve it for any robot to do any task.
00:55:43 · Mario Gabriele, Karol Hausman
[00:55:43] S1: And counter intuitively, would be the much much better commercial outcome as well. It's all just about trying to have the the right time span in your head. How does that influence how you think about the right partners for the business? Like, are you optimizing for, I don't know, a range of use cases in that case, so that you're getting the diversity of data? Is it about scale and getting as much of that data from in person deployments as you... Not in person, but real world deployments as you can? [00:56:15] S0: Right now, we're optimizing for the rate of learning. So it's still quite early, you know, unlike maybe what you can see on Twitter today. It's not that, you know, robots are about to knock at your door and show up at your home and do everything. Yes. I don't think everything is solved yet, it's just a matter of scaling up the existing recipes. I think we can push them much, much further than where they are today. But there's still a lot of research to be done and a lot of questions that are unanswered. So right now, we're mostly optimizing for speed and how we can learn as much about the problem as possible. A lot of it does come down to what kind of data to collect, how to integrate the data into the model, and how to have that loop be as tightly closed as possible. It's a long way of saying that it's a fairly nuanced question. But what we're optimizing for is learn as much as possible so that we we can figure out the the scalable recipe that then we can just scale as much as possible.
00:56:15 · Karol Hausman, Mario Gabriele
[00:56:15] S0: Right now, we're optimizing for the rate of learning. So it's still quite early, you know, unlike maybe what you can see on Twitter today. It's not that, you know, robots are about to knock at your door and show up at your home and do everything. Yes. I don't think everything is solved yet, it's just a matter of scaling up the existing recipes. I think we can push them much, much further than where they are today. But there's still a lot of research to be done and a lot of questions that are unanswered. So right now, we're mostly optimizing for speed and how we can learn as much about the problem as possible. A lot of it does come down to what kind of data to collect, how to integrate the data into the model, and how to have that loop be as tightly closed as possible. It's a long way of saying that it's a fairly nuanced question. But what we're optimizing for is learn as much as possible so that we we can figure out the the scalable recipe that then we can just scale as much as possible. [00:57:14] S1: Do you have theses at this point on what gives you the the sort of steepest rate of learning? Whether that's, you know, I don't know, very fine tuned tasks or or something totally different?