00:51:50 · Karol Hausman
So this is kind of how reinforcement learning has evolved over time, and that's what allowed us to actually kind of start applying it again and seeing if we can see the successes of it. Now from the other perspective, the reason why we thought reinforcement learning would be important is that the way most of these models are trained today is mostly based on imitation. So you collect a lot of data, whether it's in simulation or from human videos or from real teleoperation data. And then the objective for these models is try to replicate the actions that you've seen in the dataset. And the important caveat there is that it's not actually the objective that we care about. We don't care about executing every single action exactly the same as it was presented to you. What you really care about is the success of the task, whether you accomplished the task or not. And we don't really have a way of codifying this objective with these models. Right? They're all optimized for a mutation, and therefore it's very difficult to drive the success rate of those models because it's not the objective that they care about. They only care about imitating the actions as closely as possible. And reinforcement learning provides you a way to to basically codify this objective, to have them now be optimizing for actually accomplishing the task successfully. Both of these things combined kind of led us to believe that we should we should start applying reinforcement learning so that we can improve reliability of these methods and have models that work not 70% of the time, but 99.99% of the time because they actually optimize for the right objective. And at the same time, it became possible because now these models can explore in ways that are more intelligent than it was possible before.
00:53:32 · Mario Gabriele
[00:53:32] S1: To circle back to a topic you alluded to earlier, you mentioned that when you, you know, first started chatting with with Lachy, he understood the the need for patience on commercialization. And you sort of referenced the fact that, you know, there's a version of this company that if you're too impatient, maybe you can short circuit the the sort of long term vision. What are the ways that that happens and how do you have to be alert to them?
00:53:57 · Karol Hausman
[00:53:57] S0: I think there's a long history of robotics companies doing this or this happening to them. We're not the first company that starts with a big vision and broad vision for robotics. But if analyze what happened in the past, what often happens is you start with that vision, you start developing the technology, and then because there's usually some external pressure, you try to commercialize this technology at that point. When it's not fully ready yet. And you dive into a particular application, maybe the application that has the biggest TAM or something like that. As soon as you do that, you start cutting corners on the technology itself. And you short circuit that vision of it being a very generalist, general purpose technology that would actually deliver the most value, but now you're just trying to deliver the most value for that customer or for that particular vertical that you chose. And that's perfectly reasonable, Like all of your incentives are tied to that. Your revenue is tied to that, the customer satisfaction is tied to that, your valuation, how employees think about this. So you have all incentives in the world to start cutting corners and make a more special purpose solution. That's usually what happens. So you start with this broad vision of how general purpose robots are going to be solved, but you very quickly end up becoming an application company, like a warehouse pick and place company. And this would be a heartbreaking outcome for me. I think we really have a chance to solve the big problem, the problem of general physical intelligence. And if we do that, you won't just enable that one application that you could have focused on initially, but I believe you'll be able to to to solve all of it. You'd be able to solve it for any robot to do any task.
00:55:43 · Mario Gabriele, Karol Hausman
[00:55:43] S1: And counter intuitively, would be the much much better commercial outcome as well. It's all just about trying to have the the right time span in your head. How does that influence how you think about the right partners for the business? Like, are you optimizing for, I don't know, a range of use cases in that case, so that you're getting the diversity of data? Is it about scale and getting as much of that data from in person deployments as you... Not in person, but real world deployments as you can? [00:56:15] S0: Right now, we're optimizing for the rate of learning. So it's still quite early, you know, unlike maybe what you can see on Twitter today. It's not that, you know, robots are about to knock at your door and show up at your home and do everything. Yes. I don't think everything is solved yet, it's just a matter of scaling up the existing recipes. I think we can push them much, much further than where they are today. But there's still a lot of research to be done and a lot of questions that are unanswered. So right now, we're mostly optimizing for speed and how we can learn as much about the problem as possible. A lot of it does come down to what kind of data to collect, how to integrate the data into the model, and how to have that loop be as tightly closed as possible. It's a long way of saying that it's a fairly nuanced question. But what we're optimizing for is learn as much as possible so that we we can figure out the the scalable recipe that then we can just scale as much as possible.
00:56:15 · Karol Hausman, Mario Gabriele
[00:56:15] S0: Right now, we're optimizing for the rate of learning. So it's still quite early, you know, unlike maybe what you can see on Twitter today. It's not that, you know, robots are about to knock at your door and show up at your home and do everything. Yes. I don't think everything is solved yet, it's just a matter of scaling up the existing recipes. I think we can push them much, much further than where they are today. But there's still a lot of research to be done and a lot of questions that are unanswered. So right now, we're mostly optimizing for speed and how we can learn as much about the problem as possible. A lot of it does come down to what kind of data to collect, how to integrate the data into the model, and how to have that loop be as tightly closed as possible. It's a long way of saying that it's a fairly nuanced question. But what we're optimizing for is learn as much as possible so that we we can figure out the the scalable recipe that then we can just scale as much as possible. [00:57:14] S1: Do you have theses at this point on what gives you the the sort of steepest rate of learning? Whether that's, you know, I don't know, very fine tuned tasks or or something totally different?
00:57:14 · Mario Gabriele, Karol Hausman
[00:57:14] S1: Do you have theses at this point on what gives you the the sort of steepest rate of learning? Whether that's, you know, I don't know, very fine tuned tasks or or something totally different? [00:57:23] S0: I think there is there is a few of those things that we that we learn. We know that the diversity of data is really, really important. We know that the quality of data is really important. We know that closing the loop with the models are very important. I just feel like these are fairly broad statements that, you know, if you had asked me this a year ago, I would probably say something similar. But we did learn a lot about what these terms actually mean. I think, you know, very often people think about the diversity of data or quality of data or or something like that. But we don't really have very good definitions of what quality of data actually is, what does it mean, or what diversity of data actually means, or how do you measure it. So I think these are like fairly deep questions and we are are getting more and more of a of a feel of what that actually is and how we can measure it and how we can optimize for it. But that's a constant push and pull on how much do you scale the existing thing, how much can you better understand it, how much can you increase the the slope of improvement there. So I think we'll just continue on the trajectory as we scale it.