Invest Like the Best with Patrick O'Shaughnessy
· Sergey Levine, Patrick O'Shaughnessy
Like it can draw pictures that look like human pictures, write text. Deep RL is impressive for the opposite reason. It does things that humans hadn't thought of. The big challenge, and this is what I'm leaning up to and what I hope to figure out here at Physical Intelligence, is how to combine those threads, how to bring in all of that knowledge that you get with generative AI, but also go beyond just human level performance with reinforcement learning. [00:00:00] S0: What literally have you done and are you doing to make that happen?
· Sergey Levine
[00:00:00] S1: In the past few years, we started off first by developing the basic foundations. The basic foundation is what's called a vision-language-action model. The vision-language-action model, you can think of as an LLM that has been adapted for robotic control. So the way these things are trained is they're first trained on text data, then they're adapted with lots of image data from the web to understand images, and then they're adapted to robots with lots of very diverse robot data. That's a starting point. That's a way to take all of that web knowledge, get it into a model that can control robots, and get some interesting behaviors out of it. And then from there, we studied two threads – how to get this thing to handle unusual situations with common sense, and how to get it to improve with reinforcement learning. The way you get common sense is by essentially using chain of thought. The robot enters a scene, and instead of directly starting to move, it thinks about what it was asked to do. So if it was told, clean up the kitchen, looks at the scene and says, based on this, I should pick up the plate. And then it goes and does it. So that unlocks all this prior knowledge, because those intermediate inferences benefit from the web scale pre-training. That handles edge cases. And then the reinforcement learning part comes in after you've practiced it a few times, and you can keep getting better and better at the task directly through your experience. For example, we had this demo on making espresso. That system practiced making those espressos many, many times, and used that to improve robustness, improve speed, improve throughput. And we're not done with that. I think there's a lot more to do there, but we have the starting point.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: The robot data itself. Is the right way to think about it – I'm looking at the Gen 1 of these things. I see a camera here, maybe there's some sensors somewhere else. Is it effectively the data being gathered by various sensors strategically placed on the robot at different parts? [00:00:00] S1: Yeah. Something I'll say about sensors is that I think you can actually get away with less than one might think, and still do quite a lot. This platform here has three cameras, one on each wrist, and a base camera. It doesn't have touch sensing, it doesn't have force sensing. It's very bare bones and very low cost. I'm sure that more sensors could make it better, but a good learning method can actually compensate for deficient sensing fairly well. The wrist cameras are essentially a touch sensor in disguise because you can see local deformations when you touch something. [00:00:00] S0: If I think about the analogy to the expert systems of the 80s and 90s in basic AI, to the lesson that scale's all you need, and the sort of counterintuitive nature of that, that you're not teaching it any specific thing, just blasting it with data. And there's this reservoir of internet data. Talk about the reservoir, how to create the reservoir of data needed for this.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: If I think about the analogy to the expert systems of the 80s and 90s in basic AI, to the lesson that scale's all you need, and the sort of counterintuitive nature of that, that you're not teaching it any specific thing, just blasting it with data. And there's this reservoir of internet data. Talk about the reservoir, how to create the reservoir of data needed for this. [00:00:00] S1: So I don't think anybody really knows how much robot data is needed to have truly generalizable and powerful embodied AI. My sense is that we actually don't need to know. What we need to do is get to the point where these systems are useful enough that they can go out into the world and gather more data themselves. Tesla doesn't worry about how much data their cars can collect. If anything, it's the other way around, there's a little too much data. The key is not so much to quantify, here is exactly the price tag of getting the ultimate robot data set. The key is to get a system that can go into the world, that's useful enough, that does a wide variety of different things, and that can keep pulling in more data. [00:00:00] S0: You brought up the example of Tesla, the beautiful system of a thing that's useful without the AI to begin with, because the human drives it and it gathers data. Why then not start with your best guess at something that's useful as a single robot to have the same sort of flywheel thing happen? [00:00:00] S1: I think it's a good idea. [00:00:00] S0: And do you think that's an approach that you'll pursue?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: And do you think that's an approach that you'll pursue? [00:00:00] S1: I don't think that there's like one right answer, right? So I think there are some domains where deploying a system under human control makes a lot of sense. There are some domains where deploying a partially autonomous system is very reasonable. That's kind of domain dependent, because robots aren't just one thing. Some people might not want a robot in their home that is constantly being controlled by a person off-site, but maybe for some applications, that doesn't matter. [00:00:00] S0: If you mark the start of Physical Intelligence through today, what has been the most surprising thing to you that you've discovered or the nature of how the research has gone?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: If you mark the start of Physical Intelligence through today, what has been the most surprising thing to you that you've discovered or the nature of how the research has gone? [00:00:00] S1: One of the things that's been surprising to me is that I think we've made a lot more progress on dexterity than I thought we would. We had good reason to believe that if we just collect more and more data, that just steadily gets better. What was surprising is that we could also get these systems to perform very dextrous behaviors without really doing anything particularly special for that. The same, by the way, also applied to getting systems to work on different embodiments, where we could get our models to work on all sorts of other robots, including robots with multi-fingered hands, robots with different numbers of degrees of freedom. And obviously, we needed to get data and we needed to fine-tune the model, but the model itself didn't need to change. It didn't even need to be told through any kind of prompt what the robot was. And that was also surprising to me, because I would have thought that we would need some fancy techniques to adapt the system to faster, more dextrous, more complex tasks, and also to different kinds of embodiments. But it actually seems to generalize pretty well. [00:00:00] S0: I'm always interested in the spectrum of capabilities and especially where the systems today are more advanced than you think people would probably expect and where they're less advanced than people might expect.