Invest Like the Best with Patrick O'Shaughnessy
· Sergey Levine
[00:00:00] S1: So I started working in robotics in 2014 after I finished my graduate degree and started a postdoc with Professor Pieter Abbeel at UC Berkeley. I had worked on robots before, but I figured I should get a little bit more education after finishing my degree, and his lab worked on robots, so I tried to apply what I had learned previously to robotics before that I worked on computer graphics. The thing that I've always wanted to really figure out is how to get AI systems that get better and better the more they do things, because I think that's tremendously powerful. If you can have a system that gets better and better the more it does something, and it just keeps getting better, there's no limit and it can master all the skills you'd want it to do. Initially, I tried to approach it in a very blank slate way. You start with nothing, you practice a particular skill, and you get better at that skill. You can do that in a limited setting, and you get something that works, but it's very hard to turn that into a general system that can work in open world settings, because if I practice something over here, and then it goes over there, now something is different, and it needs to practice all over again. When I worked at Google afterwards, I tried to see if we can do that, but now parallelize it across many robots. So collective learning. Can you put 20 robots in a room and have them all learn together. And that works, and it generalizes, but it's very hard for that to handle these tail cases, these edge cases. Now it becomes this savant of this particular task, and that's all it knows in the world. Next step is what I mentioned before, is combining this ability to practice skills with lots of prior knowledge, and that's actually a really, really hard problem. It's not just in robotics where it's a hard problem. I think it's a hard problem in all of AI, because arguably the two big impressive results in AI over the last few decades have been generative AI and deep reinforcement learning. Like if you want a single example to epitomize this generative AI, that's like LLMs, deep reinforcement learning, AlphaGo. They're both very, very impressive, and they're very impressive for very different reasons. The generative AI is impressive because it can reproduce some of the things that humans can do.
· Sergey Levine, Patrick O'Shaughnessy
Like it can draw pictures that look like human pictures, write text. Deep RL is impressive for the opposite reason. It does things that humans hadn't thought of. The big challenge, and this is what I'm leaning up to and what I hope to figure out here at Physical Intelligence, is how to combine those threads, how to bring in all of that knowledge that you get with generative AI, but also go beyond just human level performance with reinforcement learning. [00:00:00] S0: What literally have you done and are you doing to make that happen?
· Sergey Levine
[00:00:00] S1: In the past few years, we started off first by developing the basic foundations. The basic foundation is what's called a vision-language-action model. The vision-language-action model, you can think of as an LLM that has been adapted for robotic control. So the way these things are trained is they're first trained on text data, then they're adapted with lots of image data from the web to understand images, and then they're adapted to robots with lots of very diverse robot data. That's a starting point. That's a way to take all of that web knowledge, get it into a model that can control robots, and get some interesting behaviors out of it. And then from there, we studied two threads – how to get this thing to handle unusual situations with common sense, and how to get it to improve with reinforcement learning. The way you get common sense is by essentially using chain of thought. The robot enters a scene, and instead of directly starting to move, it thinks about what it was asked to do. So if it was told, clean up the kitchen, looks at the scene and says, based on this, I should pick up the plate. And then it goes and does it. So that unlocks all this prior knowledge, because those intermediate inferences benefit from the web scale pre-training. That handles edge cases. And then the reinforcement learning part comes in after you've practiced it a few times, and you can keep getting better and better at the task directly through your experience. For example, we had this demo on making espresso. That system practiced making those espressos many, many times, and used that to improve robustness, improve speed, improve throughput. And we're not done with that. I think there's a lot more to do there, but we have the starting point.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: The robot data itself. Is the right way to think about it – I'm looking at the Gen 1 of these things. I see a camera here, maybe there's some sensors somewhere else. Is it effectively the data being gathered by various sensors strategically placed on the robot at different parts? [00:00:00] S1: Yeah. Something I'll say about sensors is that I think you can actually get away with less than one might think, and still do quite a lot. This platform here has three cameras, one on each wrist, and a base camera. It doesn't have touch sensing, it doesn't have force sensing. It's very bare bones and very low cost. I'm sure that more sensors could make it better, but a good learning method can actually compensate for deficient sensing fairly well. The wrist cameras are essentially a touch sensor in disguise because you can see local deformations when you touch something. [00:00:00] S0: If I think about the analogy to the expert systems of the 80s and 90s in basic AI, to the lesson that scale's all you need, and the sort of counterintuitive nature of that, that you're not teaching it any specific thing, just blasting it with data. And there's this reservoir of internet data. Talk about the reservoir, how to create the reservoir of data needed for this.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: If I think about the analogy to the expert systems of the 80s and 90s in basic AI, to the lesson that scale's all you need, and the sort of counterintuitive nature of that, that you're not teaching it any specific thing, just blasting it with data. And there's this reservoir of internet data. Talk about the reservoir, how to create the reservoir of data needed for this. [00:00:00] S1: So I don't think anybody really knows how much robot data is needed to have truly generalizable and powerful embodied AI. My sense is that we actually don't need to know. What we need to do is get to the point where these systems are useful enough that they can go out into the world and gather more data themselves. Tesla doesn't worry about how much data their cars can collect. If anything, it's the other way around, there's a little too much data. The key is not so much to quantify, here is exactly the price tag of getting the ultimate robot data set. The key is to get a system that can go into the world, that's useful enough, that does a wide variety of different things, and that can keep pulling in more data. [00:00:00] S0: You brought up the example of Tesla, the beautiful system of a thing that's useful without the AI to begin with, because the human drives it and it gathers data. Why then not start with your best guess at something that's useful as a single robot to have the same sort of flywheel thing happen? [00:00:00] S1: I think it's a good idea. [00:00:00] S0: And do you think that's an approach that you'll pursue?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: And do you think that's an approach that you'll pursue? [00:00:00] S1: I don't think that there's like one right answer, right? So I think there are some domains where deploying a system under human control makes a lot of sense. There are some domains where deploying a partially autonomous system is very reasonable. That's kind of domain dependent, because robots aren't just one thing. Some people might not want a robot in their home that is constantly being controlled by a person off-site, but maybe for some applications, that doesn't matter. [00:00:00] S0: If you mark the start of Physical Intelligence through today, what has been the most surprising thing to you that you've discovered or the nature of how the research has gone?