Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy
[00:00:00] S0: There's this amazing Michael Crichton novel called Prey, where it seems like for a given problem, there may be an optimal or set of optimal shapes of the robot to perform the task. And that where you should do is analyze the problem, then have something that can almost like morph or transform into the right form factor. How do you think about that, the innovation on the form factor side rather than the data and model side?
· Sergey Levine
[00:00:00] S1: I think that in general, in robotics, the ability to innovate on form factors has been very constrained because of the AI challenge. If you have a traditional AI pipeline, like you're doing some motion planning and stuff like that, it's hard to just go and cobble together some new robot. Because when you do that, you have to like characterize the dynamics of the system, you have to do SysID, you have to build up all the stuff. If you could just put together a robot in your garage, load up a robotic foundation model and tell it to do a bunch of stuff – maybe it won't be perfect at it. Maybe it needs more data to really perfect it. But you can at least get the thing moving. I think that can be a really powerful engine just to get everybody to experiment with this stuff. I don't think that I'm the right person to design the perfect robot. There are people here of course who are a lot better at that. But in general, I think that it's just like with personal computers. I think the key is to let people experiment and play around with it and radically lower the barrier to entry for that. Then we'll see a lot of creativity. When we first started using computers, there was a limited number of form factors. Now you can have a computer in your phone, a computer in your car, embed a computer in your refrigerator. They're everywhere and they're very different. Generality, good software, good foundation on top of which you can build applications. Those are key to enabling that.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Your co-founder Lachy once described to me the feeling of physical intelligence for a human is like learning how to ride a bike. There's that moment when you didn't know how to do it and then you do know how to do it. And that feeling is physical intelligence, that snap of understanding. [00:00:00] S1: There's actually a physiological explanation for this. There were studies that were done in monkeys using tools, and you can actually find where in the brain, which neurons activate for the monkey to figure out where its hand is. It turns out that if it's using a tool, they activate based on location of the tool tip, not based on location of the hand. The tool being an extension of your body is a real physiological thing. Your brain literally does that. [00:00:00] S0: Knowing that, what does that do to impact the approach to your research? [00:00:00] S1: It says that physical intelligence should be at some level agnostic to embodiment. A good foundation model should figure out how to manipulate whatever body it's controlling, whatever tools it has at hand. There's basically one problem, not many different problems. There isn't like a humanoid problem and a car problem and a bulldozer problem and a robot bolted to the table problem. There's one problem. And if you solve it at a full level of generality, that's really, really powerful.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: It says that physical intelligence should be at some level agnostic to embodiment. A good foundation model should figure out how to manipulate whatever body it's controlling, whatever tools it has at hand. There's basically one problem, not many different problems. There isn't like a humanoid problem and a car problem and a bulldozer problem and a robot bolted to the table problem. There's one problem. And if you solve it at a full level of generality, that's really, really powerful. [00:00:00] S0: We're in the early stages of seeing some of the jobs and other sorts of transformation – in businesses, in the economy, et cetera – that LLMs make possible. Certainly we've seen it in engineering. How do you think about what might happen or what you hope will happen, when we're at a similar stage – whenever that happens to be for robotics – where all of a sudden we have this thing that's general, that's useful. The world's very efficient at deploying these things. People are creative. Where do you expect to see the world start to change most in the early days?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: We're in the early stages of seeing some of the jobs and other sorts of transformation – in businesses, in the economy, et cetera – that LLMs make possible. Certainly we've seen it in engineering. How do you think about what might happen or what you hope will happen, when we're at a similar stage – whenever that happens to be for robotics – where all of a sudden we have this thing that's general, that's useful. The world's very efficient at deploying these things. People are creative. Where do you expect to see the world start to change most in the early days? [00:00:00] S1: I really don't know. I don't think anybody would have been able to predict how the LLM stuff evolved and people would have guessed, but this is why I keep coming back to this idea that maybe the key is to let people try lots of things. One of the really amazing things about applications of LLMs is that they are really accessible and somebody could put together a really cool new prototype that under the hood is just prompting ChatGPT or something. But they can experiment with it, they can try it out, see what it does. And there's an amazing power to having lots of smart people rapidly iterating and prototyping lots of things. That's a lot of why Physical Intelligence has really put a premium on engagement. We've open sourced our models. We would like to engage with lots of other companies that are building robots, because we all see a lot of power in this effect of having many people trying out lots of things. [00:00:00] S0: What are the major controversies in the robotics community?
· Sergey Levine
[00:00:00] S1: To me a controversy is when someone gets in an argument with me at a conference. But I can tell you the kind of arguments that I find myself in, and it's kind of an interesting trajectory. In the early days, the main argument I would have with people is, does learning have a place in robotic AI? I think part of why that was often a controversial point is that in a traditional engineering pipeline, robots do look very different than software artifacts. They're physical. They can affect stuff around them. There are safety considerations. There are a lot of weird situations they can get into. It took a really long time for the robotics research community to really internalize that you don't necessarily need to program in things like knowledge of physics. You don't necessarily need a physics simulator inside your robot when it's planning. We can actually have a learning system figure all that stuff out. That was a very controversial thing for a very long time. I think at this point, there's a lot of acceptance that learning is a really important part of robotics, but I don't think there's still universal acceptance that end-to-end learning is the right way to go. Basically, I don't think there's universal acceptance of the bitter lesson. The bitter lesson says that you should not program the machine to think the way you think it should think, but you should let it learn from data. And that is not a universally accepted idea. I think there's good arguments against it, but I think that in the long run, if we want that generality, especially generality in the machine's ability to improve, then we need it to primarily be learning from data.