Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy
[00:00:00] S0: I was curious before when you said superhuman ability, dexterity or something like that, where we're limited by what we can do or maybe by what we can control, even if it gets smaller. What are some of the other dimensions that we might surpass human ability on in terms of physical ability? What are the other trend lines?
· Sergey Levine
[00:00:00] S1: So here's a fun one. We were working on a task where our robot had to plug in things like power cables or ethernet cables or something like that. When a person does this – obviously if you practice it a lot, you'll get really good at it. But when a person does this without having practiced a lot, you pause frequently, right? Because it's not a physical thing. You just have to pause what's going on. You have to make sure that it's all aligned and all that stuff. So you do it very slowly. And if you're teleoperating a robot, you do it even more slowly because there's this level of indirection. It turns out to be pretty straightforward to go in and find all those pauses and remove them. You can speed things up further. So you can get to a task where a person demonstrates what it means to succeed. And then you can have the robot practice the task and succeed in the same way, but a lot more quickly, a lot more efficiently. The most general way to do this is with reinforcement learning. But there are also some simple tricks you can do if you just want speed. So that's one example of something where you can have a machine that does it a lot better. At some level, you have a processing bottleneck. That's why the person does it slowly, because they have to process what's going on. But speeding up processing is something that people understand quite well in computer science.
· Patrick O'Shaughnessy
[00:00:00] S0: There's this amazing Michael Crichton novel called Prey, where it seems like for a given problem, there may be an optimal or set of optimal shapes of the robot to perform the task. And that where you should do is analyze the problem, then have something that can almost like morph or transform into the right form factor. How do you think about that, the innovation on the form factor side rather than the data and model side?
· Sergey Levine
[00:00:00] S1: I think that in general, in robotics, the ability to innovate on form factors has been very constrained because of the AI challenge. If you have a traditional AI pipeline, like you're doing some motion planning and stuff like that, it's hard to just go and cobble together some new robot. Because when you do that, you have to like characterize the dynamics of the system, you have to do SysID, you have to build up all the stuff. If you could just put together a robot in your garage, load up a robotic foundation model and tell it to do a bunch of stuff – maybe it won't be perfect at it. Maybe it needs more data to really perfect it. But you can at least get the thing moving. I think that can be a really powerful engine just to get everybody to experiment with this stuff. I don't think that I'm the right person to design the perfect robot. There are people here of course who are a lot better at that. But in general, I think that it's just like with personal computers. I think the key is to let people experiment and play around with it and radically lower the barrier to entry for that. Then we'll see a lot of creativity. When we first started using computers, there was a limited number of form factors. Now you can have a computer in your phone, a computer in your car, embed a computer in your refrigerator. They're everywhere and they're very different. Generality, good software, good foundation on top of which you can build applications. Those are key to enabling that.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Your co-founder Lachy once described to me the feeling of physical intelligence for a human is like learning how to ride a bike. There's that moment when you didn't know how to do it and then you do know how to do it. And that feeling is physical intelligence, that snap of understanding. [00:00:00] S1: There's actually a physiological explanation for this. There were studies that were done in monkeys using tools, and you can actually find where in the brain, which neurons activate for the monkey to figure out where its hand is. It turns out that if it's using a tool, they activate based on location of the tool tip, not based on location of the hand. The tool being an extension of your body is a real physiological thing. Your brain literally does that. [00:00:00] S0: Knowing that, what does that do to impact the approach to your research? [00:00:00] S1: It says that physical intelligence should be at some level agnostic to embodiment. A good foundation model should figure out how to manipulate whatever body it's controlling, whatever tools it has at hand. There's basically one problem, not many different problems. There isn't like a humanoid problem and a car problem and a bulldozer problem and a robot bolted to the table problem. There's one problem. And if you solve it at a full level of generality, that's really, really powerful.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: It says that physical intelligence should be at some level agnostic to embodiment. A good foundation model should figure out how to manipulate whatever body it's controlling, whatever tools it has at hand. There's basically one problem, not many different problems. There isn't like a humanoid problem and a car problem and a bulldozer problem and a robot bolted to the table problem. There's one problem. And if you solve it at a full level of generality, that's really, really powerful. [00:00:00] S0: We're in the early stages of seeing some of the jobs and other sorts of transformation – in businesses, in the economy, et cetera – that LLMs make possible. Certainly we've seen it in engineering. How do you think about what might happen or what you hope will happen, when we're at a similar stage – whenever that happens to be for robotics – where all of a sudden we have this thing that's general, that's useful. The world's very efficient at deploying these things. People are creative. Where do you expect to see the world start to change most in the early days?