Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy
[00:00:00] S0: My guest today is Sergey Levine, one of the co-founders and researchers at Physical Intelligence. As a disclaimer, I'm an investor in Physical Intelligence because I believe it's one of the most important companies tackling the problem of robotics. As you'll hear us discuss today, robotics has what I would call a scarecrow problem. All of these amazing physical devices are becoming ever more possible in all sorts of cool permutations, but what they all really need is an intelligence – a brain. And that is what they're building at Physical Intelligence. They're trying to develop foundation models that can make any physical robot do any task in any environment. The nature of our conversation today is all of the problems facing robotics, and all of the promise of solving these problems across the world. I hope you enjoy this great conversation with Sergey Levine. Sergey, this is going to be a real treat and a blast to learn about possibly the most exciting impactful area of technology being developed. Just to set the stage before we go back in time, if you could just define physical intelligence as you see it.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: Fundamentally, the goal of physical intelligence is to develop robotic foundation models that can control basically any embodied system to do any task. Broadly speaking, you could imagine that in the same way that a language model is kind of rapidly evolving towards a system that can do any task that can be expressed in language, what we would like is to build a new class of models that can do any task that can be done by a physical actuated device. Part of the thesis of this company is that we believe that doing it at the full level of generality might actually in the long run be easier than trying to special case very specific narrow application domains. Again, in much the same way that for language models, it turned out to be easier in some ways to solve natural language tasks in their full generality than to narrowly target like machine translation or sentiment analysis or whatever. [00:00:00] S0: That may not be obvious why you would make that bet versus a robot that just does your dishes or something. What are the key trade-offs to understand and why make the decision that you made?
· Sergey Levine
[00:00:00] S1: In the world of natural language, we saw that there were a lot of efforts to develop domain specific solutions that tackled specific problems. Somebody would spend a lot of time thinking about how English differs from French and then build a machine translation system. The reason that language models took over for all of those different application domains is because they can leverage much broader sources of data. It's not even as simple as saying like, oh, we have this data for this application, this data for this application; we can merge everything. It's actually more than that. It's when you can leverage weakly labelled data – in the case of language models that you just mine from the web, you actually learn more about the world. So you establish a foundation of world understanding, and then on top of that foundation, it turns out to be much more effective to build out different applications. To bring this into robotics, the calculus doesn't look quite the same because in robotics, we don't have an internet sized data set that we can just draw on. But this notion of understanding the world, if anything, is actually more important in robotics. Because if you have many different tasks, maybe even many different physical systems, then you can go from training individual dishwashing specialists or laundry folding specialists and instead train a model that actually understands physical interaction. People can master new skills very, very rapidly because we understand physical interaction. We can intuitively grasp what's going to happen in this new and familiar situation, and just bootstrap things really really quickly. If we can draw on data from many sources, many applications, many robots, then we can have a model that has a physical understanding, and it'll be much, much easier to put new applications on top of that platform.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: What is the hardest part about building in this way for you? When you see other approaches that are more legible to the average person – there's a robot moving around doing this one specific thing, it looks a certain way. What's the hardest part about this approach as you're doing it? [00:00:00] S1: I think this has actually been an issue in my whole career, because when you work on robotic learning – the more general, the more this becomes important – effective robotic learning, effective generalization, isn't actually the optimal way to have a really exciting demo. The way to have a really exciting demo is to pick a really cool task, control everything else in the environment, set it up so that it's perfectly clean, perfectly pristine, and just make it work in that one setting. That's the way you make a robot demo. And generalization, you can't just show it in one spot. The point of generalization is that it does something relatively mundane that any human could do, but it does it in any situation. So we had some demos that we released last April where we showed our robot cleaning kitchens. It's cool, but if you watch an individual video out of context, it's just like, okay, it's like picking up plates, like anybody can pick up plates, except that we just put it into that home just for that demo and it never had training data from that setting. So obviously you kind of have to understand what's going on to appreciate why this is actually pushing the frontier.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: I think this has actually been an issue in my whole career, because when you work on robotic learning – the more general, the more this becomes important – effective robotic learning, effective generalization, isn't actually the optimal way to have a really exciting demo. The way to have a really exciting demo is to pick a really cool task, control everything else in the environment, set it up so that it's perfectly clean, perfectly pristine, and just make it work in that one setting. That's the way you make a robot demo. And generalization, you can't just show it in one spot. The point of generalization is that it does something relatively mundane that any human could do, but it does it in any situation. So we had some demos that we released last April where we showed our robot cleaning kitchens. It's cool, but if you watch an individual video out of context, it's just like, okay, it's like picking up plates, like anybody can pick up plates, except that we just put it into that home just for that demo and it never had training data from that setting. So obviously you kind of have to understand what's going on to appreciate why this is actually pushing the frontier. [00:00:00] S0: What is your model for the stakes of what you're doing? If you are successful – I'm curious for you to define what that would mean, successful, other than we crossed this chasm of general physical intelligence. But if you cross that line, then what?
· Sergey Levine
[00:00:00] S1: One of the things that I think would be really, really exciting that would be enabled by a general purpose embodied foundation model is the ability to unlock people's imagination in how they build robots and other embodied systems. Personal computers were a really big deal in my mind because it made it possible for lots of people to hack together all sorts of really cool stuff. And there was this Cambrian explosion of amazing applications that started in the nineties and so on, and then was further accelerated by the internet. And I think something like that might happen in the world of robotics, but it can't happen today because if you want to put together some cool new robotics application, some cool new robotics idea, you're going to have to build this monstrous stack and you need to basically solve the intelligence problem. But if there is a solution that someone can build on top of, there's a foundation model that you can prompt that'll provide basic functionality, and then you can fine tune it a little bit or adjust it in some way to your application. Now it actually makes it a lot more tractable for lots of people, lots of companies, lots of individuals to try all sorts of different things. Sometimes we think that robots are going to be one thing. There's people and now we're going to make metal people and that'll be robots. But I don't think that's how it's going to be because no technology has been like that. It's going to be more like a toolkit where you can put together all sorts of really cool applications, get really creative with it. Maybe I'm going to make a robot with like five arms and this one is going to look like that, it's going to move, this one's going to hang from the ceiling. And figure out kind of the right thing to tackle your domain. Maybe also experiment with software, but you need the right platform on top of which to do that. And I think the foundation model can be that thing.