Invest Like the Best with Patrick O'Shaughnessy
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: Fundamentally, the goal of physical intelligence is to develop robotic foundation models that can control basically any embodied system to do any task. Broadly speaking, you could imagine that in the same way that a language model is kind of rapidly evolving towards a system that can do any task that can be expressed in language, what we would like is to build a new class of models that can do any task that can be done by a physical actuated device. Part of the thesis of this company is that we believe that doing it at the full level of generality might actually in the long run be easier than trying to special case very specific narrow application domains. Again, in much the same way that for language models, it turned out to be easier in some ways to solve natural language tasks in their full generality than to narrowly target like machine translation or sentiment analysis or whatever. [00:00:00] S0: That may not be obvious why you would make that bet versus a robot that just does your dishes or something. What are the key trade-offs to understand and why make the decision that you made?
· Sergey Levine
[00:00:00] S1: In the world of natural language, we saw that there were a lot of efforts to develop domain specific solutions that tackled specific problems. Somebody would spend a lot of time thinking about how English differs from French and then build a machine translation system. The reason that language models took over for all of those different application domains is because they can leverage much broader sources of data. It's not even as simple as saying like, oh, we have this data for this application, this data for this application; we can merge everything. It's actually more than that. It's when you can leverage weakly labelled data – in the case of language models that you just mine from the web, you actually learn more about the world. So you establish a foundation of world understanding, and then on top of that foundation, it turns out to be much more effective to build out different applications. To bring this into robotics, the calculus doesn't look quite the same because in robotics, we don't have an internet sized data set that we can just draw on. But this notion of understanding the world, if anything, is actually more important in robotics. Because if you have many different tasks, maybe even many different physical systems, then you can go from training individual dishwashing specialists or laundry folding specialists and instead train a model that actually understands physical interaction. People can master new skills very, very rapidly because we understand physical interaction. We can intuitively grasp what's going to happen in this new and familiar situation, and just bootstrap things really really quickly. If we can draw on data from many sources, many applications, many robots, then we can have a model that has a physical understanding, and it'll be much, much easier to put new applications on top of that platform.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: What is the hardest part about building in this way for you? When you see other approaches that are more legible to the average person – there's a robot moving around doing this one specific thing, it looks a certain way. What's the hardest part about this approach as you're doing it? [00:00:00] S1: I think this has actually been an issue in my whole career, because when you work on robotic learning – the more general, the more this becomes important – effective robotic learning, effective generalization, isn't actually the optimal way to have a really exciting demo. The way to have a really exciting demo is to pick a really cool task, control everything else in the environment, set it up so that it's perfectly clean, perfectly pristine, and just make it work in that one setting. That's the way you make a robot demo. And generalization, you can't just show it in one spot. The point of generalization is that it does something relatively mundane that any human could do, but it does it in any situation. So we had some demos that we released last April where we showed our robot cleaning kitchens. It's cool, but if you watch an individual video out of context, it's just like, okay, it's like picking up plates, like anybody can pick up plates, except that we just put it into that home just for that demo and it never had training data from that setting. So obviously you kind of have to understand what's going on to appreciate why this is actually pushing the frontier.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: I think this has actually been an issue in my whole career, because when you work on robotic learning – the more general, the more this becomes important – effective robotic learning, effective generalization, isn't actually the optimal way to have a really exciting demo. The way to have a really exciting demo is to pick a really cool task, control everything else in the environment, set it up so that it's perfectly clean, perfectly pristine, and just make it work in that one setting. That's the way you make a robot demo. And generalization, you can't just show it in one spot. The point of generalization is that it does something relatively mundane that any human could do, but it does it in any situation. So we had some demos that we released last April where we showed our robot cleaning kitchens. It's cool, but if you watch an individual video out of context, it's just like, okay, it's like picking up plates, like anybody can pick up plates, except that we just put it into that home just for that demo and it never had training data from that setting. So obviously you kind of have to understand what's going on to appreciate why this is actually pushing the frontier. [00:00:00] S0: What is your model for the stakes of what you're doing? If you are successful – I'm curious for you to define what that would mean, successful, other than we crossed this chasm of general physical intelligence. But if you cross that line, then what?
· Sergey Levine
[00:00:00] S1: One of the things that I think would be really, really exciting that would be enabled by a general purpose embodied foundation model is the ability to unlock people's imagination in how they build robots and other embodied systems. Personal computers were a really big deal in my mind because it made it possible for lots of people to hack together all sorts of really cool stuff. And there was this Cambrian explosion of amazing applications that started in the nineties and so on, and then was further accelerated by the internet. And I think something like that might happen in the world of robotics, but it can't happen today because if you want to put together some cool new robotics application, some cool new robotics idea, you're going to have to build this monstrous stack and you need to basically solve the intelligence problem. But if there is a solution that someone can build on top of, there's a foundation model that you can prompt that'll provide basic functionality, and then you can fine tune it a little bit or adjust it in some way to your application. Now it actually makes it a lot more tractable for lots of people, lots of companies, lots of individuals to try all sorts of different things. Sometimes we think that robots are going to be one thing. There's people and now we're going to make metal people and that'll be robots. But I don't think that's how it's going to be because no technology has been like that. It's going to be more like a toolkit where you can put together all sorts of really cool applications, get really creative with it. Maybe I'm going to make a robot with like five arms and this one is going to look like that, it's going to move, this one's going to hang from the ceiling. And figure out kind of the right thing to tackle your domain. Maybe also experiment with software, but you need the right platform on top of which to do that. And I think the foundation model can be that thing.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: What are, in your mind, the pros and cons of the humanoid approach to robotics? [00:00:00] S1: There's a lot of value to that. There's a lot of value to capturing the imagination and there's a lot of value getting people to think about what the future might look like in a way that's understandable. In my mind, it's one of many possible kinds of robots that we're likely to have. The challenge of intelligence looks very similar for all of these different robots. I don't think we should be tackling intelligence in the context of one specific body. I think we should handle it in a general way because otherwise it's just really hard to get a handle on this. We need lots of data. The cool thing about being able to build robots is that ultimately they don't have to be constrained to look like humans at all. You can build the right tool for the job. You could imagine that you're building a house with a robot that is a swarm of 1,000 quadcopters. And I think that in the future we'll have a robotic foundation model which can then be adapted to all sorts of applications. And it might really run the gamut from bulldozers or something to humanoids to robotic arms. Maybe it would need to be adapted to each one. Maybe it would need to be fine-tuned. Maybe it would need something in context to understand how that body works. But the fundamentals of how you interact with objects, how things move in the world, how causality works, that's all conserved for all of these different systems.