Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: What is the hardest part about building in this way for you? When you see other approaches that are more legible to the average person – there's a robot moving around doing this one specific thing, it looks a certain way. What's the hardest part about this approach as you're doing it? [00:00:00] S1: I think this has actually been an issue in my whole career, because when you work on robotic learning – the more general, the more this becomes important – effective robotic learning, effective generalization, isn't actually the optimal way to have a really exciting demo. The way to have a really exciting demo is to pick a really cool task, control everything else in the environment, set it up so that it's perfectly clean, perfectly pristine, and just make it work in that one setting. That's the way you make a robot demo. And generalization, you can't just show it in one spot. The point of generalization is that it does something relatively mundane that any human could do, but it does it in any situation. So we had some demos that we released last April where we showed our robot cleaning kitchens. It's cool, but if you watch an individual video out of context, it's just like, okay, it's like picking up plates, like anybody can pick up plates, except that we just put it into that home just for that demo and it never had training data from that setting. So obviously you kind of have to understand what's going on to appreciate why this is actually pushing the frontier.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: I think this has actually been an issue in my whole career, because when you work on robotic learning – the more general, the more this becomes important – effective robotic learning, effective generalization, isn't actually the optimal way to have a really exciting demo. The way to have a really exciting demo is to pick a really cool task, control everything else in the environment, set it up so that it's perfectly clean, perfectly pristine, and just make it work in that one setting. That's the way you make a robot demo. And generalization, you can't just show it in one spot. The point of generalization is that it does something relatively mundane that any human could do, but it does it in any situation. So we had some demos that we released last April where we showed our robot cleaning kitchens. It's cool, but if you watch an individual video out of context, it's just like, okay, it's like picking up plates, like anybody can pick up plates, except that we just put it into that home just for that demo and it never had training data from that setting. So obviously you kind of have to understand what's going on to appreciate why this is actually pushing the frontier. [00:00:00] S0: What is your model for the stakes of what you're doing? If you are successful – I'm curious for you to define what that would mean, successful, other than we crossed this chasm of general physical intelligence. But if you cross that line, then what?
· Sergey Levine
[00:00:00] S1: One of the things that I think would be really, really exciting that would be enabled by a general purpose embodied foundation model is the ability to unlock people's imagination in how they build robots and other embodied systems. Personal computers were a really big deal in my mind because it made it possible for lots of people to hack together all sorts of really cool stuff. And there was this Cambrian explosion of amazing applications that started in the nineties and so on, and then was further accelerated by the internet. And I think something like that might happen in the world of robotics, but it can't happen today because if you want to put together some cool new robotics application, some cool new robotics idea, you're going to have to build this monstrous stack and you need to basically solve the intelligence problem. But if there is a solution that someone can build on top of, there's a foundation model that you can prompt that'll provide basic functionality, and then you can fine tune it a little bit or adjust it in some way to your application. Now it actually makes it a lot more tractable for lots of people, lots of companies, lots of individuals to try all sorts of different things. Sometimes we think that robots are going to be one thing. There's people and now we're going to make metal people and that'll be robots. But I don't think that's how it's going to be because no technology has been like that. It's going to be more like a toolkit where you can put together all sorts of really cool applications, get really creative with it. Maybe I'm going to make a robot with like five arms and this one is going to look like that, it's going to move, this one's going to hang from the ceiling. And figure out kind of the right thing to tackle your domain. Maybe also experiment with software, but you need the right platform on top of which to do that. And I think the foundation model can be that thing.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: What are, in your mind, the pros and cons of the humanoid approach to robotics? [00:00:00] S1: There's a lot of value to that. There's a lot of value to capturing the imagination and there's a lot of value getting people to think about what the future might look like in a way that's understandable. In my mind, it's one of many possible kinds of robots that we're likely to have. The challenge of intelligence looks very similar for all of these different robots. I don't think we should be tackling intelligence in the context of one specific body. I think we should handle it in a general way because otherwise it's just really hard to get a handle on this. We need lots of data. The cool thing about being able to build robots is that ultimately they don't have to be constrained to look like humans at all. You can build the right tool for the job. You could imagine that you're building a house with a robot that is a swarm of 1,000 quadcopters. And I think that in the future we'll have a robotic foundation model which can then be adapted to all sorts of applications. And it might really run the gamut from bulldozers or something to humanoids to robotic arms. Maybe it would need to be adapted to each one. Maybe it would need to be fine-tuned. Maybe it would need something in context to understand how that body works. But the fundamentals of how you interact with objects, how things move in the world, how causality works, that's all conserved for all of these different systems.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: There's a lot of value to that. There's a lot of value to capturing the imagination and there's a lot of value getting people to think about what the future might look like in a way that's understandable. In my mind, it's one of many possible kinds of robots that we're likely to have. The challenge of intelligence looks very similar for all of these different robots. I don't think we should be tackling intelligence in the context of one specific body. I think we should handle it in a general way because otherwise it's just really hard to get a handle on this. We need lots of data. The cool thing about being able to build robots is that ultimately they don't have to be constrained to look like humans at all. You can build the right tool for the job. You could imagine that you're building a house with a robot that is a swarm of 1,000 quadcopters. And I think that in the future we'll have a robotic foundation model which can then be adapted to all sorts of applications. And it might really run the gamut from bulldozers or something to humanoids to robotic arms. Maybe it would need to be adapted to each one. Maybe it would need to be fine-tuned. Maybe it would need something in context to understand how that body works. But the fundamentals of how you interact with objects, how things move in the world, how causality works, that's all conserved for all of these different systems. [00:00:00] S0: Do you have a favorite example of what might be possible with true general intelligence that might not be possible with a humanoid only intelligence or something?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Do you have a favorite example of what might be possible with true general intelligence that might not be possible with a humanoid only intelligence or something? [00:00:00] S1: There are a few things that I think are worth thinking about. One is that we can make machines that are very big and machines that are very small. This is not by any means a short term thing, but in the long run I think there's lots of really exciting applications in medicine and surgery where we not only might in the long run not be limited to robots that look like humans, we might not be limited to robots that can even be controlled by humans. Currently, for example, in robotic surgery, it's done entirely through teleoperations so you need something that a person can control in real time with the right level of dexterity. And of course, that limitation holds for current learning-enabled systems too, but in the long run we could imagine addressing that. [00:00:00] S0: Think about the most important hash marks on the timeline of robotics research that have gotten us to here. I always think it's super helpful to set the historical context before we talk about what state is today and where we're going. Can you walk us through that?