Invest Like the Best with Patrick O'Shaughnessy
· Sergey Levine
[00:00:00] S1: One of the things that I think would be really, really exciting that would be enabled by a general purpose embodied foundation model is the ability to unlock people's imagination in how they build robots and other embodied systems. Personal computers were a really big deal in my mind because it made it possible for lots of people to hack together all sorts of really cool stuff. And there was this Cambrian explosion of amazing applications that started in the nineties and so on, and then was further accelerated by the internet. And I think something like that might happen in the world of robotics, but it can't happen today because if you want to put together some cool new robotics application, some cool new robotics idea, you're going to have to build this monstrous stack and you need to basically solve the intelligence problem. But if there is a solution that someone can build on top of, there's a foundation model that you can prompt that'll provide basic functionality, and then you can fine tune it a little bit or adjust it in some way to your application. Now it actually makes it a lot more tractable for lots of people, lots of companies, lots of individuals to try all sorts of different things. Sometimes we think that robots are going to be one thing. There's people and now we're going to make metal people and that'll be robots. But I don't think that's how it's going to be because no technology has been like that. It's going to be more like a toolkit where you can put together all sorts of really cool applications, get really creative with it. Maybe I'm going to make a robot with like five arms and this one is going to look like that, it's going to move, this one's going to hang from the ceiling. And figure out kind of the right thing to tackle your domain. Maybe also experiment with software, but you need the right platform on top of which to do that. And I think the foundation model can be that thing.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: What are, in your mind, the pros and cons of the humanoid approach to robotics? [00:00:00] S1: There's a lot of value to that. There's a lot of value to capturing the imagination and there's a lot of value getting people to think about what the future might look like in a way that's understandable. In my mind, it's one of many possible kinds of robots that we're likely to have. The challenge of intelligence looks very similar for all of these different robots. I don't think we should be tackling intelligence in the context of one specific body. I think we should handle it in a general way because otherwise it's just really hard to get a handle on this. We need lots of data. The cool thing about being able to build robots is that ultimately they don't have to be constrained to look like humans at all. You can build the right tool for the job. You could imagine that you're building a house with a robot that is a swarm of 1,000 quadcopters. And I think that in the future we'll have a robotic foundation model which can then be adapted to all sorts of applications. And it might really run the gamut from bulldozers or something to humanoids to robotic arms. Maybe it would need to be adapted to each one. Maybe it would need to be fine-tuned. Maybe it would need something in context to understand how that body works. But the fundamentals of how you interact with objects, how things move in the world, how causality works, that's all conserved for all of these different systems.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: There's a lot of value to that. There's a lot of value to capturing the imagination and there's a lot of value getting people to think about what the future might look like in a way that's understandable. In my mind, it's one of many possible kinds of robots that we're likely to have. The challenge of intelligence looks very similar for all of these different robots. I don't think we should be tackling intelligence in the context of one specific body. I think we should handle it in a general way because otherwise it's just really hard to get a handle on this. We need lots of data. The cool thing about being able to build robots is that ultimately they don't have to be constrained to look like humans at all. You can build the right tool for the job. You could imagine that you're building a house with a robot that is a swarm of 1,000 quadcopters. And I think that in the future we'll have a robotic foundation model which can then be adapted to all sorts of applications. And it might really run the gamut from bulldozers or something to humanoids to robotic arms. Maybe it would need to be adapted to each one. Maybe it would need to be fine-tuned. Maybe it would need something in context to understand how that body works. But the fundamentals of how you interact with objects, how things move in the world, how causality works, that's all conserved for all of these different systems. [00:00:00] S0: Do you have a favorite example of what might be possible with true general intelligence that might not be possible with a humanoid only intelligence or something?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Do you have a favorite example of what might be possible with true general intelligence that might not be possible with a humanoid only intelligence or something? [00:00:00] S1: There are a few things that I think are worth thinking about. One is that we can make machines that are very big and machines that are very small. This is not by any means a short term thing, but in the long run I think there's lots of really exciting applications in medicine and surgery where we not only might in the long run not be limited to robots that look like humans, we might not be limited to robots that can even be controlled by humans. Currently, for example, in robotic surgery, it's done entirely through teleoperations so you need something that a person can control in real time with the right level of dexterity. And of course, that limitation holds for current learning-enabled systems too, but in the long run we could imagine addressing that. [00:00:00] S0: Think about the most important hash marks on the timeline of robotics research that have gotten us to here. I always think it's super helpful to set the historical context before we talk about what state is today and where we're going. Can you walk us through that?
· Sergey Levine
[00:00:00] S1: At some level, doing end-to-end control for robotic systems is a very, very old idea. The first, for example, autonomous driving systems that used end-to-end learning, they existed in the 1980s. ALVINN was 1986 or '87, and that was a driving system that was demonstrated to drive on highways controlled by a neural network and then from a camera. The neural network was tiny. There are some very venerable concepts, but historically what has been really difficult in robotic learning is that you need a system that handles the application you want to address, that is cost-effective to train for that application, meaning that you don't need a huge amount of data for every single application you want to tackle, handles long-tail scenarios with common sense, so if something weird takes place in the world, it needs to have a reasonable response to it, and then also for the thing that it's actually supposed to do, it needs to be robust, fast, and reliable. And getting all those things together is very, very hard because with machine learning, it works best when there's a lot of data. So if you sort of naively approach a robotic problem and say, like, I want to do washing dishes, the obvious thing to do is to collect like an enormous amount of data on washing dishes. But that's not cost-effective because then you go on to the next application and you go through that process all over again. Being able to train general-purpose models that can handle many tasks is essential to this because now you need a lot less data for each new task. But then even further, and this is the thing that has probably changed the most in the last few years, you also then need to handle the unusual scenarios. For the unusual scenarios, you are probably not going to have experience. What you need to rely on is knowledge that you've acquired from other sources that you can ground in a new situation. And people are extremely good at this. If you're driving a car and there is something going on in the middle of the road, and someone put up a sign saying, "Don't go here", there's a gas leak or something. You've probably never experienced that before, but you can put these things together and figure out what you're supposed to do in that unusual situation because you have common sense. This has been a huge mystery in robotic learning world – where do you get that common sense?
· Sergey Levine, Patrick O'Shaughnessy
And this is what's changed in the last few years because it turns out that multimodal language models are really good at pulling in knowledge and trying to articulate that knowledge. They're not very good at grounding that knowledge in physical situations, but they know stuff. There's a path to get that common sense by essentially leveraging the knowledge that is contained in multimodal LLMs, but there's also a challenge because you have to somehow plug into that knowledge in the right way. You can't just show it a picture and say, what would you do here, because it doesn't have the context. It doesn't know that you're a robot, this is what you look like, this is what's going on. That's a technological challenge and we've made some headway on addressing that technological challenge, the research community in general has, but most importantly, it's kind of that light at the end of the tunnel. Now we have this way of pulling in lots of knowledge, which can help us handle those long tail scenarios. [00:00:00] S0: Are there hashmark equivalents on the timeline of AlexNet or the Transformer? Are there big, major events that you think everyone will point to in writing the history books about this?