Invest Like the Best with Patrick O'Shaughnessy
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: There's a lot of value to that. There's a lot of value to capturing the imagination and there's a lot of value getting people to think about what the future might look like in a way that's understandable. In my mind, it's one of many possible kinds of robots that we're likely to have. The challenge of intelligence looks very similar for all of these different robots. I don't think we should be tackling intelligence in the context of one specific body. I think we should handle it in a general way because otherwise it's just really hard to get a handle on this. We need lots of data. The cool thing about being able to build robots is that ultimately they don't have to be constrained to look like humans at all. You can build the right tool for the job. You could imagine that you're building a house with a robot that is a swarm of 1,000 quadcopters. And I think that in the future we'll have a robotic foundation model which can then be adapted to all sorts of applications. And it might really run the gamut from bulldozers or something to humanoids to robotic arms. Maybe it would need to be adapted to each one. Maybe it would need to be fine-tuned. Maybe it would need something in context to understand how that body works. But the fundamentals of how you interact with objects, how things move in the world, how causality works, that's all conserved for all of these different systems. [00:00:00] S0: Do you have a favorite example of what might be possible with true general intelligence that might not be possible with a humanoid only intelligence or something?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Do you have a favorite example of what might be possible with true general intelligence that might not be possible with a humanoid only intelligence or something? [00:00:00] S1: There are a few things that I think are worth thinking about. One is that we can make machines that are very big and machines that are very small. This is not by any means a short term thing, but in the long run I think there's lots of really exciting applications in medicine and surgery where we not only might in the long run not be limited to robots that look like humans, we might not be limited to robots that can even be controlled by humans. Currently, for example, in robotic surgery, it's done entirely through teleoperations so you need something that a person can control in real time with the right level of dexterity. And of course, that limitation holds for current learning-enabled systems too, but in the long run we could imagine addressing that. [00:00:00] S0: Think about the most important hash marks on the timeline of robotics research that have gotten us to here. I always think it's super helpful to set the historical context before we talk about what state is today and where we're going. Can you walk us through that?
· Sergey Levine
[00:00:00] S1: At some level, doing end-to-end control for robotic systems is a very, very old idea. The first, for example, autonomous driving systems that used end-to-end learning, they existed in the 1980s. ALVINN was 1986 or '87, and that was a driving system that was demonstrated to drive on highways controlled by a neural network and then from a camera. The neural network was tiny. There are some very venerable concepts, but historically what has been really difficult in robotic learning is that you need a system that handles the application you want to address, that is cost-effective to train for that application, meaning that you don't need a huge amount of data for every single application you want to tackle, handles long-tail scenarios with common sense, so if something weird takes place in the world, it needs to have a reasonable response to it, and then also for the thing that it's actually supposed to do, it needs to be robust, fast, and reliable. And getting all those things together is very, very hard because with machine learning, it works best when there's a lot of data. So if you sort of naively approach a robotic problem and say, like, I want to do washing dishes, the obvious thing to do is to collect like an enormous amount of data on washing dishes. But that's not cost-effective because then you go on to the next application and you go through that process all over again. Being able to train general-purpose models that can handle many tasks is essential to this because now you need a lot less data for each new task. But then even further, and this is the thing that has probably changed the most in the last few years, you also then need to handle the unusual scenarios. For the unusual scenarios, you are probably not going to have experience. What you need to rely on is knowledge that you've acquired from other sources that you can ground in a new situation. And people are extremely good at this. If you're driving a car and there is something going on in the middle of the road, and someone put up a sign saying, "Don't go here", there's a gas leak or something. You've probably never experienced that before, but you can put these things together and figure out what you're supposed to do in that unusual situation because you have common sense. This has been a huge mystery in robotic learning world – where do you get that common sense?
· Sergey Levine, Patrick O'Shaughnessy
And this is what's changed in the last few years because it turns out that multimodal language models are really good at pulling in knowledge and trying to articulate that knowledge. They're not very good at grounding that knowledge in physical situations, but they know stuff. There's a path to get that common sense by essentially leveraging the knowledge that is contained in multimodal LLMs, but there's also a challenge because you have to somehow plug into that knowledge in the right way. You can't just show it a picture and say, what would you do here, because it doesn't have the context. It doesn't know that you're a robot, this is what you look like, this is what's going on. That's a technological challenge and we've made some headway on addressing that technological challenge, the research community in general has, but most importantly, it's kind of that light at the end of the tunnel. Now we have this way of pulling in lots of knowledge, which can help us handle those long tail scenarios. [00:00:00] S0: Are there hashmark equivalents on the timeline of AlexNet or the Transformer? Are there big, major events that you think everyone will point to in writing the history books about this?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Are there hashmark equivalents on the timeline of AlexNet or the Transformer? Are there big, major events that you think everyone will point to in writing the history books about this? [00:00:00] S1: I think it's very early on right now to answer that definitively. Probably the first end-to-end learning systems, which were in the eighties, that's definitely a milestone. The first deep reinforcement learning systems, which were in the early 2010s, those are probably a milestone because deep reinforcement learning gives us a way to go beyond human-level performance, which I think will be essential for robotic systems. And then there's the more recent stuff, but I don't know how that's going to shake out as far as whether that's something that people will point to, but I do think that the advent of multimodal LLMs that can be adapted to robotic control to bring in that common sense, I do think that's a really important advance. I think we're probably going to see quite a few important events in the next few years, and maybe those will be the things people point to. [00:00:00] S0: Can you tell us your own personal history of approaching the problem? Maybe the origin of when you first became interested and why, and then how you've decided what to spend your personal time and attention on ever since then?
· Sergey Levine
[00:00:00] S1: So I started working in robotics in 2014 after I finished my graduate degree and started a postdoc with Professor Pieter Abbeel at UC Berkeley. I had worked on robots before, but I figured I should get a little bit more education after finishing my degree, and his lab worked on robots, so I tried to apply what I had learned previously to robotics before that I worked on computer graphics. The thing that I've always wanted to really figure out is how to get AI systems that get better and better the more they do things, because I think that's tremendously powerful. If you can have a system that gets better and better the more it does something, and it just keeps getting better, there's no limit and it can master all the skills you'd want it to do. Initially, I tried to approach it in a very blank slate way. You start with nothing, you practice a particular skill, and you get better at that skill. You can do that in a limited setting, and you get something that works, but it's very hard to turn that into a general system that can work in open world settings, because if I practice something over here, and then it goes over there, now something is different, and it needs to practice all over again. When I worked at Google afterwards, I tried to see if we can do that, but now parallelize it across many robots. So collective learning. Can you put 20 robots in a room and have them all learn together. And that works, and it generalizes, but it's very hard for that to handle these tail cases, these edge cases. Now it becomes this savant of this particular task, and that's all it knows in the world. Next step is what I mentioned before, is combining this ability to practice skills with lots of prior knowledge, and that's actually a really, really hard problem. It's not just in robotics where it's a hard problem. I think it's a hard problem in all of AI, because arguably the two big impressive results in AI over the last few decades have been generative AI and deep reinforcement learning. Like if you want a single example to epitomize this generative AI, that's like LLMs, deep reinforcement learning, AlphaGo. They're both very, very impressive, and they're very impressive for very different reasons. The generative AI is impressive because it can reproduce some of the things that humans can do.