Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: We're in the early stages of seeing some of the jobs and other sorts of transformation – in businesses, in the economy, et cetera – that LLMs make possible. Certainly we've seen it in engineering. How do you think about what might happen or what you hope will happen, when we're at a similar stage – whenever that happens to be for robotics – where all of a sudden we have this thing that's general, that's useful. The world's very efficient at deploying these things. People are creative. Where do you expect to see the world start to change most in the early days? [00:00:00] S1: I really don't know. I don't think anybody would have been able to predict how the LLM stuff evolved and people would have guessed, but this is why I keep coming back to this idea that maybe the key is to let people try lots of things. One of the really amazing things about applications of LLMs is that they are really accessible and somebody could put together a really cool new prototype that under the hood is just prompting ChatGPT or something. But they can experiment with it, they can try it out, see what it does. And there's an amazing power to having lots of smart people rapidly iterating and prototyping lots of things. That's a lot of why Physical Intelligence has really put a premium on engagement. We've open sourced our models. We would like to engage with lots of other companies that are building robots, because we all see a lot of power in this effect of having many people trying out lots of things. [00:00:00] S0: What are the major controversies in the robotics community?
· Sergey Levine
[00:00:00] S1: To me a controversy is when someone gets in an argument with me at a conference. But I can tell you the kind of arguments that I find myself in, and it's kind of an interesting trajectory. In the early days, the main argument I would have with people is, does learning have a place in robotic AI? I think part of why that was often a controversial point is that in a traditional engineering pipeline, robots do look very different than software artifacts. They're physical. They can affect stuff around them. There are safety considerations. There are a lot of weird situations they can get into. It took a really long time for the robotics research community to really internalize that you don't necessarily need to program in things like knowledge of physics. You don't necessarily need a physics simulator inside your robot when it's planning. We can actually have a learning system figure all that stuff out. That was a very controversial thing for a very long time. I think at this point, there's a lot of acceptance that learning is a really important part of robotics, but I don't think there's still universal acceptance that end-to-end learning is the right way to go. Basically, I don't think there's universal acceptance of the bitter lesson. The bitter lesson says that you should not program the machine to think the way you think it should think, but you should let it learn from data. And that is not a universally accepted idea. I think there's good arguments against it, but I think that in the long run, if we want that generality, especially generality in the machine's ability to improve, then we need it to primarily be learning from data.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: What is the good argument against? [00:00:00] S1: My best attempt at steelmanning this is that if you want something reliable in a really complicated open world setting, then you can't afford not to use what you already know about the physical world. And we've got textbooks full of this stuff, so why don't we just plug in what we know from the textbooks? [00:00:00] S0: What is compositional learning? Can you describe that? [00:00:00] S1: One of my students, he had this idea where he asked a language model to provide a recipe for how to make a sandwich in International Phonetic Alphabet. International phonetic alphabet is these symbols that they use in a dictionary to explain how to pronounce a word. And it's very peculiar because it only ever appears for individual words in a dictionary. You never see freeform text written in International Phonetic Alphabet. But if you ask a good language model, it will write paragraphs in IPA for you. And that is compositional generalization. It means that you have never seen this particular language, this particular alphabet used to write paragraphs, but you understand paragraphs, you understand that it's compositional with different alphabets, so you can solve the problem. You can imagine the same thing coming up in robotics, that you've learned a repertoire of skills, and now you can combine and mix those skills and apply them to solve new problems. [00:00:00] S0: It makes me wonder what the last type of tasks you think will be possible for a robotic system to achieve.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: It makes me wonder what the last type of tasks you think will be possible for a robotic system to achieve. [00:00:00] S1: I think changing a child's diaper will be really, really hard. This really is just Moravec's paradox all over again, that people are extremely good at certain things. We're very good at physical things. We're also very good at interacting with other people. And that makes sense. We have to be. That's a lot of our existence. So things that involve behaviors, interacting with other people, where you have to help somebody – I think that's a lot harder than people appreciate. Elderly care, taking care of small children. I think those things are going to be hard, and they're probably going to be harder than people think. [00:00:00] S0: And the stakes are very high. [00:00:00] S1: It's not just that. The stakes are high in many places. It's just that it's probably the pinnacle of something that fools us into thinking that it's easier than it really is. We are so evolved for interacting with people and doing things physically. If you're helping somebody get up the stairs or get out of bed, you don't have to think very carefully about how you're going to do that. So I think it's really the pinnacle of Moravec's paradox.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: It's not just that. The stakes are high in many places. It's just that it's probably the pinnacle of something that fools us into thinking that it's easier than it really is. We are so evolved for interacting with people and doing things physically. If you're helping somebody get up the stairs or get out of bed, you don't have to think very carefully about how you're going to do that. So I think it's really the pinnacle of Moravec's paradox. [00:00:00] S0: If I think about an LLM as a brain, and now it's effectively studied everything – I don't know how else to put it. And then I think about a robotics model's brain instead. What are the dark parts of the brain? What has it not been able to study? What are the areas that have just been really difficult that matter but have been hard for us to get into?
· Sergey Levine
[00:00:00] S1: One of the things that people are remarkably good at is using physical analogies to understand other situations. I don't know whether this is something that LLMs can or can't do, but it is something that people use a lot. They use it in everyday life, and they also use it for very sophisticated problems. So for example you could say, that company has a lot of momentum. That's a physical analogy. You know exactly what it means. I don't have to explain that statement to you. But if you actually think about that, this is quite a complex thing. There's a lot riding on that word momentum. There is an interview with Richard Feynman where he talks about teaching, but he talks about analogies that he makes in regard to subatomic particles. And he says, we use the word spin. The thing is not really spinning – it's not like a spinning top. But all those analogies help us make sense of it. And not just in a way that allows explaining concepts, but it actually leads to conclusions. It actually leads to inferences, and those inferences actually make sense. We're so primed to interact with the physical world, so primed to have physical intelligence that you can use it in everyday speech by saying that company has a lot of momentum, and you can use it when advancing fundamental theoretical physics. That's kind of remarkable. I don't know if LLMs can do that. Maybe they can, but I think that really understanding physical interactions, causal structures, all that kind of stuff – there is something special about that. And it's clearly something that people get a lot of mileage out of.