Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: It makes me wonder what the last type of tasks you think will be possible for a robotic system to achieve. [00:00:00] S1: I think changing a child's diaper will be really, really hard. This really is just Moravec's paradox all over again, that people are extremely good at certain things. We're very good at physical things. We're also very good at interacting with other people. And that makes sense. We have to be. That's a lot of our existence. So things that involve behaviors, interacting with other people, where you have to help somebody – I think that's a lot harder than people appreciate. Elderly care, taking care of small children. I think those things are going to be hard, and they're probably going to be harder than people think. [00:00:00] S0: And the stakes are very high. [00:00:00] S1: It's not just that. The stakes are high in many places. It's just that it's probably the pinnacle of something that fools us into thinking that it's easier than it really is. We are so evolved for interacting with people and doing things physically. If you're helping somebody get up the stairs or get out of bed, you don't have to think very carefully about how you're going to do that. So I think it's really the pinnacle of Moravec's paradox.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: It's not just that. The stakes are high in many places. It's just that it's probably the pinnacle of something that fools us into thinking that it's easier than it really is. We are so evolved for interacting with people and doing things physically. If you're helping somebody get up the stairs or get out of bed, you don't have to think very carefully about how you're going to do that. So I think it's really the pinnacle of Moravec's paradox. [00:00:00] S0: If I think about an LLM as a brain, and now it's effectively studied everything – I don't know how else to put it. And then I think about a robotics model's brain instead. What are the dark parts of the brain? What has it not been able to study? What are the areas that have just been really difficult that matter but have been hard for us to get into?
· Sergey Levine
[00:00:00] S1: One of the things that people are remarkably good at is using physical analogies to understand other situations. I don't know whether this is something that LLMs can or can't do, but it is something that people use a lot. They use it in everyday life, and they also use it for very sophisticated problems. So for example you could say, that company has a lot of momentum. That's a physical analogy. You know exactly what it means. I don't have to explain that statement to you. But if you actually think about that, this is quite a complex thing. There's a lot riding on that word momentum. There is an interview with Richard Feynman where he talks about teaching, but he talks about analogies that he makes in regard to subatomic particles. And he says, we use the word spin. The thing is not really spinning – it's not like a spinning top. But all those analogies help us make sense of it. And not just in a way that allows explaining concepts, but it actually leads to conclusions. It actually leads to inferences, and those inferences actually make sense. We're so primed to interact with the physical world, so primed to have physical intelligence that you can use it in everyday speech by saying that company has a lot of momentum, and you can use it when advancing fundamental theoretical physics. That's kind of remarkable. I don't know if LLMs can do that. Maybe they can, but I think that really understanding physical interactions, causal structures, all that kind of stuff – there is something special about that. And it's clearly something that people get a lot of mileage out of.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: I'd love to talk about the role of researchers and the actual people doing the research. In LLM world, it's fairly shocking how few people are, at the global scale, responsible for basically all the progress in LLMs. Someone like Ilya is an example. What is that like in robotics? How many people in the world are truly impacting this trajectory? And then I want to ask what good research means. [00:00:00] S1: I think those questions are often very hard to answer about science because I think that we sometimes have a tendency – especially when we look at history – to underline particular milestones. And certainly in machine learning, this is the case. AlexNet was a big step forward. That's true. But I think it's also important to remember that these advances, they happen because lots of people are trying lots of things, and even some of the failures are actually very instructive. I complained before a little bit in a low-key way about the controversy around end-to-end robotic learning, but I don't know if robotic learning would have advanced the same way if it were not for the controversy, so to speak. It is true that you can look through the list of successes and mark them down like, "Oh, these folks have a history of repeatedly hitting home runs." But I think in reality, in the scientific community, it's not just the home runs that are responsible for progress. And even some of the failures and even some of the bad ideas are very instructive in pushing towards the good ideas.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: I think those questions are often very hard to answer about science because I think that we sometimes have a tendency – especially when we look at history – to underline particular milestones. And certainly in machine learning, this is the case. AlexNet was a big step forward. That's true. But I think it's also important to remember that these advances, they happen because lots of people are trying lots of things, and even some of the failures are actually very instructive. I complained before a little bit in a low-key way about the controversy around end-to-end robotic learning, but I don't know if robotic learning would have advanced the same way if it were not for the controversy, so to speak. It is true that you can look through the list of successes and mark them down like, "Oh, these folks have a history of repeatedly hitting home runs." But I think in reality, in the scientific community, it's not just the home runs that are responsible for progress. And even some of the failures and even some of the bad ideas are very instructive in pushing towards the good ideas. [00:00:00] S0: It's fascinating to think about. The example you gave before is so interesting, where the research insight was like, just give it some coaching, and it gets better. It seems like that sort of insight can be very powerful and high leverage, which makes me wonder, what have you learned about what makes for a great researcher?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: It's fascinating to think about. The example you gave before is so interesting, where the research insight was like, just give it some coaching, and it gets better. It seems like that sort of insight can be very powerful and high leverage, which makes me wonder, what have you learned about what makes for a great researcher? [00:00:00] S1: Research is definitely different from engineering, because in research the important thing is to get to an answer to a question, which often requires cutting some corners. One of the most delicate decisions in research is, when do you try new things and when do you stick with what you're already trying? That's very, very delicate. It's very, very hard to figure that out. And if you get it wrong, then you can miss something really remarkable. If you get it wrong and you don't stick with something for long enough, you might be right there, you might be about to get to the answer, and then you stop just short of it. That's terrible. Or you could get stuck hammering against something that's never going to give way for years, deciding when to turn a little bit and look this way and that to open yourself up to more opportunities, versus when should you keep hammering on the thing because you're about to get the solution. That's often the most important decision. And some people have an instinct for getting that right. That counts for a lot. [00:00:00] S0: You've obviously been in and around – and are – great researchers. What are these people like as people? How do they tend to be distinctive from the average person?