Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: I'd love to talk about the role of researchers and the actual people doing the research. In LLM world, it's fairly shocking how few people are, at the global scale, responsible for basically all the progress in LLMs. Someone like Ilya is an example. What is that like in robotics? How many people in the world are truly impacting this trajectory? And then I want to ask what good research means. [00:00:00] S1: I think those questions are often very hard to answer about science because I think that we sometimes have a tendency – especially when we look at history – to underline particular milestones. And certainly in machine learning, this is the case. AlexNet was a big step forward. That's true. But I think it's also important to remember that these advances, they happen because lots of people are trying lots of things, and even some of the failures are actually very instructive. I complained before a little bit in a low-key way about the controversy around end-to-end robotic learning, but I don't know if robotic learning would have advanced the same way if it were not for the controversy, so to speak. It is true that you can look through the list of successes and mark them down like, "Oh, these folks have a history of repeatedly hitting home runs." But I think in reality, in the scientific community, it's not just the home runs that are responsible for progress. And even some of the failures and even some of the bad ideas are very instructive in pushing towards the good ideas.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: I think those questions are often very hard to answer about science because I think that we sometimes have a tendency – especially when we look at history – to underline particular milestones. And certainly in machine learning, this is the case. AlexNet was a big step forward. That's true. But I think it's also important to remember that these advances, they happen because lots of people are trying lots of things, and even some of the failures are actually very instructive. I complained before a little bit in a low-key way about the controversy around end-to-end robotic learning, but I don't know if robotic learning would have advanced the same way if it were not for the controversy, so to speak. It is true that you can look through the list of successes and mark them down like, "Oh, these folks have a history of repeatedly hitting home runs." But I think in reality, in the scientific community, it's not just the home runs that are responsible for progress. And even some of the failures and even some of the bad ideas are very instructive in pushing towards the good ideas. [00:00:00] S0: It's fascinating to think about. The example you gave before is so interesting, where the research insight was like, just give it some coaching, and it gets better. It seems like that sort of insight can be very powerful and high leverage, which makes me wonder, what have you learned about what makes for a great researcher?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: It's fascinating to think about. The example you gave before is so interesting, where the research insight was like, just give it some coaching, and it gets better. It seems like that sort of insight can be very powerful and high leverage, which makes me wonder, what have you learned about what makes for a great researcher? [00:00:00] S1: Research is definitely different from engineering, because in research the important thing is to get to an answer to a question, which often requires cutting some corners. One of the most delicate decisions in research is, when do you try new things and when do you stick with what you're already trying? That's very, very delicate. It's very, very hard to figure that out. And if you get it wrong, then you can miss something really remarkable. If you get it wrong and you don't stick with something for long enough, you might be right there, you might be about to get to the answer, and then you stop just short of it. That's terrible. Or you could get stuck hammering against something that's never going to give way for years, deciding when to turn a little bit and look this way and that to open yourself up to more opportunities, versus when should you keep hammering on the thing because you're about to get the solution. That's often the most important decision. And some people have an instinct for getting that right. That counts for a lot. [00:00:00] S0: You've obviously been in and around – and are – great researchers. What are these people like as people? How do they tend to be distinctive from the average person?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: You've obviously been in and around – and are – great researchers. What are these people like as people? How do they tend to be distinctive from the average person? [00:00:00] S1: I think they're just the same. I have a very hard time thinking of a single set of personality traits. There is no constant basically. There might be a commonality in that to do effective science, you have to be very passionate about that. But even that passion can come from many different places. I've worked with people that were remarkably effective that are just driven purely by the desire for novelty. They don't give a damn about what their technology does. They don't give a damn about whether it's useful. They just want cool new ideas. I've also worked with other people that really want to solve a particular thing. And they're just as happy building stuff as they are testing out experiments, as they are hammering away at things. Whatever it takes. [00:00:00] S0: You mentioned the difference between research and engineering, which also makes me think of manufacturing. Elon would be fond of saying that the factory is the product. The hardest part of this whole equation is actually the scale up of – whatever this thing ends up looking like – making 100 million of those. How do you think about that part of the equation, or is it too remote at this stage?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: You mentioned the difference between research and engineering, which also makes me think of manufacturing. Elon would be fond of saying that the factory is the product. The hardest part of this whole equation is actually the scale up of – whatever this thing ends up looking like – making 100 million of those. How do you think about that part of the equation, or is it too remote at this stage? [00:00:00] S1: I think it's an important part of the equation. I'm not sure it's the part of the equation that we most need to figure out right now, but it's certainly part of it. A lot of how I prefer to think about this is to figure out the hard part and then enable a lot of experimentation on the other parts. Making a robot at scale is difficult. Making a robot at scale is even more difficult if you don't know what kind of software is going to run on it afterwards, and you're not even sure whether it's the right kind of robot. One of the really valuable things we can get out of general purpose AI tools like robotic foundation models is the ability to get a lot of the other stuff figured out so that at least some of the uncertainty goes away, so that when you scale things up, you have some confidence that this is really going to work.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: I think it's an important part of the equation. I'm not sure it's the part of the equation that we most need to figure out right now, but it's certainly part of it. A lot of how I prefer to think about this is to figure out the hard part and then enable a lot of experimentation on the other parts. Making a robot at scale is difficult. Making a robot at scale is even more difficult if you don't know what kind of software is going to run on it afterwards, and you're not even sure whether it's the right kind of robot. One of the really valuable things we can get out of general purpose AI tools like robotic foundation models is the ability to get a lot of the other stuff figured out so that at least some of the uncertainty goes away, so that when you scale things up, you have some confidence that this is really going to work. [00:00:00] S0: A lot of people that listen to this are entrepreneurs, people that run companies. A very popular question has become, how should a traditional company begin to think about using LLMs or preparing itself for the ongoing improvement of these models? How would you answer the same question for robotics? How would you encourage companies to think about this?