Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: You're in such an interesting position because you're at the center of research. Lots of different kinds of people are talking to you, asking you questions. What are questions that you're surprised people don't ask you? [00:00:00] S1: Well, I think the question you asked earlier actually about how somebody should prepare. There's a variant of that question, which would be something like, if I want to start using autonomous robots for a thing, what should I start setting up? Should I set up operations? Should I modify my task in some way so it's more accessible? Should I design new hardware? Maybe I should design new hardware so I can plug your software into it. And I think people make a lot of assumptions about that. For example, one assumption is machine learning requires data, so let me just figure out something that will collect data. That's not often the best assumption because you need the right kind of data. Maybe some data is easy. It's easy to get videos of people doing something, but that doesn't mean that's the right kind of data. And it might be domain-dependent. It might be dependent on the thesis about the technology that will succeed. So I think that people do make a lot of assumptions about that. Not that I necessarily have a better answer for them even if they ask me, but it's something where there's a big space of possibilities. [00:00:00] S0: We talked about these big uncertain long-term timelines. What is the very next thing you are trying to solve?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: We talked about these big uncertain long-term timelines. What is the very next thing you are trying to solve? [00:00:00] S1: A big focus for us right now is actually, better understanding this mid-level reasoning part of the problem. Because we think that we have a pretty good sense for how to acquire low-level physical behaviors, but getting those low-level physical behaviors to generalize requires bringing to bear a lot of this common sense knowledge. The representation of that might be really important. So LLMs make certain kinds of representations very convenient. They make it very convenient to basically turn text into other text, but that's not necessarily the best representation for what an embodied system needs to do. Sometimes it needs to think about things more spatially, sometimes semantically, sometimes other representations, and trying to figure out exactly how to structure that internal thinking process might be a very important question. The answer to that question might be different in the world of embodied foundation models than it is in the world of LLMs. So that's a concrete thing that we're working on now. [00:00:00] S0: If I could somehow get the 100 most informed and active robotics researchers in the room at once and poll them on how certain they are that things will have unlimited capabilities and how soon that might happen, where do you fall in that distribution? [00:00:00] S1: Probably I'm on the optimistic end when it comes to established robotics researchers, and on the pessimistic end relative to robotics entrepreneurs.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: Probably I'm on the optimistic end when it comes to established robotics researchers, and on the pessimistic end relative to robotics entrepreneurs. [00:00:00] S0: I understand the entrepreneur part for sure. You're optimistic by nature. Why are you on the optimistic end of the researcher community? [00:00:00] S1: Robotics has a very long history which has precious few successes, especially when it comes to robotic AI. So I think if we're being honest about it, most robots that are out there doing useful work are still running state-of-the-art technology from the 1980s. Because the robotics problem is hard. Not our fault, it's just a difficult problem. Because of that, I do think that there is good reason for caution – to say that, well, maybe we've made a lot of headway on this part of the problem, but there's many other problems that still remain. Part of why I'm optimistic about this is that I have a sense of what has proven tough for me before, and I can see a lot of the puzzle pieces that I'm imagining could be slotted in to address many of those things. As my co-founder Karol likes to say, when you've climbed the mountain, only then do you see if there's another mountain after it. In robotics, there's been a lot of experience of lots of mountains. Some caution is justified. [00:00:00] S0: Given that endurance is required, who or what most inspires you?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Given that endurance is required, who or what most inspires you? [00:00:00] S1: Boston Dynamics. I think there's a lot of things that we can debate on the technology side, but there is a lot of value in repeatedly showing something that people wouldn't have thought possible, even if there's all sorts of caveats and assumptions and so on. And certainly in robotics, whatever we might say about demos and whatnot, I think it's very fair to say that people have revised their thoughts about what's possible from seeing some of that stuff. I think I'm also inspired by organizations that create an atmosphere for experimentation. There are some research labs that have done a very good job of this. OpenAI has historically done a great job of this – of creating an atmosphere where individual researchers can experiment with things and be empowered to see those things through. ChatGPT was basically John Schulman's pet experiment for a while. It wasn't a concerted corporate strategy with lots of spreadsheets and pie charts. It was a pet project. I think there's something pretty inspiring about organizations that empower people to have pet projects turn into world-changing successes. Certainly one of the aspirations that I and my co-founders have here at Physical Intelligence is to provide some of that to the best of our ability. It's hard to do. [00:00:00] S0: I feel like Google used to have that one day "you can do whatever you want" thing. Is that the spirit of it?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: I feel like Google used to have that one day "you can do whatever you want" thing. Is that the spirit of it? [00:00:00] S1: I was absolutely shocked, when I started working at Google, at the level of leverage that I felt I could have. One of the projects that I did with many of my colleagues there in 2015 was colloquially referred to as the arm farm. So we took a couple dozen robots, put them in a lab and had them collect data. I found out from somebody that they had a warehouse full of robots that nobody was using. I asked Jeff Dean and Vincent Vanhoucke if we could stick them in a lab. I was just thinking like, okay, they're not going to take me seriously. I was a level four research scientist. Jeff was like, "Yeah, let's do it. What do you need?" I just remember feeling like, wow, I had never in my life thought that I'd have that leverage. I mean, I was very young at the time. That's very special. And I think getting to a place where people can unlock their creativity and have that kind of agency can make for a very remarkable place.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: I was absolutely shocked, when I started working at Google, at the level of leverage that I felt I could have. One of the projects that I did with many of my colleagues there in 2015 was colloquially referred to as the arm farm. So we took a couple dozen robots, put them in a lab and had them collect data. I found out from somebody that they had a warehouse full of robots that nobody was using. I asked Jeff Dean and Vincent Vanhoucke if we could stick them in a lab. I was just thinking like, okay, they're not going to take me seriously. I was a level four research scientist. Jeff was like, "Yeah, let's do it. What do you need?" I just remember feeling like, wow, I had never in my life thought that I'd have that leverage. I mean, I was very young at the time. That's very special. And I think getting to a place where people can unlock their creativity and have that kind of agency can make for a very remarkable place. [00:00:00] S0: My friend Jesse has this great question, which is for companies that you're not involved with, which one do you most hope succeeds and why? People used to say Boom a lot because they want to fly places faster. Increasingly, as I've asked this question, people have said PI, because the sheer impact that it might have if you're successful is massive on such a global scale. And it's been really fun just to hear about all the ins and outs of how you're thinking about the problem and attacking it. When I do these interviews, I have the same traditional last question for everyone. What is the kindest thing that anyone's ever done for you?