Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: When you study the history of general purpose technologies – certainly this would be a major one if it comes to fruition – you often find this constellation of things happening around that thing that enable it. LLMs are a direct complement to what you're doing. Are there any other surprising technology areas or trends that help you do what you do but are different? [00:00:00] S1: Robotics hardware has become dramatically more affordable over the last few years. When I started working in robotics about a decade ago, I worked with a robot called a PR2, which I believe had a cost of about $400,000. When I started my lab at UC Berkeley, I used a robot that was in the ballpark of $30,000. Now, each arm on this thing is maybe a tenth of that. We think that can be even less. That's not due to any one single technology. It involves both hardware and software. So the kind of low-cost arms that we have here, they wouldn't be useful in an industrial setting because traditional control methods that rely on a great deal of precision wouldn't be able to use them. And I think that does make it a lot more practical to think about general purpose robotics today. [00:00:00] S0: For people that would want to be fairly technical about following major milestones that are happening in this field, where does that information show up? What do you read to stay informed about what's going on, or watch?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: For people that would want to be fairly technical about following major milestones that are happening in this field, where does that information show up? What do you read to stay informed about what's going on, or watch? [00:00:00] S1: So a lot of it shows up in research papers. Research papers, unfortunately, are not a very accessible source of information because it takes a bit of care to sort through everything and figure out what is the signal and what does something really mean. Research results are sort of intended for an audience that already understands the starting point from all the past research results. Robotics, and I think technology in general, is one of those things where the public-facing artifacts, the demos and the videos that somebody might post on social media, are often actually not very good for providing a sense for the true underlying state of things because they're sort of meant more as a demonstration of the edge of capability and grounding them. What does the demo really mean requires digging deeper. Probably research papers are the way to go. Sometimes even worse than that, you have to actually go talk to the individual people and find out what the inside story really is. And maybe that's not a great situation to be in, but that's kind of how science works. [00:00:00] S0: As we look forward to the future in your mission, what feels the most uncertain?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: As we look forward to the future in your mission, what feels the most uncertain? [00:00:00] S1: I do think the timeline is uncertain. If anything, my sense of the timeline has gotten more optimistic since we started, but it's uncertain because of the nature of the technology. This is something where there's a bootstrap challenge. Getting to a particular level of usefulness so that robots can be deployed, so they can do useful tasks, so they can start collecting data from open-world settings at scale. Because that's such a sudden event – getting past the activation energy – I think there is a lot of uncertainty about the timing of that. That's exacerbated by the fact that the timeline looks different depending on what kind of technology is deployed. The example I gave before about whether it should be data collection through teleoperation, or data collection with autonomous systems, or something in between – maybe shared autonomy, maybe this coaching thing. Those all change the picture in terms of how deployments work and how in-the-wild data collection works. So because of that, I do think there's quite a bit of uncertainty. [00:00:00] S0: You're in such an interesting position because you're at the center of research. Lots of different kinds of people are talking to you, asking you questions. What are questions that you're surprised people don't ask you?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: You're in such an interesting position because you're at the center of research. Lots of different kinds of people are talking to you, asking you questions. What are questions that you're surprised people don't ask you? [00:00:00] S1: Well, I think the question you asked earlier actually about how somebody should prepare. There's a variant of that question, which would be something like, if I want to start using autonomous robots for a thing, what should I start setting up? Should I set up operations? Should I modify my task in some way so it's more accessible? Should I design new hardware? Maybe I should design new hardware so I can plug your software into it. And I think people make a lot of assumptions about that. For example, one assumption is machine learning requires data, so let me just figure out something that will collect data. That's not often the best assumption because you need the right kind of data. Maybe some data is easy. It's easy to get videos of people doing something, but that doesn't mean that's the right kind of data. And it might be domain-dependent. It might be dependent on the thesis about the technology that will succeed. So I think that people do make a lot of assumptions about that. Not that I necessarily have a better answer for them even if they ask me, but it's something where there's a big space of possibilities. [00:00:00] S0: We talked about these big uncertain long-term timelines. What is the very next thing you are trying to solve?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: We talked about these big uncertain long-term timelines. What is the very next thing you are trying to solve? [00:00:00] S1: A big focus for us right now is actually, better understanding this mid-level reasoning part of the problem. Because we think that we have a pretty good sense for how to acquire low-level physical behaviors, but getting those low-level physical behaviors to generalize requires bringing to bear a lot of this common sense knowledge. The representation of that might be really important. So LLMs make certain kinds of representations very convenient. They make it very convenient to basically turn text into other text, but that's not necessarily the best representation for what an embodied system needs to do. Sometimes it needs to think about things more spatially, sometimes semantically, sometimes other representations, and trying to figure out exactly how to structure that internal thinking process might be a very important question. The answer to that question might be different in the world of embodied foundation models than it is in the world of LLMs. So that's a concrete thing that we're working on now. [00:00:00] S0: If I could somehow get the 100 most informed and active robotics researchers in the room at once and poll them on how certain they are that things will have unlimited capabilities and how soon that might happen, where do you fall in that distribution? [00:00:00] S1: Probably I'm on the optimistic end when it comes to established robotics researchers, and on the pessimistic end relative to robotics entrepreneurs.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: Probably I'm on the optimistic end when it comes to established robotics researchers, and on the pessimistic end relative to robotics entrepreneurs. [00:00:00] S0: I understand the entrepreneur part for sure. You're optimistic by nature. Why are you on the optimistic end of the researcher community? [00:00:00] S1: Robotics has a very long history which has precious few successes, especially when it comes to robotic AI. So I think if we're being honest about it, most robots that are out there doing useful work are still running state-of-the-art technology from the 1980s. Because the robotics problem is hard. Not our fault, it's just a difficult problem. Because of that, I do think that there is good reason for caution – to say that, well, maybe we've made a lot of headway on this part of the problem, but there's many other problems that still remain. Part of why I'm optimistic about this is that I have a sense of what has proven tough for me before, and I can see a lot of the puzzle pieces that I'm imagining could be slotted in to address many of those things. As my co-founder Karol likes to say, when you've climbed the mountain, only then do you see if there's another mountain after it. In robotics, there's been a lot of experience of lots of mountains. Some caution is justified. [00:00:00] S0: Given that endurance is required, who or what most inspires you?