Invest Like the Best with Patrick O'Shaughnessy
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: I can think of a few reasons. My suspicion is that there is a long tail of challenges that has to do with the interaction of technology and people. Autonomous cars aren't that different in this regard, where getting to a level of comfort with deploying autonomous vehicles on the road was a significant challenge that ran in parallel with getting the technology to that level. Early Tesla self-driving was a bit controversial because it wasn't perfect. There was a question like, are people comfortable with this level of imperfection? Probably there are some tasks for robots where people will be comfortable with something that's not perfect, something that needs to learn from its mistakes. There are some areas where people will not be comfortable. Are you comfortable with it occasionally breaking your dishes? Maybe in a few years it will stop breaking those dishes but maybe in the meantime it's not quite there. Are you comfortable with a robot like that in a home where there's small children? Maybe not and that's okay. I think that figuring out how those factors interact and what that means for the timeline and for how these systems get better with experience, I think that's a tricky question, and I think it needs to be approached very carefully with a lot of sensitivity. There may be some domains where it makes a lot more sense for these systems to be deployed and bootstrapped and collect more data, and maybe other domains require more care. [00:00:00] S0: Can you imagine a purely technical explanation for why something might not work?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Can you imagine a purely technical explanation for why something might not work? [00:00:00] S1: I think the place where I would see the biggest technical risk is dealing with the breadth of different situations. If we're talking about a well-defined but slightly chaotic environment like cleaning hotel rooms or assisting human cooks in a restaurant, I have a very good sense for how to get that under control. If you're imagining a robot going into a home – one place where I can anticipate a challenge is that there are a lot of other unexpected things that can happen and you need a system that's very good at inferring what's going on and adapting to it or reacting intelligently. And I think we have a lot of ideas for how we can approach it, but that is the hardest part of the problem because when you're in a situation where just about anything could happen and you're controlling a physical device that affects the world around it, then you really need to get things right, at least at some level, pretty much in every case. It doesn't mean that you always have to succeed, but it doesn't mean that you always have to do something sensible that people are okay with. And I think there are a lot of really good ideas for how to do that, but that is probably the most challenging part of the equation.
· Patrick O'Shaughnessy
[00:00:00] S0: If I go back to thinking about the right model to think about the Physical Intelligence approach to doing this whole exercise – help me make it as simple as possible. So one might be, we're going to build a whole variety of different kinds of form factors to do a whole variety of different kinds of things and match all this data together and start to experiment with how we can, on evals, make it better. Is that just the simplest way of doing it? Is there an even simpler way? And I'm asking because I'd love to then contrast it with some other approaches that you're interested in that you're not doing that others are doing.
· Sergey Levine, Patrick O'Shaughnessy
[00:00:00] S1: In my mind, the most important thing to get right is to get the system to be general, in particular, to get it to be general with respect to how it can be improved. For example, hand-designed robotic controllers are not very general with respect to how they can be improved because it requires a human engineer to go in and improve it. A learning-based perception system is more general because all it requires is human labelers to go in and label more data. A system that learns autonomously from data that it gathers through its own experience is even more general because you don't need the human labelers. The key is this generality, particularly with respect to improvement. And the decisions we make are, to a very large extent, centered around that. I don't know if the correct design for a robot is to have three cameras. I don't know if it needs a touch sensor. I think we're very agnostic to that. I think we'll try a lot of those different choices. I'm not even sure if in the long run it's going to have a language model. Maybe we'll have some other kind of model that's trained on very diverse data. The key is this level of generality. [00:00:00] S0: What other approaches are the most interesting to you?
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: What other approaches are the most interesting to you? [00:00:00] S1: One thing that's a very important question in this area, and something that I think the research community and the tech community has not fully answered is the dichotomy between different data sources, particularly with respect to real data and simulation. It's a very controversial topic. I have a very strong opinion about it. But I think that – it's worth acknowledging that if we look, for example, at humanoids, you've seen videos of humanoids doing all these acrobatics. There's a particular pipeline that makes that work, which is very heavily reliant on simulation and very light on real-world data. Often actually, zero real-world data. And then there are the approaches that work well for robotic manipulation that often are the opposite. They often use very little simulated data, often use large amounts of real-world data and very large foundation models. And it is surprising that in these two robotic domains, the dominant approaches look so different. It may be that one will win out and there is a particular approach that can handle everything in the long run. Or maybe there's some sort of synthesis of these ideas. That's important. I don't know the answer to that. I have subjective opinions. I think the approach we're taking is a very good one. But I think that it's interesting to look at that and see why is it that these things are so different.
· Patrick O'Shaughnessy, Sergey Levine
[00:00:00] S0: Can you talk about the contrast between cool and useful? The Boston Dynamics robot is very cool. The backflip is super cool. Inverting the body. It all looks really good. I don't know what I need that requires a robot to do a backflip. So I'm curious how you think about optimizing around cool versus useful. [00:00:00] S1: The strategy we've taken is, subject to the constraint that it's useful, make it as cool as possible. We make decisions first and foremost based on our assessment of what will drive the tech forward towards this truly general, broadly applicable robotic foundation model. But in doing that, we try to stress test it against the toughest challenges we can throw at it. The toughest challenges are the ones that look cool. We didn't set out, for example, to build a robot that can make espresso or can fold laundry. But in the process of building these general systems, we figure like these would be particularly challenging, particularly exciting things to try with them to see how far we can push them. [00:00:00] S0: Can you talk about the Robot Olympics?