00:30:13 · Aditya Agarwal, Sergey Levine
[00:30:13] S1: So it's like the bitter lesson applied to, like, robotics. And I think it's actually an interesting answer, which I hadn't thought of, which is you are, I guess... You know, it's easy to kind of conflate humanoids with all robotics, but they're not actually the same thing. It's kind of like the point you're also making. Right? [00:30:28] S0: There's nothing wrong with humanoids. I think humanoids are are amazing, and I I would love to see humanoid robots all around us. But I also think that tech generally doesn't work that way. Take computers for instance. It used to be that computers were like the humanoid companies. You would go to IBM in the fifties and sixties and you would buy a computer and it was a vertically integrated business. IBM comes in, they install the computer, it's a room size thing. The IBM technicians come in, the IBM, you know, software engineers come in, everything is is is IBM. And now, how does it work? Now we have operating systems, we have hardware, we have software. A software company usually doesn't worry about like building their own, you know, CPUs because everything is nicely factorized. And I think we'll see that in robotics. We'll see this kind of explosion of creativity with all sorts of different robots, all sorts of different robotic applications. And the key to that is the right kind of physical intelligence layer that enables that factorization to happen.
00:31:25 · Aditya Agarwal, Sergey Levine
[00:31:25] S1: Yeah. It's interesting. I think one of the... You know, certainly, I I don't know how many of you have done this or remember doing this, but certainly, like, when I was younger, I would actually kind of hand assemble all of my CPUs. You basically go figure out what different components to buy. You assemble it. It was kind of fun. But I think one of the things that has happened even to myself, I haven't hand built a computer, I guess, for a long time. But I think our relationship to hardware over the last fifteen years, ever since this magical device came out, is that our only mental model is that great hardware almost exists in a vertically integrated way. And so it has made us, I think, almost forget that there are different ways of doing this. 85% of the world still uses the Android operating system with its insane variety of different components and parts and configurations. And so I think your answer there is great because both can happen. You will see vertical, you will see horizontal, you'll see innovation at almost every different layer in a bunch of different ways. Do you think that there is still time to create a new robotics foundation model company? [00:32:40] S0: That's an interesting question. It's hard to answer that question because it really depends on kind of an an answer to something that nobody really knows, which is how will the the Ultimate Robot Foundation model work? [00:32:52] S1: Yeah.
00:32:52 · Aditya Agarwal, Sergey Levine
[00:32:52] S1: Yeah. [00:32:52] S0: So I certainly have my own theory about that. I have my road map and and the the kind of research that physical intelligence is pursuing. But if I'm being very upfront about this, like, that is an open science problem. And when you have an open problem like that, it could be the answer somebody nobody expects. Like, you know, certainly, if you if you talk to Yann LeCun, he'll have a very different perspective on this than I do. And maybe he's right. Like, you know, the great thing about science is that it's not a, you know, a purely abstract philosophical debate Yeah. Where we just tell stories. Like, there is an actual truth out there, and somebody's right and somebody's wrong. [00:33:24] S1: Yeah. I'll see you out in the field. Right? So maybe, you know, speaking of Yan, probably the the... I was gonna say easiest, but certainly the... One of the hottest areas of, I would say, company formation and, hence, company, I would say, investments have been world models. World models as applied to robotics, world action models. What's your... Like, how does... And, you know, again, world models is a small term with, like, at least, like, you know, 20 different definitions of how can people end up, like, you know, applying it to the thing that they're doing. But how do you think about almost that kind of, like, strand, if you will, or that that kind of, like, you know, node of kind of the research tree, and how it intersects with how you think about the world at, like, you know, physical intelligence?
00:34:10 · Sergey Levine
[00:34:10] S0: I'll give a little bit of a roundabout answer because it's it's not that there's anything inherently good or bad about world models. It's that the most important thing for building foundation models paradoxically is not the model itself. It's the data. Mhmm. And if you want a foundation model that really works, especially in a new area, the most important thing to figure out is what should the data be, where is it gonna come from, the long term trajectory about how the model will get better and better, meaning where the additional data will come from, and you kinda have to subordinate the other choices to that. So I I talked in the beginning when you asked me about Pi Zero about why it's based on LLM and a diffusion action generation mechanism. I don't think those choices are fundamental. I think the fundamental thing is figuring out the trajectory for data. And I think that trajectory is gonna be a bit complex. Like, it's something where perhaps we start off with a lot of data that is produced by humans, eventually move towards more like intervention style data, eventually more autonomous data. And whether the thing that we train on that data is a video model, a world model, an LLM, a diffusion based action generation mechanism, is sort of a downstream choice. Now, my own hypothesis is that, actually, the ultimate models will do all of those things. Yep. That they will have a world model component because that is how you do reasoning that is below the semantic level of representation. So if you want to, you know, cook a recipe, maybe you think about it semantically. You say, oh, I'm gonna add the vegetables. I'm gonna add the salt. If you are rock climbing, you're not thinking I'm gonna grab the stone that is to the right of that thing over there. You're thinking of a spatial concept. Mhmm. And you probably need a robotic foundation model to be able to imagine things visually, and that is basically a world model. But again, these choices are downstream. The big question is how you're gonna get the data that will enable all this to function. And that's why I think that, you know, some of the world model focus is maybe a little bit distracting in that it kinda shifts more of the focus to what kind of model you have away from the most important question.
00:36:10 · Aditya Agarwal, Sergey Levine
[00:36:10] S1: Every almost successive era of company that comes out of Silicon Valley, they share traits with how the company is structured, what the company does, but they're also kind of different in their own ways. Right? Certainly, was... There were shades of Google that were quite similar to Microsoft, but then also pretty different. Certainly, when I was at Facebook, we were heavily inspired by a lot of the way Google was structured, but then we had our own crazy things like move fast and break things. Then certainly, the Frontier Model Labs look pretty different than the prior generation. So how... What kinds of things perhaps have you been, you know, borrowed from kind of the state of the art in your company? And what have you kind of tried to, you know, in some ways, innovate from first principles in how you run the company? [00:36:59] S0: It's like a little tricky for me to give you an answer that I can put into words because I think that, you know, the truth is that I am extremely happy with how physical intelligence functions, but I don't think I I or even my other cofounders can really take credit for that because it was to some degree an accident. But let me try to, like, reverse it. Like, I I think the truth is, like, we didn't plan on some, like [00:37:19] S1: You have the data. We have to create the
00:37:19 · Aditya Agarwal, Sergey Levine
[00:37:19] S1: You have the data. We have to create the [00:37:21] S0: brilliant strategy. We got, like, super lucky, and now I can sort of look back and reverse engineer maybe some of the things we might have done right. But one thing that I think is really important and uniquely important in robotics that is not, I think, as critical for Google or for OpenAI or or for Meta is that because robotics is so interdisciplinary and because it's so important to have that attention detail and see all parts of the stack, you kinda have to have a culture that both embraces heterogeneous skill sets and at the same time sort of encourages people to gain complete context. And that's a slightly paradoxical thing because specialization might imply that, like, oh, you're the hardware expert. I'm the model expert. You do your thing. I'll do mine, and let's just try to, like, not meet too much in the hallway because we're all doing separate things. And the right thing is, like, a very healthy respect for the expertise that other people have that you don't have, but at the same time, like, a burning desire to really get at least some kind of understanding of everything that's going on. And you you gotta strike that balance somehow. And if you do that, then you get this magical place where everyone has their superpower. Everyone is, like, very respecting of the different skill sets people have, but at the same time, has at least, like, some rough conception of what the whole stack is doing.