00:03:45 · Aditya Agarwal, Sergey Levine
[00:03:45] S1: They will fall on their faces. [00:03:46] S0: They will not have a good time. Yeah. Three kids. Yes. And they're certainly not gonna figure out lots of physical So the missing piece, and this is coming to the end of the story, is that the missing piece is not just robots collecting experience, but also endowing them with prior knowledge. And prior knowledge is something that, in the meantime, the AI community had figured out a really great way to get from data on the web by training foundation models. [00:04:06] S1: Mhmm. [00:04:06] S0: So that idea of putting lots of robots in a room, I think there is something to it when it is combined with the right source of prior knowledge from foundation models. And perhaps, you know, I'm still not done with this adventure of trying to figure out how to replicate human intelligence, but perhaps now I'm on to something that feels more promising to me than the video game stuff from the beginning. [00:04:24] S1: Before I kind of ask you to take us inside some of the work that physical intelligence has done, Walk us through a little bit about the early stages of starting the company. How did you go from being a postdocbeing in the research lab to getting the crew together and really having, in some ways, the momentum and I'm just going to go for it. Was there a particular either insight about the state of the technology, the state of LLMs, or some kind of research breakthrough that you had? How did you know that it was time?
00:05:05 · Sergey Levine
[00:05:05] S0: Well, so to be... To to give credit where credit is due, it was Karol Hausman's idea. So Karol is the the CEO of Physical Intelligence, and I very, very vividly remember that I was having dinner with him. And he he said to me something like, well, there's going to be some kind of robotics research lab that really makes, like, the really big breakthroughs in robotics, and there's... I think there's probably only gonna be one of those, so we should try to start something. Mhmm. Now I I I don't think he phrased it with this sort of arrogant under... Undertone of, like, we're gonna... We're we're the ones, but more like, you know, someone... Someone's gotta do it, so to speak. And he had really the impetus to get this thing launched. But I think we were also both very fortunate to have a really good network of like minded researchers that we knew that also felt that it was time, including our cofounders, all the folks that joined the company from the beginning. Because we we had been working on this for a long time in academia, at Google DeepMind, some of us worked at other places. And I think that really what made the the early days of physical intelligence, which were very difficult, much more manageable is having the right group of friends around us who were like minded in the mission and also willing to suffer through a lot of the challenges. And I think, you know, to anybody thinking of starting a company, think having the right people early on in your journey, that's like really the most important thing.
00:06:30 · Aditya Agarwal
[00:06:30] S1: That's a great answer. So maybe, you know, this is gonna be perhaps a new one for us, but I wanted to... You know, I figured there's no one better to ask Sergey than take this room inside the first model that you guys released, Pi Zero. Right? And this was kind of an interesting architecture. It was obviously kind of a pre trained vision language backbone using a bunch of, I would say, state of the art LLMs at that point. You had some interesting things around flow matching, I would say, on top of that. But you made a bunch of interesting architectural choices in there for it to be the first, I would say, release from the lab. Take us through it. Kind of like, take us a little bit through kind of like some of the underlying, almost, insights and perhaps motivations for the decisions that you made there.
00:07:22 · Sergey Levine
[00:07:22] S0: Yeah. So a robot foundation model needs to be able to take as input the things that the robot is observing. So those are typically going to be camera images. You could not use depth. You could use LIDAR, but in the end, it's some sort of, like, image structured data. And it needs to produce actions. And, again, you could have a few choices for actions, but they're all kind of the same. Something continuous that represents either positions of the robot's end effectors or joint angles or joint motor commands, torques, whatever. The... Basically, a vector of numbers, roughly, you know, 30 to 50 numbers. So that's the basics. Now if you want it to be a foundation model, you need one more thing, which is you need to tell it what you want it to do. And, you know, the most obvious way to do that is to tell it in language, So therefore, it also gets language as input. There are other ways to tell it what to do as well that are a little bit more involved, but language is a fairly basic one. So that gives us, like, the starting outlines of this thing. It needs to produce continuous actions. It needs to take in lots of images, and it needs to take in language. So now you have a few choices. One very obvious choice is train the model from scratch on all of your robot data. So that's basic imitation learning. The trouble with this is it's very difficult from just the robot data you have to get everything that you need for the system to be successful. So that's that prior knowledge piece that I mentioned before. So then the next logical step is to say, well, if you need prior knowledge, let's start it off from a model that kind of provides the complement to your robot data. So you don't wanna start from a model that's as close as possible to a robot, because you need to give it the stuff that the robot data doesn't have, and that's the prior knowledge. So that's why the LLM is a really good starting point, because the LLM kinda has like exactly what you lack. It has lots of knowledge, but it is not grounded. So the LLM is a good starting point. Now I don't think that's the only way to do it. I think that once you have that, that... That's kinda like the outlines of a thing, but then you can fill in... You can color in the outlines and put in more stuff.
00:09:11 · Sergey Levine, Aditya Agarwal
So you could, for example, and I think we've already seen some of this, and we'll see more of it in the future. You can include other data and training. You can include videos. You can include, you know, data from humans, all this other stuff. But the Pi Zero model really was not intended to be the ultimate thing that does everything. It was intended to be the outline. And once we draw the outline, then we can color in the stuff inside. We can add in reasoning, chain of thought, other data sources, and so on. That's lot of what we've been doing since then. [00:09:38] S1: Okay. So maybe to move on from Pi Zero to Pi 0.5, I think one of the cool things about the second model... It was the second model. Right? Yep. Was that it started to demonstrate some amount of generalization. It started to kind of like work in environments where it had not been explicitly trained. It cleaned kitchens and bedrooms that it had never seen before. Also, it just did more long horizon kind of like work that was more dexterous. So what is the key ingredient that kind of unlocked the capability kind of, like, step function change or, I guess, generalization from Pi zero to Pi 0.5?
00:10:14 · Sergey Levine
[00:10:14] S0: So there are quite a few things, but, you know, a a few of the really important principles for robot foundation models. These principles are not actually all that surprising because they look just like the principles for other foundation models, but they have to be mapped onto the robotics problem in the right way. So the the first and most obvious one is that you need the right data, and getting the right data is not is not just an industrial problem, it is also a research problem. In fact, that's why it's such a difficult problem is that it combines, you know, economics, industrial scale operations, and research. You have to figure out the right thing, scale it up in the right way. You usually can't answer what the right thing is until you scale it up to some level. So that's the data piece. You also need the model to be able to understand complex instructions and break them down into manageable pieces. So this is where the LLM part actually becomes really important. So the Pi0.5 model actually had an internal reasoning component. So if you tell it to clean a kitchen, it doesn't just try to map clean the kitchen to robot actions. It breaks it down into thoughts, and the thoughts might be like, pick up the dish, pick up the the towel, wipe the spill, and then from there grounds that into actions. And this is tremendously powerful. Like, this is why... This is this is the same reasons why coding agents work. Right? Coding agents today, you don't just tell them what you want and they spit out lots of code. They actually make a plan. They break it down into steps. And Pi0.5 was our first model that did something like that. And since then, we've extended it. We can do reasoning in other modalities. We can do reasoning of variable length and so on. And these are very important things for doing really complex tasks.