00:00:00 · Sergey Levine, Aditya Agarwal
[00:00:00] S0: The universe is really big, and almost none of the universe is inside computers. Almost none of it is on the Internet. If you want a machine that understands the universe, that understands physical phenomena, it has to be a robot. Pi0.7 is a model that is really designed around getting zero shot generalization out of the box. When you're faced with a really difficult intelligence problem, somewhat counterintuitively, making the problem as big as possible actually makes it easier. [00:00:29] S1: Sergey, walk us through your kind of path to obviously, you know, being at Stanford, being at Berkeley, kinda doing a bunch of computer graphic stuff to now kinda starting physical intelligence. Like, what is the through line there?
00:00:41 · Sergey Levine
[00:00:41] S0: So I... Initially, when I was in college, what I really wanted to do is I wanted to make video games, which is, you know, maybe not an uncommon thing that people want to do when they're in college. I spent a summer interning at a video game company that I won't name. I didn't have such a great time and decided that, you know, contrary to my very romantic notion of what making video games was like, it's actually, you know, a corporate job like any other with all sorts of corporate job problems and decided that maybe what I should do is try something more difficult. Maybe more difficult at technical level and maybe easier at the organizational level, which is to do research. But since I really liked the idea of, you know, interactive virtual environments, I worked on computer graphics. And in computer graphics, you know, computer graphics at some level is about simulating the universe. And one of the hardest parts of the universe to simulate is the is the brain. So if you want your virtual environment to be populated by other characters that behave like people, then you need to replicate some aspect of human intelligence. So I worked on character animation. And I kind of realized that towards the end of my PhD that working on character animation is fun, but, you know, if the best that you can do with character animation is like make Hollywood movies a little nicer, make video games a little more fun, Perhaps there's something greater to aspire to in life. And I figured, well, if this stuff that I'm building makes virtual characters perform complex physical tasks, perhaps it could also work for getting physical robots to perform complex tasks. So I went and interviewed with Pieter Abbeel at UC Berkeley for a post doc position, because I figured I need a little bit more education to figure out how to actually do this. Peter called me a couple days later and told me that he already felt like his lab was doing too many different things, and all this weird character animation stuff I talked about doesn't really fit. But then a week later, I don't know what happened, he called me again and said that he wanted to offer me a postdoc job.
00:02:26 · Sergey Levine
So I never actually found out why there was that change of heart, but I'm very glad that it happened because then I got to learn how to actually make stuff work on robots, which was in some ways like surprisingly similar, in other ways surprisingly different. And, you know, from there, I I kind of realized that there's this wide open world robotics where I could actually apply some of the stuff I'd figured out, but those were some some new challenges, and that was a lot of fun. After that, I worked at Google for a while. The right idea, but also a bit of a naive idea, which is that to make robotics work, you need to get the robots to collect lots of data. And what we did is we set up a whole bunch of robots in a lab. It was actually a repurposed conference room, and we had them just grasping objects all the time. Why grasping objects? Well, because I figured that's like a very basic building block for robotic behavior, so you could just launch lots of robots, have them try it, and they'll collect experience. And that kind of worked, but it hit a wall, and I... And this was kind of the mistake in hindsight. The wall that it hit is you could get robots that are really good at grasping stuff, but you couldn't get them to be good at anything else. And the reason is that all the other stuff doesn't just take lots of experience, it also takes lots of knowledge. And having the robots discover that knowledge from scratch is very difficult. I mean, even people wouldn't discover from scratch. If you take a newborn infant, you lock them in a room by themselves with nobody around, they're... Well, first thing
00:03:45 · Aditya Agarwal, Sergey Levine
[00:03:45] S1: They will fall on their faces. [00:03:46] S0: They will not have a good time. Yeah. Three kids. Yes. And they're certainly not gonna figure out lots of physical So the missing piece, and this is coming to the end of the story, is that the missing piece is not just robots collecting experience, but also endowing them with prior knowledge. And prior knowledge is something that, in the meantime, the AI community had figured out a really great way to get from data on the web by training foundation models. [00:04:06] S1: Mhmm. [00:04:06] S0: So that idea of putting lots of robots in a room, I think there is something to it when it is combined with the right source of prior knowledge from foundation models. And perhaps, you know, I'm still not done with this adventure of trying to figure out how to replicate human intelligence, but perhaps now I'm on to something that feels more promising to me than the video game stuff from the beginning. [00:04:24] S1: Before I kind of ask you to take us inside some of the work that physical intelligence has done, Walk us through a little bit about the early stages of starting the company. How did you go from being a postdocbeing in the research lab to getting the crew together and really having, in some ways, the momentum and I'm just going to go for it. Was there a particular either insight about the state of the technology, the state of LLMs, or some kind of research breakthrough that you had? How did you know that it was time?
00:05:05 · Sergey Levine
[00:05:05] S0: Well, so to be... To to give credit where credit is due, it was Karol Hausman's idea. So Karol is the the CEO of Physical Intelligence, and I very, very vividly remember that I was having dinner with him. And he he said to me something like, well, there's going to be some kind of robotics research lab that really makes, like, the really big breakthroughs in robotics, and there's... I think there's probably only gonna be one of those, so we should try to start something. Mhmm. Now I I I don't think he phrased it with this sort of arrogant under... Undertone of, like, we're gonna... We're we're the ones, but more like, you know, someone... Someone's gotta do it, so to speak. And he had really the impetus to get this thing launched. But I think we were also both very fortunate to have a really good network of like minded researchers that we knew that also felt that it was time, including our cofounders, all the folks that joined the company from the beginning. Because we we had been working on this for a long time in academia, at Google DeepMind, some of us worked at other places. And I think that really what made the the early days of physical intelligence, which were very difficult, much more manageable is having the right group of friends around us who were like minded in the mission and also willing to suffer through a lot of the challenges. And I think, you know, to anybody thinking of starting a company, think having the right people early on in your journey, that's like really the most important thing.
00:06:30 · Aditya Agarwal
[00:06:30] S1: That's a great answer. So maybe, you know, this is gonna be perhaps a new one for us, but I wanted to... You know, I figured there's no one better to ask Sergey than take this room inside the first model that you guys released, Pi Zero. Right? And this was kind of an interesting architecture. It was obviously kind of a pre trained vision language backbone using a bunch of, I would say, state of the art LLMs at that point. You had some interesting things around flow matching, I would say, on top of that. But you made a bunch of interesting architectural choices in there for it to be the first, I would say, release from the lab. Take us through it. Kind of like, take us a little bit through kind of like some of the underlying, almost, insights and perhaps motivations for the decisions that you made there.