00:11:45 · Aditya Agarwal
[00:11:45] S1: How much of the capabilities increase, whether it be Pi 0.5 or 0.7, is coming in some ways of the afterglow of what we are seeing of, like, you know, kind of frontier, more standard LLM kind of research or, I guess, capabilities versus, hey. It's great that we figured out reasoning models. Maybe we'll figure out how to get, you know, swarms or tens of thousands of essentially agents collaborating to do things and then figuring out how to apply it to robotics. Guess, do you think of that as a kind of like a, you know, concrete strategy at Pi, which is that, like, it's great. We have, you know, OpenAI and Anthropic with, like, hundreds of billions of dollars worth of kind of research capabilities going in, and we'll be able to kind of draft off of that kind of like, you know, for more specific domains.
00:12:34 · Sergey Levine
[00:12:34] S0: So the trouble is that the the ideas are not actually the same. Like, there's analogies to be drawn, but the analogies from my experience often end up being more obvious in hindsight than they are in foresight. So for example, one of the things that turned out to be really important to get great, zero shot generalization in Pi0.7, you know, Pi0.7 is a very powerful model. It can actually generalize to entirely new tasks. You can give it like a kitchen appliance that the robot has never used, and it can kinda muddle through and figure out how to use that kitchen appliance. One of the important things for that is to have the right annotations for your robot examples. Now in hindsight, this is blindingly obvious. Like, the biggest innovation for video models has been prompt expansion. Prompt expansion is what enables video models to actually... Not just to take complex instructions, but to fit complex data. But the trouble is that for a robot task, obviously, you can't do the same thing as you do for video models. Because for video models, a prompt expansion, you tell it, you know, make a video of a dog running through the grass, and it'll say, you know, there's beautiful green grass with, like, dandelions growing on it, and the dog has, like, flowy golden fur. Like, the robot doesn't care about any of that. The robot needs something else. It needs something that is much more tied to the behavior that needs to exhibit. So the kind of... The principle is kind of shared. Why does prompt expansion work? Because more detailed prompts have higher mutual information with the output that you want so that the same principle... You still have to figure out how to instantiate that principle in a robotics context so that actually works.
00:13:57 · Aditya Agarwal
[00:13:57] S1: Talk a little bit about Pi 0.7, your latest model. Right? Like, what is the big delta between zero point five and zero point seven? And insofar as you're able to share, is the
00:14:06 · Sergey Levine
[00:14:06] S0: next one gonna be one dot o? So Pi o seven is actually the latest model that we wrote a blog post about. There is a a secret, even later model that, of course, some people might actually already be using through, you know, a beta test that we have. But Pi0.7 is the last model where we... The most recent model that we actually described in detail. So we we still need to figure out how to present all the newer stuff. Pi0.7 is a model that is really designed around getting zero shot generalization out of the box. So So in some ways, this kind of parallels the evolution that we saw in LLMs where initially we had models that were really good starting points for fine tuning. Then we had models that could be kind of prompt engineered to to do something, and then eventually we had instruction tune models where you could just, like, tell them what what they want. So robotics is not at the stage yet where you can just tell it exactly what you want and will do anything. But, the Pi0.7 model does do a wide range of things out of the box, and you can, explain tasks to it. You can basically coach the robot in language to acquire new skills, and you can do that without additional action supervision. Because the model can break down complex skills into steps, so it's enough to just supervise the steps rather than the actual actions. It also reasons in different modalities, so it can reason in language as well as with images. That's very important because for a lot of tasks, can transfer more knowledge from web scale pretraining in the visual modality as opposed to just in language. So if you, for example, tell the robot, you know, I... I'd like you to push the power button on this appliance, just saying, oh, the power button is, like, three inches to the left of the right edge of the toaster or something that's, like, not so useful, but generating a picture where the robot finger is touching the button, that tells it exactly what to do. So this kind of visual reasoning, it's not that it's better than textual reasoning. It's it's, again, it's like the compliment. It gives you exactly the stuff that is hardest to get out of text. There are a bunch of other things, but the principle really is to move towards this out of the box zero shot generalization without requiring the fine tuning for every task you wanna do.
00:16:09 · Aditya Agarwal, Sergey Levine
[00:16:09] S1: Oh, that's super interesting. So is the is the model then actually doing intermediate visual reasoning steps? [00:16:16] S0: So in the Pi0.7 model, the visual... The the image generation is actually done by a separate model. And this is very similar to how you would... You know, if you load up, like, Gemini or ChatGPT and you ask it to generate a picture, that's going to be calling into a separate model, but the the input to that image generation model is produced by the LM. So it's kind of a similar principle. I think, ultimately, we'll see integration of all these things where there'll be one model that does everything. But, you know, with where the tech is now, the image generation part is usually separate. [00:16:45] S1: What does it mean to be a researcher, like, in kind of the robotics field? What is the day to day of, like, a kind of, like, in a researcher at Physical Intelligence? Like, you know, I think it's interesting. Right? Like, a lot of folks who either have computer science backgrounds or kind of worked as software engineers, you know, a particular mental model of, like, you know, their day to day job. But I'm curious always, like, what does it mean specifically to be a researcher? Like, what is... Like, what what are you doing on a day to day basis?
00:17:15 · Sergey Levine
[00:17:15] S0: So robotics is very, I guess, heterogeneous. Right? So to get a robot to work, you need to do lots of things that will range from, like, tightening the screws that attach the gripper to the arm to figuring out how to train a model on, you know, thousands of GPUs. There isn't really one answer because every day is different, which is one of the things that makes us so much fun. But, you know, honestly, if there was, like, one one trait that I think is really important for a robotics researcher, it's attention to detail. And the reason you need such attention to detail for robotics, even more so than other disciplines, is because the stack is really deep. And if something isn't working, it could be not working at the level of, like, someone didn't attach the connector that goes into the camera properly Mhmm. All the way up to the highest level of, like, well, you know, our model is, you know, it... It's like 16,000,000,000 parameters, but it needs to be 32,000,000,000 parameters, something like that. So it could be anywhere in the stack, and having a really good understanding of many different levels of abstraction is really important.