00:14:06 · Sergey Levine
[00:14:06] S0: next one gonna be one dot o? So Pi o seven is actually the latest model that we wrote a blog post about. There is a a secret, even later model that, of course, some people might actually already be using through, you know, a beta test that we have. But Pi0.7 is the last model where we... The most recent model that we actually described in detail. So we we still need to figure out how to present all the newer stuff. Pi0.7 is a model that is really designed around getting zero shot generalization out of the box. So So in some ways, this kind of parallels the evolution that we saw in LLMs where initially we had models that were really good starting points for fine tuning. Then we had models that could be kind of prompt engineered to to do something, and then eventually we had instruction tune models where you could just, like, tell them what what they want. So robotics is not at the stage yet where you can just tell it exactly what you want and will do anything. But, the Pi0.7 model does do a wide range of things out of the box, and you can, explain tasks to it. You can basically coach the robot in language to acquire new skills, and you can do that without additional action supervision. Because the model can break down complex skills into steps, so it's enough to just supervise the steps rather than the actual actions. It also reasons in different modalities, so it can reason in language as well as with images. That's very important because for a lot of tasks, can transfer more knowledge from web scale pretraining in the visual modality as opposed to just in language. So if you, for example, tell the robot, you know, I... I'd like you to push the power button on this appliance, just saying, oh, the power button is, like, three inches to the left of the right edge of the toaster or something that's, like, not so useful, but generating a picture where the robot finger is touching the button, that tells it exactly what to do. So this kind of visual reasoning, it's not that it's better than textual reasoning. It's it's, again, it's like the compliment. It gives you exactly the stuff that is hardest to get out of text. There are a bunch of other things, but the principle really is to move towards this out of the box zero shot generalization without requiring the fine tuning for every task you wanna do.
00:16:09 · Aditya Agarwal, Sergey Levine
[00:16:09] S1: Oh, that's super interesting. So is the is the model then actually doing intermediate visual reasoning steps? [00:16:16] S0: So in the Pi0.7 model, the visual... The the image generation is actually done by a separate model. And this is very similar to how you would... You know, if you load up, like, Gemini or ChatGPT and you ask it to generate a picture, that's going to be calling into a separate model, but the the input to that image generation model is produced by the LM. So it's kind of a similar principle. I think, ultimately, we'll see integration of all these things where there'll be one model that does everything. But, you know, with where the tech is now, the image generation part is usually separate. [00:16:45] S1: What does it mean to be a researcher, like, in kind of the robotics field? What is the day to day of, like, a kind of, like, in a researcher at Physical Intelligence? Like, you know, I think it's interesting. Right? Like, a lot of folks who either have computer science backgrounds or kind of worked as software engineers, you know, a particular mental model of, like, you know, their day to day job. But I'm curious always, like, what does it mean specifically to be a researcher? Like, what is... Like, what what are you doing on a day to day basis?
00:17:15 · Sergey Levine
[00:17:15] S0: So robotics is very, I guess, heterogeneous. Right? So to get a robot to work, you need to do lots of things that will range from, like, tightening the screws that attach the gripper to the arm to figuring out how to train a model on, you know, thousands of GPUs. There isn't really one answer because every day is different, which is one of the things that makes us so much fun. But, you know, honestly, if there was, like, one one trait that I think is really important for a robotics researcher, it's attention to detail. And the reason you need such attention to detail for robotics, even more so than other disciplines, is because the stack is really deep. And if something isn't working, it could be not working at the level of, like, someone didn't attach the connector that goes into the camera properly Mhmm. All the way up to the highest level of, like, well, you know, our model is, you know, it... It's like 16,000,000,000 parameters, but it needs to be 32,000,000,000 parameters, something like that. So it could be anywhere in the stack, and having a really good understanding of many different levels of abstraction is really important.
00:18:14 · Aditya Agarwal
[00:18:14] S1: You know, one of the kind of the classic question that I get from people outside of Silicon Valley is, why are AI researchers so highly paid? And the answer is pretty simple. There's just not that many of them, Right? Like, relative to kind of like the impact of these foundation models is the basic supply demand problem. So I'm curious how many great robotics researchers do you think there are in the world right now that could have that level of attention to detail? And, you know, could like... Is it like a few 100? If there's just not that many people who have done this. Right?
00:18:47 · Sergey Levine, Aditya Agarwal
[00:18:47] S0: It... It's a tricky question because I think there's a big difference between the... You know, saying, like, how many people have a robotics PhD from a top university university and, like, a publication record and the willingness to get stuff to actually work and have that attention to detail as opposed to how many people Okay. Yeah, have the the mindset necessary to get this right. And there's a big difference. Like, you know, there are certainly people that have strong publication record, PhD from top school, but just don't have quite the mindset to go to work. And on the flip side, there are people that have the right mindset, don't have the credentials, but have the ability to get there, you know, in the right environment. So it's a very hard question to answer, and, you know, probably the main lesson for me is, like, not to prejudge these things and to really pay attention to how somebody approaches the problem, what is their, you know, their actual thought process, and what is their, I guess, like, life philosophy for lack of another word. [00:19:39] S1: So I guess maybe to to kinda put you on the spot here, like, do you think it is possible to get hired at Physical Intelligence as a robotics researcher without having a great publication record, but having a great mindset? It's like, listen. I I am a great software engineer. I care about this. I can learn. I have attention to detail. I've kinda done, you know, kind of at least the basics of how all these models work. Do you think that's, like, do you think that's a possibility?
00:19:39 · Aditya Agarwal, Sergey Levine
[00:19:39] S1: So I guess maybe to to kinda put you on the spot here, like, do you think it is possible to get hired at Physical Intelligence as a robotics researcher without having a great publication record, but having a great mindset? It's like, listen. I I am a great software engineer. I care about this. I can learn. I have attention to detail. I've kinda done, you know, kind of at least the basics of how all these models work. Do you think that's, like, do you think that's a possibility? [00:20:02] S0: Yeah. Absolutely. I think the biggest challenge with that is is the the signaling aspect of it. Right? That in any such situation, if you want a job, the other party needs to have some way to evaluate [00:20:14] S1: Yes.