Y Combinator Startup Podcast
00:34:35 · Chelsea Finn
And then quantitatively, we also see that, as we kind of get to these more advanced models like the pyo seven model, the performance of folding, towels and folding shirts on this platform that hasn't seen before, increases dramatically. And it even approaches the performance of human teleop, despite the fact that we didn't have any robot specific training data, for folding clothes. And then for the last experiment that we did here, I think this is maybe perhaps the most interesting experiment is we wanted to test how important are the two ingredients that I mentioned? How important is diverse data and how important is this sort of like capacity or detailed prompting for the kinds of results that I showed? And so if we remove the most diverse data from the model training shown in like the grayish color, we find that the performance on held out tasks decreases dramatically. Whereas if we, just take out a random 20% of the data that's less diverse than the most diverse subset, the performance, only decreases a little bit. So this suggests that actually having really diverse data, plays an important role in enabling it to generalize to new tasks. And then we tried to also ablate the fact that we are prompting the model with metadata. And for this experiment, we looked at, with and without prompting with metadata. So with prompting is shown in yellow and without prompting is shown in the gray color. And, with prompting, helps, kind of significantly. But the most interesting thing is if you look at, when you add... So this plot is showing as you add more and more data, and specifically as you add more and more low quality data, what is the performance? And without metadata prompting, when you add lower quality data from 80% data to a 100% data, the performance actually decreases, which is perhaps not too surprising because you're adding low quality data to your data mixture. Whereas with the metadata prompting, the performance actually increases when you add that low quality data. So Justin is actually able to make... Get get a lot more juice out of even low quality data, when you include this kind of prompting. Cool.
00:36:43 · Chelsea Finn
So the takeaways here are that, we found that we're able to train a single model to control the robots that matches or exceeds the performance of specialized post trained models, kind of like going from a BERT like pretrained model to a model that really works out of the box like GPT. We also saw strong signs of compositional generalization in a dolly like way. For example, in compositionally generalizing skills applied to appliances and skills applied to new robots in ways that weren't seen in the training data. Cool. And then all the, like, videos and experiments that I showed were just evaluating the model out of the box without any post training. And the paper and the technical report online have a lot more experiments and a lot more details. Okay. So we talked about long term autonomy. We then showed how we can kind of develop that in a single general purpose model. Where are we at now? The first thing that I'll mention is if we go back to the timeline of generalist AI, I think that we now kind of firmly have physical intelligence in the right side of this timeline. We're kind of firmly more in like a GPT and DALL-E like era for robotics and physical intelligence, which is really exciting. And I think that... And we've kind of went there in just a few years. And lastly, we also have these models that are actually deployed in real world circumstances. So the two videos on the top are actually two YC companies, Ultra and Weave, that have taken find, PI models and post train them to do, in deployment to do tasks like folding laundry and packaging in a warehouse. The kind of video on the bottom, bottom left is, the video that I showed previously. And this kind of model works for a really diverse set of robot embodiments. The ones on kind of the top and the left are kind of a more standard bimanual platform, but it also can be adapted to things like drones, quadcopters, surgical robots, and on the bottom right, for things like tractors. So this is really kind of truly showing how physical intelligence can make an impact, not just in demos, in research and so forth, but actually in real world deployment. And I think that we'll, as we go, we'll start to see robots that are, more and more robots actually deployed for real in the physical world with all the advances that we've been seeing over the past few years. Awesome.
00:39:05 · Chelsea Finn, S1
So the last thing that I'll mention shamelessly is that, we are hiring at Physical Intelligence. So if you're excited about some of the stuff that I talked about, we encourage you to take a look at some of the open roles and apply. And yeah, definitely have time for questions and happy to get all your thoughts. Thanks. [00:39:24] S1: Okay, so the first question is, how far away are we from a ChatGPT moment for robotics and what will that look like? [00:39:31] S0: So I'll start with the second part actually, which is that I'm not sure it will really look like the ChatGPT moment that we saw in language models, which is that with something like ChatGPT, we saw like, it passed like what, a million new users in five days. I think that the distribution channel for physical models is gonna be slower, unfortunately, because you actually need a physical robot there. Mean, I think that we've seen for something like Waymo, it's actually been incredible to see the rollout that they've had, but it still takes time, to actually deploy things on physical devices. So I don't know if we'll have a single moment that has the distribution that ChatGPT had. At the same time, in terms of the capabilities of these models, I think that we are really starting to get to the point where these models are actually useful in the real world. And I think that getting to the kind of the capabilities of ChatGPT, I I think is, yeah, very much on the horizon in the next few years. Cool.
00:39:31 · Chelsea Finn, S1
[00:39:31] S0: So I'll start with the second part actually, which is that I'm not sure it will really look like the ChatGPT moment that we saw in language models, which is that with something like ChatGPT, we saw like, it passed like what, a million new users in five days. I think that the distribution channel for physical models is gonna be slower, unfortunately, because you actually need a physical robot there. Mean, I think that we've seen for something like Waymo, it's actually been incredible to see the rollout that they've had, but it still takes time, to actually deploy things on physical devices. So I don't know if we'll have a single moment that has the distribution that ChatGPT had. At the same time, in terms of the capabilities of these models, I think that we are really starting to get to the point where these models are actually useful in the real world. And I think that getting to the kind of the capabilities of ChatGPT, I I think is, yeah, very much on the horizon in the next few years. Cool. [00:40:31] S1: The second question is when should a small team switch from scaling per site models to a generalist policy and what does that transition actually look like? What signals tell you it's time?
00:40:42 · Chelsea Finn
[00:40:42] S0: So that's a good question. I think that the... I mean, at the very least, I actually think that just starting with the generalist policy and then fine tuning it, even like right off the bat, can be really effective. And we've, And fortunately, a lot of generalist policies, like they're actually really powerful generalist policies that are open source. And so the Pi0 and Pi0.5 models are open source, for example, and we've seen a lot of people, get a lot of use out of those models already. And we're also working with a lot of partners like the tractor company, like Ultra, like Weave, to develop to take our most recent models and get even more juice out of them, even more powerful models for their own applications. So even right off the bat, I think that you can use them. The only scenario in which I actually wouldn't use them is if you're really in a really constrained environment. I've talked to some folks working on surgical robots that are in an Operating Room in the basement with no internet connection and a really bad GPU. And so sometimes it's just really hard to use a larger model, but you still can't do local inference on a workstation with these models. And so I think that, yeah, right away, just taking Pi0.5 or your favorite model and fine tuning it is the way to go. And I think that we'll see lots of these small companies. And I think there's, like, yeah, so much work to do in terms of actually getting these robots to work, with this technology in the real world. Great.
00:42:12 · S1
[00:42:12] S1: The next question is, given how fast robotics is moving in industry, what are the real advantages and drawbacks of doing a PhD today, especially for someone who wants to go into industry afterwards?