Y Combinator Startup Podcast
00:25:22 · Chelsea Finn
But if you have to fine tune a model, you actually aren't getting a general purpose model for the things that you want it to do because you have to fine tune it for each individual thing. And so our first goal is to move towards a single general purpose model that can actually do all of the things that you want it to do. Then the second goal that I mentioned is compositional generalization. And this is inspired by, the DALL E result from 2021. And I think that this was a really important and exciting milestone because of the compositional generalization that it achieved. And specifically, when you have compositional generalization, when you can basically bridge the concept like of an avocado and a chair and show that you kind of can combine those two, it means that the model has at least some kind conceptual understanding of what an avocado is and what a chair is to the point that it can combine them into something that exhibits both concepts at the same time. And then second, it means that you have some degree of data efficiency where your data doesn't need to cover all of the possible combinations of concepts represented in your data. You don't need pictures of avocado chairs in your dataset in order to generate something like this. Or you don't need combinations of other things that you might ask the model to do at, when it's deployed. Now even back in 2021, it wasn't perfect, but these signs of compositional generalization were really exciting for, demonstrating these two attributes of the model. So we have these two goals that we'd like to do. We like it out of the box model and compositional generalization. Now the tried and tested recipe for developing this kind of model is to first take a sufficiently large and diverse dataset and to second, train a model with sufficient capacity. And so what we're gonna do is we're gonna do that. We're gonna try to use all of the data that we have available. This includes really diverse robot demonstration data, including really low quality demonstration data. It's also gonna include policy rollout data, basically attempts from the robot of doing the task. Basically, all of the training data that was used for reinforcement learning for the previous tasks will be included in the training recipe.
00:27:38 · Chelsea Finn
We're also gonna include videos of humans, and we're also going to include data from the web, basically all of the data that we have. And then to train a model with sufficient capacity, of course, we'll train a model that's large enough. But to fit data that's so heterogeneous, we also find it, particularly important to prompt the model with all of the context that it needs in order to predict actions. And we found that this idea was really the key unlock to using this kind of data and this data of this kind of degree of heterogeneity. And specifically, what this looks like is we're gonna train a foundation model that takes as input, the memory that I mentioned before, an instruction of what to do. But it's also gonna take as input a subtask instruction, of kind of what the next immediate thing it should do is. It'll also take as input metadata that kind of indicates the quality of the data, the length of the episodes, and so forth. And this metadata gives it a lot more information about how it should predict the next action. And then optionally, we'll also, train the model with a subgoal image as a prompt to the model. This is essentially saying, like, a few seconds from now, you should try to reach something that looks like this image. With this detailed prompting, we find that the model can really make use of much more heterogeneous data, and I'll show some comparisons later that really, kind of show how important it is. And now to actually deploy this model, we need to then provide things like this subtask instruction and subgoal images. And so with that, we can train, a high level policy that predicts the subtask instruction, like what to do next, what is the next subtask for the task of, cleaning the kitchen. And we'll additionally train a world model to generate images for what the robot should do next as sub goal image conditioning. And so with this, we'll train a single model on, with those attributes on all of the diverse data that we had available. And here are some examples of what that single model can do. So all of these videos are from a single models, specifically a model that we called the pyo seven model. And on left, you can see it doing things like, folding, a collared shirt. On the top right, it's doing a really precise assembly step where it needs to insert a screw and drill that screw into a robot arm.
00:30:00 · Chelsea Finn
And on the bottom right, the robot is replacing a trash bag in a trash can. And so we had two goals at the start of this. The first was to move towards an out of the box model. And even those videos showed that out of the box, the model is able to do quite a bit. But really the key question here is how does this pre trained model compare to the specialists that were trained specifically for coffee making, specifically for box building that I talked about previously? And if we measure the throughput and the success rate of this single pi0.7 model versus the fine tuned pi0.6 model, we see that the... Across the board, the single PIO... Like, pre trained pi0.7 model matches or outperforms the fine tuned specialists that were developed with reinforcement learning post training for those downstream tasks. And so we see that it's able to match the performance of specialists, it also holds for SFT specialists, just RL post trained models as well, suggesting that we do indeed have a single model that can do a lot of different tasks with a really high degree of performance out of the box. Okay, so that was the first goal of out of the box models. And the second goal is compositional generalization. And there's a few different ways to measure this. There's many different ways you might try to combine concepts in robotics. The first test that we wanted to do was to see if a robot could interact with an appliance that's quite rare, like an air fryer. And this is kind of an example. We basically wanted to see if it could open an air fryer, put a sweet potato in the air fryer and close the air fryer. And we picked this because we thought that the data set didn't have any air fryers in it. We didn't intentionally collect any training data with air fryers. After we did some analysis on the data set, we actually found that our data set was so diverse that it did actually have three episodes with air fryers in it. We expect that they likely weren't having an impact and that even if you didn't include those exact three episodes, it likely would still work. But what we found generally is that the robot was able to interact with the appliance that was basically hardly represented at all in the training dataset and combine the skill of interacting with it, like opening it, closing it, and so forth with this object that it hasn't seen before.
00:32:21 · Chelsea Finn
And then after instructing it, like Lucy did, we can train a high level policy to do this task fully autonomously. And you can see, the robot doing that in this video. So that's the first form of compositional generalization. The second compositional generalization test that we wanted to look at is we wanted to see if we could compositionally generalize between tasks and robot platforms. And so we wanted to take a robot platform called the Biarm UR5E robot. It's actually a very large, kind of industrial robot platform. And we want to see if it could fold clothes, despite the fact that we didn't collect any folding data on, this robot platform. So specifically, we had data of, folding clothes, like folding a shirt on the robot platform that's pictured here on the left. And then we wanted to see like out of the box without collecting any folding data on this very different robot platform on the right, could the robot successfully do the task? And what we see in this video is, the... We indeed did see that, it kind of compositionally generalized in this manner. The first time we saw the robot do this, we were like floored, because we... Like, there was no training data for this, for this task. And the robot here is quite different from the other robot, not just in size, but also in the lengths of the linkages of the robot, in the configuration of the joints of the robot, and so forth. Okay, we can, and this is a 1x speed video, so it's not the fastest thing. And obviously, if you haven't seen any training data on something, might not, it's literally the robot's first time folding a shirt, it might take a few attempts. But eventually, it will get to the folded shirt. You can also see the generated subgoal images on the top left. So those are basically the models trying to generate images that will make progress on the folding task. And then those are passes input to the model. And we see the folded shirt here. I think it's going make a couple small corrections at the end to try to make it a little bit smoother. Cool. So the takeaway here is that both in terms of language object interactions and in terms of task robot interactions, we see strong signs of compositional generalization in this model. Okay.
00:34:35 · Chelsea Finn
And then quantitatively, we also see that, as we kind of get to these more advanced models like the pyo seven model, the performance of folding, towels and folding shirts on this platform that hasn't seen before, increases dramatically. And it even approaches the performance of human teleop, despite the fact that we didn't have any robot specific training data, for folding clothes. And then for the last experiment that we did here, I think this is maybe perhaps the most interesting experiment is we wanted to test how important are the two ingredients that I mentioned? How important is diverse data and how important is this sort of like capacity or detailed prompting for the kinds of results that I showed? And so if we remove the most diverse data from the model training shown in like the grayish color, we find that the performance on held out tasks decreases dramatically. Whereas if we, just take out a random 20% of the data that's less diverse than the most diverse subset, the performance, only decreases a little bit. So this suggests that actually having really diverse data, plays an important role in enabling it to generalize to new tasks. And then we tried to also ablate the fact that we are prompting the model with metadata. And for this experiment, we looked at, with and without prompting with metadata. So with prompting is shown in yellow and without prompting is shown in the gray color. And, with prompting, helps, kind of significantly. But the most interesting thing is if you look at, when you add... So this plot is showing as you add more and more data, and specifically as you add more and more low quality data, what is the performance? And without metadata prompting, when you add lower quality data from 80% data to a 100% data, the performance actually decreases, which is perhaps not too surprising because you're adding low quality data to your data mixture. Whereas with the metadata prompting, the performance actually increases when you add that low quality data. So Justin is actually able to make... Get get a lot more juice out of even low quality data, when you include this kind of prompting. Cool.
00:36:43 · Chelsea Finn
So the takeaways here are that, we found that we're able to train a single model to control the robots that matches or exceeds the performance of specialized post trained models, kind of like going from a BERT like pretrained model to a model that really works out of the box like GPT. We also saw strong signs of compositional generalization in a dolly like way. For example, in compositionally generalizing skills applied to appliances and skills applied to new robots in ways that weren't seen in the training data. Cool. And then all the, like, videos and experiments that I showed were just evaluating the model out of the box without any post training. And the paper and the technical report online have a lot more experiments and a lot more details. Okay. So we talked about long term autonomy. We then showed how we can kind of develop that in a single general purpose model. Where are we at now? The first thing that I'll mention is if we go back to the timeline of generalist AI, I think that we now kind of firmly have physical intelligence in the right side of this timeline. We're kind of firmly more in like a GPT and DALL-E like era for robotics and physical intelligence, which is really exciting. And I think that... And we've kind of went there in just a few years. And lastly, we also have these models that are actually deployed in real world circumstances. So the two videos on the top are actually two YC companies, Ultra and Weave, that have taken find, PI models and post train them to do, in deployment to do tasks like folding laundry and packaging in a warehouse. The kind of video on the bottom, bottom left is, the video that I showed previously. And this kind of model works for a really diverse set of robot embodiments. The ones on kind of the top and the left are kind of a more standard bimanual platform, but it also can be adapted to things like drones, quadcopters, surgical robots, and on the bottom right, for things like tractors. So this is really kind of truly showing how physical intelligence can make an impact, not just in demos, in research and so forth, but actually in real world deployment. And I think that we'll, as we go, we'll start to see robots that are, more and more robots actually deployed for real in the physical world with all the advances that we've been seeing over the past few years. Awesome.