Y Combinator Startup Podcast
00:20:39 · Chelsea Finn
Next, it's going to put away the mustard into the fridge, then it will put the dishes away into the cabinet, wash some of the dirty dishes in the sink, and so on. And so, by incorporating memory, it's able to do a task that requires keeping track of all of these different steps, that is done to, clean the kitchen and successfully operate for, ten to fifteen minutes completely autonomously. Great. So those were a couple ingredients for long term autonomy. Now I'd like to build on that and actually take those ingredients and put it into a general purpose model, that can do everything that I showed before, but also can do that in a single model and can do some other things as well. And to think about developing such a general purpose model, I think it's really helpful to contextualize like where robotics is at, within the timeline of other developments in generalist AI. And if we think about how kind of generalist AI systems have evolved over the past, like fifteen years, I think the first major milestone was in 2012, when we saw that a deep learning system trained from scratch. This was the first time that it topped an external benchmark. And all of the previous methods for that benchmark were specifically designed for that application. So all of the previous methods, specifically, this was the ImageNet benchmark, were designed specifically for image classification. Whereas this was the first time that a deep learning based system actually outperformed those more specialist systems. And this is kind of a much more general algorithm that wasn't specifically designed for image recognition. And then just a couple of years later, we found that we weren't just training algorithms from scratch, but we were able to get models like pre trained models that are useful for fine tuning to downstream tasks. And so it became the norm to take a model that was pre trained on ImageNet and then fine tune it on a downstream task. And we actually saw better performance from using that pre trained model like BERT or like an ImageNet pre trained model. And then from there, I think the next big phase and the next big transition in generalist AI models wasn't using pretrained models, but kind of moving from a pretraining fine tuning regime to a regime where we're just using generalist models out of the box. This is with models like the start of GPT-two.
00:23:01 · Chelsea Finn
And then, of course, almost all the models that we interact with today worked just out of the box without fine tuning, at least most of the consumer models. There's actually other models that still use a lot of fine tuning. And then I think one other milestone that I wanna highlight was a milestone in 2021, where we think I saw the first signs of compositional generalization in these models. And one specific instance of that was with DALL E, and I'll talk a little bit more about that in a later slide. So this is how kind of generalist AI has advanced, over the past fifteen years. And then meanwhile, if we kind of think about physical AI, even just three years ago, in 2023, it was extremely common for people working on robotics to collect a bespoke dataset from scratch for an individual project and train from scratch on that dataset. And this is analogous to say, collecting ImageNet from scratch and training ImageNet or training on the dataset that you just collected from scratch on that dataset. And so if you wanna develop a general purpose model, if you have to collect the dataset from scratch for every single project, you're probably not gonna make a lot of progress. And so until just a few years ago, I think we are kind of pretty far on the left of this timeline. And until recently, think that... More recently, I think that we've been kind of in the 2014 phase where we have some good pre trained models, but we haven't really been truly in the regime on the right. So how do we get to that regime on the right? Specifically, how do we develop a single general purpose model that works out of the box and also shows compositional generalization? So this has two goals. The first is out of the box model. This is analogous to going from, like, BERT to GPT. And right now, the best robot performance, if you wanna get, like, the... Your model to perform, the best that it can on a given task, it always requires fine tuning. So some of the videos that I showed at the beginning, they were fine tuned models, to do things like, unlocking a lock. Other work that we've done on measuring human to robot transfer also needed fine tuning to get the best performance. And of course, all of the videos that I showed with RL post training were also fine tuning on an individual task to get the best performance on something like making espresso.
00:25:22 · Chelsea Finn
But if you have to fine tune a model, you actually aren't getting a general purpose model for the things that you want it to do because you have to fine tune it for each individual thing. And so our first goal is to move towards a single general purpose model that can actually do all of the things that you want it to do. Then the second goal that I mentioned is compositional generalization. And this is inspired by, the DALL E result from 2021. And I think that this was a really important and exciting milestone because of the compositional generalization that it achieved. And specifically, when you have compositional generalization, when you can basically bridge the concept like of an avocado and a chair and show that you kind of can combine those two, it means that the model has at least some kind conceptual understanding of what an avocado is and what a chair is to the point that it can combine them into something that exhibits both concepts at the same time. And then second, it means that you have some degree of data efficiency where your data doesn't need to cover all of the possible combinations of concepts represented in your data. You don't need pictures of avocado chairs in your dataset in order to generate something like this. Or you don't need combinations of other things that you might ask the model to do at, when it's deployed. Now even back in 2021, it wasn't perfect, but these signs of compositional generalization were really exciting for, demonstrating these two attributes of the model. So we have these two goals that we'd like to do. We like it out of the box model and compositional generalization. Now the tried and tested recipe for developing this kind of model is to first take a sufficiently large and diverse dataset and to second, train a model with sufficient capacity. And so what we're gonna do is we're gonna do that. We're gonna try to use all of the data that we have available. This includes really diverse robot demonstration data, including really low quality demonstration data. It's also gonna include policy rollout data, basically attempts from the robot of doing the task. Basically, all of the training data that was used for reinforcement learning for the previous tasks will be included in the training recipe.
00:27:38 · Chelsea Finn
We're also gonna include videos of humans, and we're also going to include data from the web, basically all of the data that we have. And then to train a model with sufficient capacity, of course, we'll train a model that's large enough. But to fit data that's so heterogeneous, we also find it, particularly important to prompt the model with all of the context that it needs in order to predict actions. And we found that this idea was really the key unlock to using this kind of data and this data of this kind of degree of heterogeneity. And specifically, what this looks like is we're gonna train a foundation model that takes as input, the memory that I mentioned before, an instruction of what to do. But it's also gonna take as input a subtask instruction, of kind of what the next immediate thing it should do is. It'll also take as input metadata that kind of indicates the quality of the data, the length of the episodes, and so forth. And this metadata gives it a lot more information about how it should predict the next action. And then optionally, we'll also, train the model with a subgoal image as a prompt to the model. This is essentially saying, like, a few seconds from now, you should try to reach something that looks like this image. With this detailed prompting, we find that the model can really make use of much more heterogeneous data, and I'll show some comparisons later that really, kind of show how important it is. And now to actually deploy this model, we need to then provide things like this subtask instruction and subgoal images. And so with that, we can train, a high level policy that predicts the subtask instruction, like what to do next, what is the next subtask for the task of, cleaning the kitchen. And we'll additionally train a world model to generate images for what the robot should do next as sub goal image conditioning. And so with this, we'll train a single model on, with those attributes on all of the diverse data that we had available. And here are some examples of what that single model can do. So all of these videos are from a single models, specifically a model that we called the pyo seven model. And on left, you can see it doing things like, folding, a collared shirt. On the top right, it's doing a really precise assembly step where it needs to insert a screw and drill that screw into a robot arm.
00:30:00 · Chelsea Finn
And on the bottom right, the robot is replacing a trash bag in a trash can. And so we had two goals at the start of this. The first was to move towards an out of the box model. And even those videos showed that out of the box, the model is able to do quite a bit. But really the key question here is how does this pre trained model compare to the specialists that were trained specifically for coffee making, specifically for box building that I talked about previously? And if we measure the throughput and the success rate of this single pi0.7 model versus the fine tuned pi0.6 model, we see that the... Across the board, the single PIO... Like, pre trained pi0.7 model matches or outperforms the fine tuned specialists that were developed with reinforcement learning post training for those downstream tasks. And so we see that it's able to match the performance of specialists, it also holds for SFT specialists, just RL post trained models as well, suggesting that we do indeed have a single model that can do a lot of different tasks with a really high degree of performance out of the box. Okay, so that was the first goal of out of the box models. And the second goal is compositional generalization. And there's a few different ways to measure this. There's many different ways you might try to combine concepts in robotics. The first test that we wanted to do was to see if a robot could interact with an appliance that's quite rare, like an air fryer. And this is kind of an example. We basically wanted to see if it could open an air fryer, put a sweet potato in the air fryer and close the air fryer. And we picked this because we thought that the data set didn't have any air fryers in it. We didn't intentionally collect any training data with air fryers. After we did some analysis on the data set, we actually found that our data set was so diverse that it did actually have three episodes with air fryers in it. We expect that they likely weren't having an impact and that even if you didn't include those exact three episodes, it likely would still work. But what we found generally is that the robot was able to interact with the appliance that was basically hardly represented at all in the training dataset and combine the skill of interacting with it, like opening it, closing it, and so forth with this object that it hasn't seen before.
00:32:21 · Chelsea Finn
And then after instructing it, like Lucy did, we can train a high level policy to do this task fully autonomously. And you can see, the robot doing that in this video. So that's the first form of compositional generalization. The second compositional generalization test that we wanted to look at is we wanted to see if we could compositionally generalize between tasks and robot platforms. And so we wanted to take a robot platform called the Biarm UR5E robot. It's actually a very large, kind of industrial robot platform. And we want to see if it could fold clothes, despite the fact that we didn't collect any folding data on, this robot platform. So specifically, we had data of, folding clothes, like folding a shirt on the robot platform that's pictured here on the left. And then we wanted to see like out of the box without collecting any folding data on this very different robot platform on the right, could the robot successfully do the task? And what we see in this video is, the... We indeed did see that, it kind of compositionally generalized in this manner. The first time we saw the robot do this, we were like floored, because we... Like, there was no training data for this, for this task. And the robot here is quite different from the other robot, not just in size, but also in the lengths of the linkages of the robot, in the configuration of the joints of the robot, and so forth. Okay, we can, and this is a 1x speed video, so it's not the fastest thing. And obviously, if you haven't seen any training data on something, might not, it's literally the robot's first time folding a shirt, it might take a few attempts. But eventually, it will get to the folded shirt. You can also see the generated subgoal images on the top left. So those are basically the models trying to generate images that will make progress on the folding task. And then those are passes input to the model. And we see the folded shirt here. I think it's going make a couple small corrections at the end to try to make it a little bit smoother. Cool. So the takeaway here is that both in terms of language object interactions and in terms of task robot interactions, we see strong signs of compositional generalization in this model. Okay.