Y Combinator Startup Podcast
00:16:24 · Chelsea Finn
And for the Espresso task, if we look specifically at the success rate, we achieved a 90% or over 90% success rate on making Espresso. So the takeaways for this part is that we we can develop a scalable recipe for high reliability of complex robotic manipulation tasks. And we saw, in this case, a two x higher throughput from using experience and interventions. But most importantly, we saw how we can achieve long term autonomy in real workflows that people actually care about in the real world. And this is what it's going to take, I think, for robots to be useful in the real world. Now there's also a lot more work and a lot more opportunities. The... We actually only ran a few iterations of improvement of this algorithm. And with more iterations, we should be able to see even greater improvement or even greater reliability. And even with this improvement, the robot still makes mistakes. It's also still slower than people. And so, there's many... Like a ton of room for improvement for developing even more powerful recipes. So we've seen long term autonomy, for these different workflows. But there's actually one more ingredient that I'd like to talk about for enabling robots to be autonomous and useful for long periods of time. And that ingredient is memory. So you might be surprised to hear that most state of the art foundation models for robotics have no memory or no context. They're just operating on the current sensor observations, the current camera readings, and predicting actions based off of that. And you actually can, do short motor skills. You can do repetitive tasks without memory. The videos that I showed before didn't have any context either. But if you wanna do a long task that involves multiple different steps in sequence, then memory is critical for tracking progress of the steps that you've completed so far. So if it's critical for doing these kinds of long horizon tasks, then why don't these models have any context or memory?
00:18:22 · Chelsea Finn
There's a couple of reasons for this that are technical and I'll talk through one of them, which is that if you naively approach memory, and try to feed in context, like past video to a robot foundation model, say that you would just pass in ten seconds of video, maybe this ten seconds of video is sampled at 50 Hertz, which is a common control frequency in robotics, and you feed in all four camera streams on the robot, and you use around kind of two fifty six tokens per image, this corresponds to passing in half a million tokens into your model, which is a lot of tokens. And trying to do that in real time, into your model right now is quite challenging. Even if you subsample to one frame per second, you're still gonna be passing in 10,000 tokens into your model, which at least right now is prohibitively expensive for these models, and that's still only ten seconds of memory. So, I don't have time to go into the technical details of exactly what we did here, but, we also developed a solution for this context problem. And specifically, we developed a system that has memory at multiple timescales. The first is a short term video memory that has about ten seconds of video memory, but is done so and computed much more efficiently than naively passing it into the model. And then for longer memory, for memory that spans multiple minutes or multiple hours, we don't necessarily need video of exactly what happened in that past history. And so instead, we represent memory for those parts in text, where we summarize what happened, in text based and then incorporate that much more compressed textual summary of what happened over the past ten, fifteen minutes into the model as well. And with this sort of kind of memory at multiple different timescales, we're able to enable robots to do tasks that can operate for ten or fifteen minutes at a time completely autonomously. And what's different from the previous slide, or the kind of what I showed previously is that this task isn't repetitive. So this is gonna be a ten to fifteen minute task that's involved cleaning a kitchen. And the robot isn't just repeatedly making espresso over and over again. What it involves is wiping the counter with a sponge, kind of then drying the counter with a paper towel, throwing away the paper towel.
00:20:39 · Chelsea Finn
Next, it's going to put away the mustard into the fridge, then it will put the dishes away into the cabinet, wash some of the dirty dishes in the sink, and so on. And so, by incorporating memory, it's able to do a task that requires keeping track of all of these different steps, that is done to, clean the kitchen and successfully operate for, ten to fifteen minutes completely autonomously. Great. So those were a couple ingredients for long term autonomy. Now I'd like to build on that and actually take those ingredients and put it into a general purpose model, that can do everything that I showed before, but also can do that in a single model and can do some other things as well. And to think about developing such a general purpose model, I think it's really helpful to contextualize like where robotics is at, within the timeline of other developments in generalist AI. And if we think about how kind of generalist AI systems have evolved over the past, like fifteen years, I think the first major milestone was in 2012, when we saw that a deep learning system trained from scratch. This was the first time that it topped an external benchmark. And all of the previous methods for that benchmark were specifically designed for that application. So all of the previous methods, specifically, this was the ImageNet benchmark, were designed specifically for image classification. Whereas this was the first time that a deep learning based system actually outperformed those more specialist systems. And this is kind of a much more general algorithm that wasn't specifically designed for image recognition. And then just a couple of years later, we found that we weren't just training algorithms from scratch, but we were able to get models like pre trained models that are useful for fine tuning to downstream tasks. And so it became the norm to take a model that was pre trained on ImageNet and then fine tune it on a downstream task. And we actually saw better performance from using that pre trained model like BERT or like an ImageNet pre trained model. And then from there, I think the next big phase and the next big transition in generalist AI models wasn't using pretrained models, but kind of moving from a pretraining fine tuning regime to a regime where we're just using generalist models out of the box. This is with models like the start of GPT-two.
00:23:01 · Chelsea Finn
And then, of course, almost all the models that we interact with today worked just out of the box without fine tuning, at least most of the consumer models. There's actually other models that still use a lot of fine tuning. And then I think one other milestone that I wanna highlight was a milestone in 2021, where we think I saw the first signs of compositional generalization in these models. And one specific instance of that was with DALL E, and I'll talk a little bit more about that in a later slide. So this is how kind of generalist AI has advanced, over the past fifteen years. And then meanwhile, if we kind of think about physical AI, even just three years ago, in 2023, it was extremely common for people working on robotics to collect a bespoke dataset from scratch for an individual project and train from scratch on that dataset. And this is analogous to say, collecting ImageNet from scratch and training ImageNet or training on the dataset that you just collected from scratch on that dataset. And so if you wanna develop a general purpose model, if you have to collect the dataset from scratch for every single project, you're probably not gonna make a lot of progress. And so until just a few years ago, I think we are kind of pretty far on the left of this timeline. And until recently, think that... More recently, I think that we've been kind of in the 2014 phase where we have some good pre trained models, but we haven't really been truly in the regime on the right. So how do we get to that regime on the right? Specifically, how do we develop a single general purpose model that works out of the box and also shows compositional generalization? So this has two goals. The first is out of the box model. This is analogous to going from, like, BERT to GPT. And right now, the best robot performance, if you wanna get, like, the... Your model to perform, the best that it can on a given task, it always requires fine tuning. So some of the videos that I showed at the beginning, they were fine tuned models, to do things like, unlocking a lock. Other work that we've done on measuring human to robot transfer also needed fine tuning to get the best performance. And of course, all of the videos that I showed with RL post training were also fine tuning on an individual task to get the best performance on something like making espresso.
00:25:22 · Chelsea Finn
But if you have to fine tune a model, you actually aren't getting a general purpose model for the things that you want it to do because you have to fine tune it for each individual thing. And so our first goal is to move towards a single general purpose model that can actually do all of the things that you want it to do. Then the second goal that I mentioned is compositional generalization. And this is inspired by, the DALL E result from 2021. And I think that this was a really important and exciting milestone because of the compositional generalization that it achieved. And specifically, when you have compositional generalization, when you can basically bridge the concept like of an avocado and a chair and show that you kind of can combine those two, it means that the model has at least some kind conceptual understanding of what an avocado is and what a chair is to the point that it can combine them into something that exhibits both concepts at the same time. And then second, it means that you have some degree of data efficiency where your data doesn't need to cover all of the possible combinations of concepts represented in your data. You don't need pictures of avocado chairs in your dataset in order to generate something like this. Or you don't need combinations of other things that you might ask the model to do at, when it's deployed. Now even back in 2021, it wasn't perfect, but these signs of compositional generalization were really exciting for, demonstrating these two attributes of the model. So we have these two goals that we'd like to do. We like it out of the box model and compositional generalization. Now the tried and tested recipe for developing this kind of model is to first take a sufficiently large and diverse dataset and to second, train a model with sufficient capacity. And so what we're gonna do is we're gonna do that. We're gonna try to use all of the data that we have available. This includes really diverse robot demonstration data, including really low quality demonstration data. It's also gonna include policy rollout data, basically attempts from the robot of doing the task. Basically, all of the training data that was used for reinforcement learning for the previous tasks will be included in the training recipe.
00:27:38 · Chelsea Finn
We're also gonna include videos of humans, and we're also going to include data from the web, basically all of the data that we have. And then to train a model with sufficient capacity, of course, we'll train a model that's large enough. But to fit data that's so heterogeneous, we also find it, particularly important to prompt the model with all of the context that it needs in order to predict actions. And we found that this idea was really the key unlock to using this kind of data and this data of this kind of degree of heterogeneity. And specifically, what this looks like is we're gonna train a foundation model that takes as input, the memory that I mentioned before, an instruction of what to do. But it's also gonna take as input a subtask instruction, of kind of what the next immediate thing it should do is. It'll also take as input metadata that kind of indicates the quality of the data, the length of the episodes, and so forth. And this metadata gives it a lot more information about how it should predict the next action. And then optionally, we'll also, train the model with a subgoal image as a prompt to the model. This is essentially saying, like, a few seconds from now, you should try to reach something that looks like this image. With this detailed prompting, we find that the model can really make use of much more heterogeneous data, and I'll show some comparisons later that really, kind of show how important it is. And now to actually deploy this model, we need to then provide things like this subtask instruction and subgoal images. And so with that, we can train, a high level policy that predicts the subtask instruction, like what to do next, what is the next subtask for the task of, cleaning the kitchen. And we'll additionally train a world model to generate images for what the robot should do next as sub goal image conditioning. And so with this, we'll train a single model on, with those attributes on all of the diverse data that we had available. And here are some examples of what that single model can do. So all of these videos are from a single models, specifically a model that we called the pyo seven model. And on left, you can see it doing things like, folding, a collared shirt. On the top right, it's doing a really precise assembly step where it needs to insert a screw and drill that screw into a robot arm.