Y Combinator Startup Podcast
00:02:22 · Chelsea Finn
And this was a really exciting advance because deep learning is an algorithm that actually isn't, you can kind of really apply it out of the box, to scenarios that involve really complex, inputs and outputs and so forth, and it makes it easier to translate to other applications. But from there, I think that even more exciting kind of moment in time that we saw in terms of machine learning and AI in production was in 2022, with the launch of ChatGPT. And this was the first time where we saw a general purpose model truly being used by many different people in the real world. Within five days, ChatGPT had reached a million users. And then, of course, more recently, we've seen things, like Claude Code, also be incredibly useful, hopefully, to many of us, in the real world and other coding agents. Now if we look at how AI has been used in the real world, and and kind of look at this, I think there's a few different takeaways we could make. The first is that generalist models are increasingly being used for real world problems. And so we're actually seeing, general purpose, like generalist AI models that can do many, many different things actually be used in the real world, and we see that transition from the left to the right. But I also think that there's a more nuanced observation that we can make, from looking at these applications. And in particular, if we look at all of these different applications that are used, where machine learning has actually been useful in the real world and actually, been profitable and so forth, But in all of these applications, the customer is making a decision based off of the recommendation of the AI model, more or less. And this means that if the customer is ultimately, like, kind of making the decision, this means that if the system makes a mistake, that's okay because usually the person can kind of recognize that or or decide what to do even despite that mistake. And so even when these sorts of systems aren't perfect, they're still incredibly useful to different people and there's less pressure on them to be completely perfect. And I think that actually physical AI and robotics is pretty different from this. Where if we think about physical AI that are actually operating in the physical world, they have to be directly making decisions that affect the physical world.
00:04:42 · Chelsea Finn
And this means that they're gonna be far more useful when they're operating fully autonomously. And as a result, this requires us to develop physical AI systems that make far fewer mistakes than the machine learning systems that have been deployed thus far. Now, one really exciting thing to kind of highlight, that has happened recently is, a year ago, Waymo passed the, quarter of a million weekly autonomous rides suggesting that it is really possible to develop a machine learning based system that can, operate in a trustworthy and autonomous way, directly in the physical world. And I think that brings a lot of hope and optimism for actually, doing the same with the rest of AI in the physical world. So, if we wanna develop general purpose robots in the real world, I think we need to think about how we're gonna make them autonomous for long periods of time, so that they're actually useful, rather than having them, rather than something where a human is basing decisions on the predictions of the model. So, to think about long term autonomy, I wanna ground this in a specific example and say that we wanted a robot to make espresso. If we wanted it to be actually be useful for us, we needed to make espresso reliably so that we don't have to babysit the robot very frequently in order for it to help serve drinks. Though even on its own, this task is really difficult. So actually operating the porter filter requires very precise and forceful control to insert it, appropriately. It also needs to smoothly handle cups with liquid in it and not spill those cups. And it also needs to have an accurate sense of timing, which often isn't actually an issue with other areas of machine learning. And not only do we wanna do this pretty challenging task, we wanna do it with over 90% reliability. So how can we do this? So the first step in machine learning is always to collect some dataset, train a model, and evaluate how good your model is. And unfortunately, this rarely works reliably on the very first try. Now, in practice, then it's a bit better to then iterate on the model that you've developed, where you will try to, collect more data or improve the quality of the labels in your dataset or make the labels more detailed, collect more data of the edge cases, the scenarios where it's not working well, adjust the balancing of the dataset, and so forth.
00:07:04 · Chelsea Finn
And while this generally improves the reliability of the model, people eventually get tired, and it's hard to get really, really high reliability with a person that's manually tuning this. And so what would be even better is if the AI system itself can iterate on the scenario in which you want it to have high reliability, where it, on its own, automatically seeks out places where it needs more data, where it needs more supervision. And if we can do this for many, many more iterations because it's automatic rather than a person doing it, then this might be the way to get really, really high, like 99 plus percent reliability from physical AI systems. And so this is the approach that we'll take. And this looks a lot like a reinforcement learning algorithm that's trying to attempt the task, learn from its failures, and get better and better on its own. So then how do we develop a scalable reinforcement learning recipe for robotics? Well, in language models, we have algorithms like PPO and GRPO. And these have scaled to large language models and have enabled really complex reasoning. But there's a bit of a challenge in applying this to robotics, which is that these algorithms have been trained with millions of attempts or sometimes even tens of millions of attempts by scaling up the compute because each attempt is simply running the language model in a data center just by using compute. And if we were to translate this very, very approximately to robotics, say we had maybe not millions or tens of millions, but just 1,000,000 trajectories of a one minute robot task is even shorter than the espresso task that I talked about. This would correspond to seven hundred robot days to get high reliability for that task. Now, maybe this isn't completely out of the question, but this would be quite challenging, to do. And that's because the calculus is a little bit different. We're not just running compute to optimize for a use case. We're actually running the robot in the real world and using the hardware and attempting the task in the real world. And so we'd like to have an algorithm that can iterate much more efficiently. And there are actually ways that we can make these algorithms a lot more efficient. So, there's a couple inefficiencies, like large inefficiencies in these reinforcement learning algorithms for language models.
00:09:25 · Chelsea Finn
The first is that they spend a lot of time on dead end trajectories. And maybe this is okay if you're just spending compute on it, but this would cost a lot in the physical world. So we can look at a concrete example, say that we want a robot to construct cardboard boxes and stack them, on the right. And in this trajectory right here, the robot accidentally grabbed two boxes that are flush against each other. And if we let it continue, it would just continue to try to fold that box rather than separate out the two boxes. And trying to fold two boxes together isn't useful data that will teach the model how to get better at the task. And so that would be kind of wasting a lot of time on the robot attempting to go down the wrong path for solving the problem. And so instead of spending a lot of time trying to do that task, what we'll do is we'll actually have a human intervene and show the robot what to do and how to recover from that situation. And so what you can see here is a human is tele operating and intervening with the robot and showing it that to recover from this situation, it needs to essentially try to separate out the two boxes. It then kind of puts its gripper in, sees it if the... Then if the robot could autonomously recover, it doesn't autonomously recover. And so the person then intervenes again, to help, help it get back on the right track so that we're efficiently using the data on the robot. Okay. So this is the first thing that we can do, which is we can show the robot how to recover early, or how to recover so that we're not spending time on dead end trajectories, or at the very least, just terminate the episode early. Now, the second thing that we can do is, PPO and GRPO and these kinds of algorithms, they make many attempts at a single prompt. And depending on the algorithm, they're they're essentially trying to estimate for these different responses, what is a good response and what's a bad response. And so even for an individual prompt, they're actually going to kind of roll out like 10 or 50 times, for that individual prompt. And they're doing this because they're trying to estimate the value of these different, attempts to then kind of up weight or kind of increase the likelihood of good things and decrease the likelihood of bad things.
00:11:42 · Chelsea Finn
But we can actually amortize this cost, rather than trying to collect a lot of attempts for a single prompt, we can amortize this across different prompts and learn a much more general value estimate of what's good and what's bad and use this to improve with our autonomous experience. And so what this looks like is, we can train a general purpose value function on lots of kind of videos of the robot experience. And this can learn things like if it accidentally unfolds a shirt when it's trying to fold, that that's bad and that's making negative progress, it's shown in red. Or if it's making kind of forward progress, it recognizes that as well. And the same value function can also estimate what's good and bad for a completely different scenario, in this case, for retrieving an item from a fridge. And so this sort of kind of general purpose value model that's kind of predicting basically the time to success can basically significantly reduce the amount of attempts needed to learn how to improve from experience. So with these two kind of improvements to a reinforcement learning system, we kind of have a general improvement algorithm that trains a foundation model on diverse data, then collects experience from that with a human intervening as necessary to help prevent dead end trajectories, and then train a general purpose estimate of what's good and bad, the value function, and then use that to improve the model. And with this sort of improvement, we're able to, kind of fine tune a foundation model to higher degrees of performance. So, in the task of making, a latte, in this case, we'll be making a latte in collaboration with a person, where the robot is in charge of making the espresso, and the person is in charge of steaming the milk. This is what the, what the kind of model looks like. The model is directly controlling the joints of the robot using, the images from the robot's cameras as input. And we can see that the model, is able to do the pretty challenging task of inserting the portafilter, waiting the appropriate amount of time for the espresso to dispense, pouring the steamed milk into the cup. And then the last part of this task is actually the most challenging, where it needs to take a very full latte, cup of latte and transfer that over to the coaster. And so here's actually the observation that the robot sees directly.
00:14:04 · Chelsea Finn
And you can see that the policy is super delicate and able to balance the cup appropriately and smoothly so that the latte doesn't spill. So this is the kind of gives you a sense of the difficulty of this kind of task. And kind of going back to this reliability question, we took this policy and we ran it not just once, but we ran it for thirteen hours straight. And we basically wanted to evaluate, is this policy not only good at making a latte once, but can it do so reliably to the extent that it would be needed to be useful in the real world? And so here's a time lapse of that process. And indeed, we found that the robot was reliable enough to be useful for long stretches of time without making mistakes frequently. Now, the same algorithm isn't specific for making lattes, of course. And so we also applied this to other applications as well. Dandelion Chocolate Factory is a few blocks from our office. And so we took a workflow that they typically have a person do, which is to construct these cardboard boxes, label them, and stack them. And we trained our robot to basically do exactly their real workflow, and trained it with the reinforcement learning algorithm that I talked about to get a policy that is far more reliable at constructing, labeling, and stacking these boxes. And then we also applied this algorithm to fold and close as well. And we wanted, in this case, to not just test how well the model could do one task in one environment, but to do it in many environments. And so these are clothing items that the robot has never seen before in a home that's never seen before, it's able to do so and act autonomously for an extended period of time. Now, videos aren't, don't always kind of show everything. And so we also quantitatively measured the reliability of these models. And we care both about the reliability as well as the speed, like how many boxes can it build per hour. And so we're gonna measure throughput, which kind of couples both success rate and speed. And we find that over the phases of training from pre training to like an SFT like stage to an RL post training like stage, we see a drastic increase in success rate and seed in throughput. And specifically around a 2x throughput just from the RL stage itself, showing how we can get much greater reliability from reinforcement learning.