Y Combinator Startup Podcast
00:06:28 · Han Ming, S0
[00:06:28] S1: Like, the the key difference between, like, these VLAs and... Or, like, g p d six or LLM isn't necessarily the architecture, but kind of the approach you take towards training. We want to be bitter, less impaled. Right? Like, we want to benefit from all kinds of data. We want to pour in computer computer used data into our robot models. We wanna pour encoding data into our robot models. But when you do that, you just end up getting what's... What we think of as, like, these general purpose LLM agents. And then so to me, it's like, okay. Why not just build a really good foundational LLM and then use that to control robots rather than training, like, a model that's specifically for robotics and that's more dependent on just robot data. [00:07:05] S0: On that note, maybe one of the things that I think, if if we could wind the clock a couple years, may have been a good sign that we were trending in a good direction to this is research on code as policies. Right? Do guys wanna talk a little bit about when the research community refers to code as policies, what exactly that means? And I think it... It's worth putting this in the context of when this paper came out, which was coding agents just starting to work. I think this was still in the era in which people were, like, putting comments and auto filling, you know, Python code blocks, which, you know, now we think of as agent history, but this was, like, two years ago. So, yeah, how how does Code as policies work, and how did that inspire some of what you guys are now seeing as possible?
00:07:40 · Francois
[00:07:40] S4: I probably rewind back to Voyager, where like Voyager was like the one of the first... What does a coding agent mean? Coding agent requires good tool use and on the fly tool creation. That's called code. Okay. You have Python built ins. Let's just say I have only Python built ins. Those are tools. And I have sort, and I have like if, and I have for, and I have while, and I have all these things. And I need to use those tools to assemble a new tool called a new Francois. Py. And now that's a new tool, and I get to use that. That was Voyager. And like, yeah, I'm gonna say Voyager was the first one to do this. Voyager was the most popular one to do this for Minecraft. And then they created tools that they could invoke on the fly to help them play the game better, and compressed thinking and experience into a new tool that I can later invoke. And then I think because everyone in 2024 had this insight, was like, okay, if we just pour, you know, Dario and Sam both all realized if I pour all of my heart and soul and resources and compute and intelligence into getting better at coding, we will have liftoff. Then I can automate the ML engineer, then I can have liftoff and I'll have all the things. I'll solve all the things. And so then that's what everyone did. And I don't think that they had the insight. I don't think they were so insightful to know that, oh, if I did this, then we will have LMs that would be good enough to write policies for robotics. And I can displace all of robotics with actual coding agents. And I can displace artists with, like, you know, coding agents that are writing JavaScript to make beautiful, like, art. Like, I'm not sure that they had that insight, be honest.
00:09:18 · S0
[00:09:18] S0: Could we talk a little bit about what some of these early Code as Policies policy methods were even able to do? So even... You know, this is well ahead of us having coding agents that are widely available and people intuitively understanding how they work. Well ahead of people using RL methods to actually train those coding agents to be really good at tool calling. What did some of those early ones, maybe even before Voyager or, like, Code as Policies policies, which is... Which I think the paper came out in 2022 at the end, right around when ChatGPT came out. What was the capability of those compared to what you can see now?
00:09:43 · Han Ming, S0
[00:09:43] S1: Yeah. Those coders policy papers, especially ones for Google DeepMind team, were incredible because what they did is they created these kind of functions, like pick up an object, lift up, or, like, move to a certain pose. Like, they provided this list of functions to a coding agent in the form of, like, literary Python functions. And then decoding agent will be able to write code involving these functions to then control the robot to do very complex hacks. And what was very surprising, what what where this excelled was that coding agents can do this very one shot. They did not need additional robot data to... In order to work with this code because they're already trained on so much coding data. They don't have a sense of, like, how to... What what to do first, what to do second in order to, like, move a block into a ball, for example. And I think it's this one shot ability, this in context exploration ability that really motivated a lot of later work to continue exploring, including us, to continue exploring how we can apply LLMs into robotics. [00:10:41] S0: I know, Francois, you have this framework we've talked about of, you know, the various ways that learning can happen. There's gonna be learning that's embedded into the weights versus being in context learning and so on. Do wanna quickly talk about that? And then maybe we can think about where all the research over the last few years has fit into that framework, especially the direction it seems to be going.
00:10:41 · S0, Francois, Jay
[00:10:41] S0: I know, Francois, you have this framework we've talked about of, you know, the various ways that learning can happen. There's gonna be learning that's embedded into the weights versus being in context learning and so on. Do wanna quickly talk about that? And then maybe we can think about where all the research over the last few years has fit into that framework, especially the direction it seems to be going. [00:10:57] S4: I mean, I've I've been working on this experiment, and and actually, I haven't really crystallized it into a paper yet. But basically, it's like, what is the most efficient way from a, intelligence per sample perspective to input a learning, let's say, in this context, a state action result or state action reward tuple back into the policy. There's ICL, there is... That's in context learning. We're describing context learning, where like, I just like append it. This is how most people are using LLMs. They're just like, Oh, no, don't do it like this, do it like this. And I was like, Okay. And then it's still in the context, and then I can remember, I can iterate. And that really only works if you've trained the LLM or at least post trained the LLM to be able to learn and improve. And that self refines a reflection, like literature from way back that like actually allowed multi turn into the training set. If you don't have that, it doesn't learn, it doesn't, any kind [00:11:53] S2: of learn.
00:11:53 · Jay, Francois, S0
[00:11:53] S2: of learn. [00:11:53] S4: But even then there's a limit to how good ICL can go. So I've done this experiment where like you take an LLM that we've trained on and I've held out a task, let's just say GSM8K for simplicity. And then I ICL it and I wanna measure on the vals set how much it improves on a per sample basis. And it improves greatly, very cheaply. It doesn't cost much. Don't have, there's no SGD, so I don't have any flops. And so very quickly I can adapt and improve on my vowel set. Example, example, example. One is non monotonic improvement, which is wild. So it it gets worse, it gets better, it gets worse, it gets better, like, pretty aggressively. Number two is that it caps out very quickly. And so after, like, 20, 30, maybe 40 examples, it is basically saturated and more examples back into the context does not improve. [00:12:41] S0: You're like you're constrained by the model's ability to intelligently use all of its context.