Y Combinator Startup Podcast
00:09:18 · S0
[00:09:18] S0: Could we talk a little bit about what some of these early Code as Policies policy methods were even able to do? So even... You know, this is well ahead of us having coding agents that are widely available and people intuitively understanding how they work. Well ahead of people using RL methods to actually train those coding agents to be really good at tool calling. What did some of those early ones, maybe even before Voyager or, like, Code as Policies policies, which is... Which I think the paper came out in 2022 at the end, right around when ChatGPT came out. What was the capability of those compared to what you can see now?
00:09:43 · Han Ming, S0
[00:09:43] S1: Yeah. Those coders policy papers, especially ones for Google DeepMind team, were incredible because what they did is they created these kind of functions, like pick up an object, lift up, or, like, move to a certain pose. Like, they provided this list of functions to a coding agent in the form of, like, literary Python functions. And then decoding agent will be able to write code involving these functions to then control the robot to do very complex hacks. And what was very surprising, what what where this excelled was that coding agents can do this very one shot. They did not need additional robot data to... In order to work with this code because they're already trained on so much coding data. They don't have a sense of, like, how to... What what to do first, what to do second in order to, like, move a block into a ball, for example. And I think it's this one shot ability, this in context exploration ability that really motivated a lot of later work to continue exploring, including us, to continue exploring how we can apply LLMs into robotics. [00:10:41] S0: I know, Francois, you have this framework we've talked about of, you know, the various ways that learning can happen. There's gonna be learning that's embedded into the weights versus being in context learning and so on. Do wanna quickly talk about that? And then maybe we can think about where all the research over the last few years has fit into that framework, especially the direction it seems to be going.
00:10:41 · S0, Francois, Jay
[00:10:41] S0: I know, Francois, you have this framework we've talked about of, you know, the various ways that learning can happen. There's gonna be learning that's embedded into the weights versus being in context learning and so on. Do wanna quickly talk about that? And then maybe we can think about where all the research over the last few years has fit into that framework, especially the direction it seems to be going. [00:10:57] S4: I mean, I've I've been working on this experiment, and and actually, I haven't really crystallized it into a paper yet. But basically, it's like, what is the most efficient way from a, intelligence per sample perspective to input a learning, let's say, in this context, a state action result or state action reward tuple back into the policy. There's ICL, there is... That's in context learning. We're describing context learning, where like, I just like append it. This is how most people are using LLMs. They're just like, Oh, no, don't do it like this, do it like this. And I was like, Okay. And then it's still in the context, and then I can remember, I can iterate. And that really only works if you've trained the LLM or at least post trained the LLM to be able to learn and improve. And that self refines a reflection, like literature from way back that like actually allowed multi turn into the training set. If you don't have that, it doesn't learn, it doesn't, any kind [00:11:53] S2: of learn.
00:11:53 · Jay, Francois, S0
[00:11:53] S2: of learn. [00:11:53] S4: But even then there's a limit to how good ICL can go. So I've done this experiment where like you take an LLM that we've trained on and I've held out a task, let's just say GSM8K for simplicity. And then I ICL it and I wanna measure on the vals set how much it improves on a per sample basis. And it improves greatly, very cheaply. It doesn't cost much. Don't have, there's no SGD, so I don't have any flops. And so very quickly I can adapt and improve on my vowel set. Example, example, example. One is non monotonic improvement, which is wild. So it it gets worse, it gets better, it gets worse, it gets better, like, pretty aggressively. Number two is that it caps out very quickly. And so after, like, 20, 30, maybe 40 examples, it is basically saturated and more examples back into the context does not improve. [00:12:41] S0: You're like you're constrained by the model's ability to intelligently use all of its context.
00:12:45 · Francois
[00:12:45] S4: Exactly. From the post training. Yeah. How many multi turns did it actually get to be able to improve and do self reflection? And then the most important one for surely can't do is after it hits the context window, context length of the model that it was trained on. If it was trained on a 100,000, really that means you have a context window of like 50,000. Once you exceed 50,000, you don't improve anymore. You actually just get worse because the model can't attend over everything. There's a really good example you said, which I didn't really think about is like, well, if I can rag or I can load in my act into active memory, and this is very prime aging continual harness kind of like thinking, where I can pull stuff in on the fly, example, similar examples, then it's like me, like, have an exam, or I have an exam with a textbook. I'm gonna do better if I have the textbook to look up as reference. And so that is definitely gonna work, and it's still, and it also scales, I would imagine much better. And then the last two paradigms would be LoRA with rank one or rank two or whatever. LoRA with rank 10 or rank 100, and then all the way to full SFTRL. And if you're Tesla, and you have all the data, infinite data, and you're learning... Doing self driving car by ICL, what are you doing? Right. Like, you kidding? It's like, clearly, wouldn't do that. Right? And if you're figure or something like that. But it's amazing how good in the in low data regimes you can do with ICL. And then even cooler, we can compress the ICL into tool use, into tools, or, like, compress it into learnings. And that's that's the hierarchy that we haven't really figured out that you guys are probably on the forefront of.
00:14:22 · S0, Vincent, Jay
[00:14:22] S0: Yeah. Do you guys wanna elaborate on that a little bit? Because I imagine that's a central part of how you guys think about building Waddle. [00:14:27] S3: There's a few things that are quite interesting here, I think, to unpack. [00:14:29] S2: Yeah. The the first