Y Combinator Startup Podcast
00:11:53 · Jay, Francois, S0
[00:11:53] S2: of learn. [00:11:53] S4: But even then there's a limit to how good ICL can go. So I've done this experiment where like you take an LLM that we've trained on and I've held out a task, let's just say GSM8K for simplicity. And then I ICL it and I wanna measure on the vals set how much it improves on a per sample basis. And it improves greatly, very cheaply. It doesn't cost much. Don't have, there's no SGD, so I don't have any flops. And so very quickly I can adapt and improve on my vowel set. Example, example, example. One is non monotonic improvement, which is wild. So it it gets worse, it gets better, it gets worse, it gets better, like, pretty aggressively. Number two is that it caps out very quickly. And so after, like, 20, 30, maybe 40 examples, it is basically saturated and more examples back into the context does not improve. [00:12:41] S0: You're like you're constrained by the model's ability to intelligently use all of its context.
00:12:45 · Francois
[00:12:45] S4: Exactly. From the post training. Yeah. How many multi turns did it actually get to be able to improve and do self reflection? And then the most important one for surely can't do is after it hits the context window, context length of the model that it was trained on. If it was trained on a 100,000, really that means you have a context window of like 50,000. Once you exceed 50,000, you don't improve anymore. You actually just get worse because the model can't attend over everything. There's a really good example you said, which I didn't really think about is like, well, if I can rag or I can load in my act into active memory, and this is very prime aging continual harness kind of like thinking, where I can pull stuff in on the fly, example, similar examples, then it's like me, like, have an exam, or I have an exam with a textbook. I'm gonna do better if I have the textbook to look up as reference. And so that is definitely gonna work, and it's still, and it also scales, I would imagine much better. And then the last two paradigms would be LoRA with rank one or rank two or whatever. LoRA with rank 10 or rank 100, and then all the way to full SFTRL. And if you're Tesla, and you have all the data, infinite data, and you're learning... Doing self driving car by ICL, what are you doing? Right. Like, you kidding? It's like, clearly, wouldn't do that. Right? And if you're figure or something like that. But it's amazing how good in the in low data regimes you can do with ICL. And then even cooler, we can compress the ICL into tool use, into tools, or, like, compress it into learnings. And that's that's the hierarchy that we haven't really figured out that you guys are probably on the forefront of.
00:14:22 · S0, Vincent, Jay
[00:14:22] S0: Yeah. Do you guys wanna elaborate on that a little bit? Because I imagine that's a central part of how you guys think about building Waddle. [00:14:27] S3: There's a few things that are quite interesting here, I think, to unpack. [00:14:29] S2: Yeah. The the first
00:14:30 · Vincent
[00:14:30] S3: thing is we've been sort of thinking about this idea of a harness almost as a form of, like, domain specificity. Like, when when you, for example, deploy a robot in a new environment, maybe it's in a wet lab, and it just needs to do a lot of test you're picking up, for example. You can learn a lot of this in context, but then one way of consolidating this context for future agents is sort of packaging each of these skills that you've learned into specific programs, for example. Like, writing out these skills and writing out these memories is a form of consolidating. It... It's it's like a form of distillation from past experience for your future agents to use. So this is quite interesting. The other thing that's quite interesting is it almost seems like there's some relationship to the broader meta learning literature throughout in machine learning history. Like, you have this sort of bigger model that then programs a smaller model to do things. And one of the very interesting things is that, yes, these smaller models are gonna learn in context. They're sort of wrapped inside the harness, and they're doing the task. You can make them maybe smaller. You can make them run faster as long as the bigger model can sort of do this domain specialization of your harness quite well. So so this is something that we're we're pretty interested in right now. And last point as well, I think the the the really interesting threaded question is how far in context learning can get you. Right. And I think it's really anyone's guess as to that. Like, there's papers that show that, you know, these models sort of can approximate gradient descent during in context learning as well. So it's this this line between weight space and symbolic space, I think, is is a little bit blurry.
00:16:00 · S5, S0, Jay, Han Ming
[00:16:00] S5: YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com/apply. It's never too early, and filling out the app will level up your idea. Okay. Back to the video. [00:16:14] S0: So what exactly are we watching here? [00:16:16] S2: We are watching Astra controlling the arms to pick a block off the table and put it into the bowl. [00:16:22] S0: So here, Astra is using just these camera inputs, and presumably, it's aware of the robot that it's controlling, and it's gonna write code that controls the robot at its individual joints. [00:16:31] S2: Yes. So it it does look at all the cameras. It has all the camera feeds. Instead of code, it's more like a tool call. So it sends commands to the robot to to control and put the block into the bowl. [00:16:46] S0: So in this situation, we're using Astra directly. But if we were using, say, Waddle's harness, what's the kind of difference between directly using Astra to control this versus doing this through Waddle's API? [00:16:56] S1: Sometimes directly, like, having Astra directly command the post robot should go to is not direct... The optimal tool to use. If it's a repetitive task, you don't want Astra in the loop. Maybe you wanna write code that can run repeatedly very, very fast. Or if it's a task... You've done a similar task before, you should be able to call it a skill that has compiled and use that skill to do the task faster and handle edge cases better. [00:17:17] S0: So here we can see it seems to be approaching in on this guy. Oh, very dramatic.
00:17:17 · S0, Jay, Han Ming
[00:17:17] S0: So here we can see it seems to be approaching in on this guy. Oh, very dramatic. [00:17:22] S2: Yeah. You can look... You can notice that the latency is a bit slow because we are bottlenecked by the latency of Astra. But if you look at the trends, the latency of these models are improving very rapidly. Right. So one thing that we saw is that for Fable class LLMs, their latency is improving by around two x per month, which is very, [00:17:43] S1: very fast. [00:17:44] S0: Very fast. [00:17:44] S2: Yeah. Yeah. If the trends continue, we could get real time control by end of the year. [00:17:50] S0: So now that we saw this task go from end to end, why don't we walk through what that actually entailed? So there's a coding agent in the loop here running this. Yes. What are the steps that the coding agent would have taken? And let's kind of contrast the direct coding agent version versus the Waddle harness in the loop version. [00:18:06] S2: So what just happened was that in a series of turns, Astra received the images from the cameras, and it outputted the end effect of pose for the robot to go to. And it does it over multiple turns to complete the task. This is less of code as policies, but more of tool calls.