Y Combinator Startup Podcast
00:05:22 · Jay, S0, Vincent
[00:05:22] S2: Yeah. And to add to that, the reason why r two... RT two is such a good model is because it is using a language model that taps into the modality of of all the data that a language model is trained on. So all those web images and web tags actually improves the VLA compared to just training a specific robotics foundation model Right. Without any pretraining. [00:05:45] S0: So so you're benefiting in the case of r t two, these early VLA approaches from the... From pretraining. I guess, what's the limitation in the, I guess, action taking fine tuning approach that you can now get around if you can directly write code? Like, you're you're referring to, like, a difference in data there. What exactly is that difference in data you're referring to? [00:06:05] S3: There's a few things that models have got much better at since r d two. Like, one, I mean, they're much better tool use, for example. So, you know, one one kind of data is now these models can... Well, I mean, they they write code much better, so now they can write complex policies as code. They're also much better using tools to explore the kinds of environments they're in, like what arms they have access to. And a lot of this comes from in context learning.
00:06:05 · Vincent, Han Ming
[00:06:05] S3: There's a few things that models have got much better at since r d two. Like, one, I mean, they're much better tool use, for example. So, you know, one one kind of data is now these models can... Well, I mean, they they write code much better, so now they can write complex policies as code. They're also much better using tools to explore the kinds of environments they're in, like what arms they have access to. And a lot of this comes from in context learning. [00:06:28] S1: Like, the the key difference between, like, these VLAs and... Or, like, g p d six or LLM isn't necessarily the architecture, but kind of the approach you take towards training. We want to be bitter, less impaled. Right? Like, we want to benefit from all kinds of data. We want to pour in computer computer used data into our robot models. We wanna pour encoding data into our robot models. But when you do that, you just end up getting what's... What we think of as, like, these general purpose LLM agents. And then so to me, it's like, okay. Why not just build a really good foundational LLM and then use that to control robots rather than training, like, a model that's specifically for robotics and that's more dependent on just robot data.
00:06:28 · Han Ming, S0
[00:06:28] S1: Like, the the key difference between, like, these VLAs and... Or, like, g p d six or LLM isn't necessarily the architecture, but kind of the approach you take towards training. We want to be bitter, less impaled. Right? Like, we want to benefit from all kinds of data. We want to pour in computer computer used data into our robot models. We wanna pour encoding data into our robot models. But when you do that, you just end up getting what's... What we think of as, like, these general purpose LLM agents. And then so to me, it's like, okay. Why not just build a really good foundational LLM and then use that to control robots rather than training, like, a model that's specifically for robotics and that's more dependent on just robot data. [00:07:05] S0: On that note, maybe one of the things that I think, if if we could wind the clock a couple years, may have been a good sign that we were trending in a good direction to this is research on code as policies. Right? Do guys wanna talk a little bit about when the research community refers to code as policies, what exactly that means? And I think it... It's worth putting this in the context of when this paper came out, which was coding agents just starting to work. I think this was still in the era in which people were, like, putting comments and auto filling, you know, Python code blocks, which, you know, now we think of as agent history, but this was, like, two years ago. So, yeah, how how does Code as policies work, and how did that inspire some of what you guys are now seeing as possible?
00:07:40 · Francois
[00:07:40] S4: I probably rewind back to Voyager, where like Voyager was like the one of the first... What does a coding agent mean? Coding agent requires good tool use and on the fly tool creation. That's called code. Okay. You have Python built ins. Let's just say I have only Python built ins. Those are tools. And I have sort, and I have like if, and I have for, and I have while, and I have all these things. And I need to use those tools to assemble a new tool called a new Francois. Py. And now that's a new tool, and I get to use that. That was Voyager. And like, yeah, I'm gonna say Voyager was the first one to do this. Voyager was the most popular one to do this for Minecraft. And then they created tools that they could invoke on the fly to help them play the game better, and compressed thinking and experience into a new tool that I can later invoke. And then I think because everyone in 2024 had this insight, was like, okay, if we just pour, you know, Dario and Sam both all realized if I pour all of my heart and soul and resources and compute and intelligence into getting better at coding, we will have liftoff. Then I can automate the ML engineer, then I can have liftoff and I'll have all the things. I'll solve all the things. And so then that's what everyone did. And I don't think that they had the insight. I don't think they were so insightful to know that, oh, if I did this, then we will have LMs that would be good enough to write policies for robotics. And I can displace all of robotics with actual coding agents. And I can displace artists with, like, you know, coding agents that are writing JavaScript to make beautiful, like, art. Like, I'm not sure that they had that insight, be honest.
00:09:18 · S0
[00:09:18] S0: Could we talk a little bit about what some of these early Code as Policies policy methods were even able to do? So even... You know, this is well ahead of us having coding agents that are widely available and people intuitively understanding how they work. Well ahead of people using RL methods to actually train those coding agents to be really good at tool calling. What did some of those early ones, maybe even before Voyager or, like, Code as Policies policies, which is... Which I think the paper came out in 2022 at the end, right around when ChatGPT came out. What was the capability of those compared to what you can see now?
00:09:43 · Han Ming, S0
[00:09:43] S1: Yeah. Those coders policy papers, especially ones for Google DeepMind team, were incredible because what they did is they created these kind of functions, like pick up an object, lift up, or, like, move to a certain pose. Like, they provided this list of functions to a coding agent in the form of, like, literary Python functions. And then decoding agent will be able to write code involving these functions to then control the robot to do very complex hacks. And what was very surprising, what what where this excelled was that coding agents can do this very one shot. They did not need additional robot data to... In order to work with this code because they're already trained on so much coding data. They don't have a sense of, like, how to... What what to do first, what to do second in order to, like, move a block into a ball, for example. And I think it's this one shot ability, this in context exploration ability that really motivated a lot of later work to continue exploring, including us, to continue exploring how we can apply LLMs into robotics. [00:10:41] S0: I know, Francois, you have this framework we've talked about of, you know, the various ways that learning can happen. There's gonna be learning that's embedded into the weights versus being in context learning and so on. Do wanna quickly talk about that? And then maybe we can think about where all the research over the last few years has fit into that framework, especially the direction it seems to be going.