Y Combinator Startup Podcast
00:01:25 · Vincent, S0
[00:01:25] S3: I think [00:01:25] S0: both of them are actually quoted in that Philip Isola essay talking about robot use agents. The videos showed things like LLMs being able to unscrew caps, being able to communicate between multiple robots, being able to do tasks like, uncapping a pen, for example. And so I thought maybe what would be fun is for Francois and I to dig in with you guys on some of the research that led to this moment to begin, and then maybe we can also show some demonstrations of what this can actually do. To start, why don't we talk about some of the early research around transfer between language models and robot control. So I wanna talk about some of the early work on pretraining language models for decision making, and then also the r t two paper, which established the VLA. So maybe, Jay, you wanna tell us a little bit about some of this early work involving being able to use language models to do any kind of robot control?
00:02:12 · Jay
[00:02:12] S2: One of the earliest successful approaches of using AI on robots is the r t two paper, where they use a pretrained language model on web text and images and use that to control robots. And the interesting thing about that is that it's a fine tuned version of a language model, and instead of outputting English, for example, they just output what we call an an effector pose, which is the coordinates that you can then translate into join commands that can control the robots. In some cases, it's... In some sense, it's very similar to what we see now with large language models, just that instead of fine tuning, the models are good enough to just do it out of the box.
00:02:58 · Francois, Jay
[00:02:58] S4: And this is actually quite similar to, like, the mapping that I have in my head is kinda like the COT moment. And so like when we did GSM eight ks and you said, Susie had $8 she spent $5 how much does she have now? It had to output like four hash marks and then the answer and an EOS. There was no ability for it to chain of thought. And so then we allowed it to actually have like, okay, let me go eight minus five is three, blah, blah, And so like, hash, hash, hash, hash, three. And so you allowed it to do this kind of thinking before it actually gave an action or an answer. And similarly now, like VLAs were basically like an RT2 is basically like, it has to output an action. There's no, like I can't allocate more compute for a more complex task. And then now I have this code chain of thought thing that I can do. And I can say, okay, even if you were using the example where you're not actually outputting the code, just giving it to Astra and let it think, think, think, think, think, and then output an action. Similarly now, and then with code, it's a more Kolmogorov complexity or lower Kolmogorov complexity code code length to just use code and really compact code to just, here's the code. Just go. [00:04:07] S2: I totally agree with you on that. I feel like at the end of the day, it's a lot about a bit of less. Right? If you give the agents or you give the AI model more autonomy and you... If you unshackle it a bit more and give it more resources, it can actually do a lot of the things that we fine tune it to do.
00:04:07 · Jay, Vincent
[00:04:07] S2: I totally agree with you on that. I feel like at the end of the day, it's a lot about a bit of less. Right? If you give the agents or you give the AI model more autonomy and you... If you unshackle it a bit more and give it more resources, it can actually do a lot of the things that we fine tune it to do. [00:04:25] S3: I'm just gonna hop in here as well. I think the the really interesting thing about the bitter lesson here is that, like, VLAs by design architecturally, they're built on top of language models as well. Right? They they can potentially reason. They can write code. So perhaps the better lesson here isn't necessarily what architecture you build necessarily, but it's what data is most useful. Right? Like, people have been sort of nagging at this VLA data bottleneck for years now, and we're seeing very, very slow progress. And so the lesson here is maybe we we have a modality of data that we know works very well, and this idea of transferring across different modalities, which Han and I are super excited about. Like, how do you take something that's traditionally out of distribution, like robotics, and make it something that's in distribution? Like, maybe the way of harnessing a bit of that sin is saying, let's pick a data for which we know this modality is pretty blessed. There's a lot of data, these language... What these models understand it well, and then use this as a way of unlocking a lot of other domains as well.
00:04:25 · Vincent, Jay
[00:04:25] S3: I'm just gonna hop in here as well. I think the the really interesting thing about the bitter lesson here is that, like, VLAs by design architecturally, they're built on top of language models as well. Right? They they can potentially reason. They can write code. So perhaps the better lesson here isn't necessarily what architecture you build necessarily, but it's what data is most useful. Right? Like, people have been sort of nagging at this VLA data bottleneck for years now, and we're seeing very, very slow progress. And so the lesson here is maybe we we have a modality of data that we know works very well, and this idea of transferring across different modalities, which Han and I are super excited about. Like, how do you take something that's traditionally out of distribution, like robotics, and make it something that's in distribution? Like, maybe the way of harnessing a bit of that sin is saying, let's pick a data for which we know this modality is pretty blessed. There's a lot of data, these language... What these models understand it well, and then use this as a way of unlocking a lot of other domains as well. [00:05:22] S2: Yeah. And to add to that, the reason why r two... RT two is such a good model is because it is using a language model that taps into the modality of of all the data that a language model is trained on. So all those web images and web tags actually improves the VLA compared to just training a specific robotics foundation model Right. Without any pretraining.
00:05:22 · Jay, S0, Vincent
[00:05:22] S2: Yeah. And to add to that, the reason why r two... RT two is such a good model is because it is using a language model that taps into the modality of of all the data that a language model is trained on. So all those web images and web tags actually improves the VLA compared to just training a specific robotics foundation model Right. Without any pretraining. [00:05:45] S0: So so you're benefiting in the case of r t two, these early VLA approaches from the... From pretraining. I guess, what's the limitation in the, I guess, action taking fine tuning approach that you can now get around if you can directly write code? Like, you're you're referring to, like, a difference in data there. What exactly is that difference in data you're referring to? [00:06:05] S3: There's a few things that models have got much better at since r d two. Like, one, I mean, they're much better tool use, for example. So, you know, one one kind of data is now these models can... Well, I mean, they they write code much better, so now they can write complex policies as code. They're also much better using tools to explore the kinds of environments they're in, like what arms they have access to. And a lot of this comes from in context learning.