Y Combinator Startup Podcast
00:00:00 · S0, Han Ming, Jay, Vincent
[00:00:00] S0: One of the big surprises the last few years has been the generalizability of coding agents across different domains. And now frontier researchers are showing that this includes controlling robots. This led MIT professor Philip Isola to suggest in a recent viral essay that we may be entering the era of robot use agents, where general purpose models could make different robots more capable. So today, Francois and I invited the founders of Waddle Labs and RoboCurve, two groups of startups that are working at the very frontier of making robots more capable with LLMs. Maybe you guys wanna briefly introduce yourselves and just say a little bit about what each of your companies focuses on. [00:00:40] S1: I'm Han May. I'm from Waddle Labs together with Vincent. We work on building LLMs that control robots, and we do this by doing two things, building a harness that allows the LMs to do this very effectively, collecting data, and then using that data to train better LLMs. [00:00:54] S2: Hi. I'm Jay, cofounder of RoboCurve. We are an EVOS company for physical AI. We measure everything, any robot, any model, including LLMs, and also classical approaches such as VLAs, vision language action models, and world action models. And we evaluate all kinds of embodiments, like hands, grippers, arms, humanoids, quadrupeds, all kinds of stuff. [00:01:20] S0: So in the last few weeks, videos from both of you guys went quite viral on Twitter, and I think inspired... [00:01:25] S3: I think
00:01:25 · Vincent, S0
[00:01:25] S3: I think [00:01:25] S0: both of them are actually quoted in that Philip Isola essay talking about robot use agents. The videos showed things like LLMs being able to unscrew caps, being able to communicate between multiple robots, being able to do tasks like, uncapping a pen, for example. And so I thought maybe what would be fun is for Francois and I to dig in with you guys on some of the research that led to this moment to begin, and then maybe we can also show some demonstrations of what this can actually do. To start, why don't we talk about some of the early research around transfer between language models and robot control. So I wanna talk about some of the early work on pretraining language models for decision making, and then also the r t two paper, which established the VLA. So maybe, Jay, you wanna tell us a little bit about some of this early work involving being able to use language models to do any kind of robot control?
00:02:12 · Jay
[00:02:12] S2: One of the earliest successful approaches of using AI on robots is the r t two paper, where they use a pretrained language model on web text and images and use that to control robots. And the interesting thing about that is that it's a fine tuned version of a language model, and instead of outputting English, for example, they just output what we call an an effector pose, which is the coordinates that you can then translate into join commands that can control the robots. In some cases, it's... In some sense, it's very similar to what we see now with large language models, just that instead of fine tuning, the models are good enough to just do it out of the box.
00:02:58 · Francois, Jay
[00:02:58] S4: And this is actually quite similar to, like, the mapping that I have in my head is kinda like the COT moment. And so like when we did GSM eight ks and you said, Susie had $8 she spent $5 how much does she have now? It had to output like four hash marks and then the answer and an EOS. There was no ability for it to chain of thought. And so then we allowed it to actually have like, okay, let me go eight minus five is three, blah, blah, And so like, hash, hash, hash, hash, three. And so you allowed it to do this kind of thinking before it actually gave an action or an answer. And similarly now, like VLAs were basically like an RT2 is basically like, it has to output an action. There's no, like I can't allocate more compute for a more complex task. And then now I have this code chain of thought thing that I can do. And I can say, okay, even if you were using the example where you're not actually outputting the code, just giving it to Astra and let it think, think, think, think, think, and then output an action. Similarly now, and then with code, it's a more Kolmogorov complexity or lower Kolmogorov complexity code code length to just use code and really compact code to just, here's the code. Just go. [00:04:07] S2: I totally agree with you on that. I feel like at the end of the day, it's a lot about a bit of less. Right? If you give the agents or you give the AI model more autonomy and you... If you unshackle it a bit more and give it more resources, it can actually do a lot of the things that we fine tune it to do.
00:04:07 · Jay, Vincent
[00:04:07] S2: I totally agree with you on that. I feel like at the end of the day, it's a lot about a bit of less. Right? If you give the agents or you give the AI model more autonomy and you... If you unshackle it a bit more and give it more resources, it can actually do a lot of the things that we fine tune it to do. [00:04:25] S3: I'm just gonna hop in here as well. I think the the really interesting thing about the bitter lesson here is that, like, VLAs by design architecturally, they're built on top of language models as well. Right? They they can potentially reason. They can write code. So perhaps the better lesson here isn't necessarily what architecture you build necessarily, but it's what data is most useful. Right? Like, people have been sort of nagging at this VLA data bottleneck for years now, and we're seeing very, very slow progress. And so the lesson here is maybe we we have a modality of data that we know works very well, and this idea of transferring across different modalities, which Han and I are super excited about. Like, how do you take something that's traditionally out of distribution, like robotics, and make it something that's in distribution? Like, maybe the way of harnessing a bit of that sin is saying, let's pick a data for which we know this modality is pretty blessed. There's a lot of data, these language... What these models understand it well, and then use this as a way of unlocking a lot of other domains as well.
00:04:25 · Vincent, Jay
[00:04:25] S3: I'm just gonna hop in here as well. I think the the really interesting thing about the bitter lesson here is that, like, VLAs by design architecturally, they're built on top of language models as well. Right? They they can potentially reason. They can write code. So perhaps the better lesson here isn't necessarily what architecture you build necessarily, but it's what data is most useful. Right? Like, people have been sort of nagging at this VLA data bottleneck for years now, and we're seeing very, very slow progress. And so the lesson here is maybe we we have a modality of data that we know works very well, and this idea of transferring across different modalities, which Han and I are super excited about. Like, how do you take something that's traditionally out of distribution, like robotics, and make it something that's in distribution? Like, maybe the way of harnessing a bit of that sin is saying, let's pick a data for which we know this modality is pretty blessed. There's a lot of data, these language... What these models understand it well, and then use this as a way of unlocking a lot of other domains as well. [00:05:22] S2: Yeah. And to add to that, the reason why r two... RT two is such a good model is because it is using a language model that taps into the modality of of all the data that a language model is trained on. So all those web images and web tags actually improves the VLA compared to just training a specific robotics foundation model Right. Without any pretraining.