Y Combinator Startup Podcast
00:22:56 · S0, Han Ming
[00:22:56] S0: The interesting thing that... That's maybe not entirely intuitive, and maybe I'd be curious to hear you guys all think about, is why specifically it seems like these newest models, specifically Astra, seems to be so much better at spatial intelligence. You know, we've all seen the demos online of, you know, controlling Blender and making great three d images. Now it seems like, you know, you showed Jay in a benchmark that is, like, a meaningful step function improvement on certain tasks. Tasks. Where do you guys think the the pretraining and and post training, I guess, of those models, what has likely changed about OpenAI's approach there that made them so much better compared to models even six months earlier? Like, you know, that's still we're benefiting from their lesson, and we're still extremely good at code, and we're still solving math problems but haven't quite cracked the spatial intelligence needed here. [00:23:36] S1: I think what Astra does incredibly well is its, like, vision capabilities. It was probably pre trained on way more computer used data than ever before. Startup pretrained on so much like CAD data. And it's like, all of these kinds of data probably teach the model, like, similar, like, understanding of, like, the physical world as, like, a lot of robot data might.
00:23:36 · Han Ming, S0
[00:23:36] S1: I think what Astra does incredibly well is its, like, vision capabilities. It was probably pre trained on way more computer used data than ever before. Startup pretrained on so much like CAD data. And it's like, all of these kinds of data probably teach the model, like, similar, like, understanding of, like, the physical world as, like, a lot of robot data might. [00:23:58] S0: So the computer use data is an interesting point because it's not totally intuitive, I think, why computer use data is useful for understanding the physical world and that it, you know, tends to be like a computer where you're clicking around. Like, say more about why you think that is a big unlock because I totally buy they trained way more computer use data than ever before.
00:23:58 · S0, Han Ming
[00:23:58] S0: So the computer use data is an interesting point because it's not totally intuitive, I think, why computer use data is useful for understanding the physical world and that it, you know, tends to be like a computer where you're clicking around. Like, say more about why you think that is a big unlock because I totally buy they trained way more computer use data than ever before. [00:24:13] S1: Right. I think immediately, if you were just training a model for computer use, that would not be able apply for robotics. But if you feed a computer use data into a big model like Astra... And by computer use data, I mean, like, you drag a cursor around on a screen to orbit some CAD object in order to design in Blender. This tells you, like, how to reason about spaces. It at least tells you about, like, top down, left, right, all these concepts that you need to control a robot. Right? And I think that's why this data helps so much for making these LLMs so much better at robot use. I think, like, from the other direction, like, a lot of people in the robotics community were trying more and more different kinds of data as well. Like, egocentric videos kinda went from just teleoperation to to, like, you know, broader range of this data because it's not just robot data that can teach a model how to use a robot. So if we take this to the extreme, it's like, not feed every kind of data, coding, computer use, egocentric, into the same model? I think that's how we get to the most capable, like, robot use agent.
00:25:11 · Francois, Han Ming
[00:25:11] S4: And it's so funny that, like, you know, if you go to nineteen eighties, Xerox PARC or whatever, like, we literally made the graphical user interface to be more like the physical world so that we could interface with it. And, like, what we ended up doing is building an environment that was actually helpful for robotics to learn how to use the physical world. Right? We we have file systems. We have, like, files. We have folders. We have, you know, the GUIs for, like, SolidWorks and, like, Autodesk to, like, spin around stuff to make it simulate... Similar to the physical world. And then, like, we couldn't get robots to work in the physical world, so we just trained on that. And then now it works on the physical Exactly. [00:25:46] S1: There were incredible papers, I think, from Princeton and also robotics touches on this. It's like, where you just... If you design a right harness where you make the tools that interface with the robot look like computer used tools, like you have a agent drag a cursor around to control where the robot goes, That improves how well the LLM is able to perform on these physical tasks.
00:25:46 · Han Ming, S0, Jay
[00:25:46] S1: There were incredible papers, I think, from Princeton and also robotics touches on this. It's like, where you just... If you design a right harness where you make the tools that interface with the robot look like computer used tools, like you have a agent drag a cursor around to control where the robot goes, That improves how well the LLM is able to perform on these physical tasks. [00:26:05] S0: Where do you guys see... You know, given, like, I think reasonable guesses as to where the base models are gonna continue improving and your guys' own investments in either evaluating these models or building harnesses around them. What do you think is gonna be capabilities that we maybe now see as challenging to do, but that are gonna be increasingly possible or even trivial for a few months from now? Like, you know, six months ago, even the demonstrations that I've seen you guys post on your Twitter, I think, would have been kinda mind blowing to imagine coming from an LM. [00:26:32] S2: There's some consensus within the Frontier Labs and also in the robotics foundation models companies that we will have general purpose robots within the next two years or even earlier. And this is something that society is probably unaware of or even unprepared for. And when we say general purpose robots, we mean something like if you give any natural language instruction, it can do what a competent teenager could do with their bare hands. It's kinda like the ChatGPT moment, but for robotics in terms of capabilities where it can generalize to unseen tasks and unseen environments.
00:26:32 · Jay, S0, Han Ming
[00:26:32] S2: There's some consensus within the Frontier Labs and also in the robotics foundation models companies that we will have general purpose robots within the next two years or even earlier. And this is something that society is probably unaware of or even unprepared for. And when we say general purpose robots, we mean something like if you give any natural language instruction, it can do what a competent teenager could do with their bare hands. It's kinda like the ChatGPT moment, but for robotics in terms of capabilities where it can generalize to unseen tasks and unseen environments. [00:27:06] S0: For you guys, for the Waddle Labs guys, what does that mean for what you guys are building? [00:27:10] S1: I think we're gonna take steps to get there in about two years' time. I think there's still many challenges that are very visible. For example, when I left the online talk about latency, this is a big problem. If you just have Astra being in the loop thinking at every step, this is really slow. It's not gonna be economically, like, useful. So how do you, you know, consolidate kind of a first pass by Astra, like, this this contextual learning into, like, a faster skill or policy that you can then run repeatedly at incredibly high throughput?