Y Combinator Startup Podcast
00:20:51 · Francois, S0
[00:20:51] S4: way to think about this.... Neurosymbolic. That is neurosymbolic. We have neurons, and then they're emitting symbols with [00:20:55] S0: some code. I think it was actually a pretty good segue to a kind of final topic, which is around... Inspired by this paper also from Philip Isola that he titles the Platonic Representation Hypothesis. Mhmm. Right? Where platonic because it's a reference to Plato's cave and, you know, the idea that, you know, they're... You're seeing shadows of various realities. And I think here, the point he's making is that there's extensive evidence to show that language models and various representations of different datas actually learn these distance mappings between similar things under different training policies that are overlapping. And so, like, the idea there would be that as you train these systems on increasingly large amounts of data, they converge to a curve consistent mapping of the world. And maybe this implies that we would expect language models to get better and better at things over time that would make them useful for new tasks like RoboControl. But maybe I'm... I I imagine this representation hypothesis is actually pretty central to all of your guys' worldview or or your view of your companies. And so maybe why don't you guys tell me a little about how you think about this? And, also, maybe we can use that to make some guesses as to why we think models like Astra seem to be so much better at RoboControl than previous models and where... You know, make some predictions for where we might be going here.
00:22:13 · Jay, Han Ming
[00:22:13] S2: Yeah. I think one way to make this concrete for the robotics models versus language models debate is that the very, very strong language models will have very similar representations of the world with very strong robotics models. And if that's the case, then if you have a really strong language model, you also have a really strong robotics model [00:22:34] S1: Right. [00:22:35] S2: Using... If we believe in the platonic representation hypothesis. And if that's true, that's... Then beta lesson is... That is the best manifestation of the beta lesson. Right? Because you just need one really strong model regardless of architecture, and they would be outperforming any specific models Yeah. That is slightly weaker.
00:22:35 · Jay, S0
[00:22:35] S2: Using... If we believe in the platonic representation hypothesis. And if that's true, that's... Then beta lesson is... That is the best manifestation of the beta lesson. Right? Because you just need one really strong model regardless of architecture, and they would be outperforming any specific models Yeah. That is slightly weaker. [00:22:56] S0: The interesting thing that... That's maybe not entirely intuitive, and maybe I'd be curious to hear you guys all think about, is why specifically it seems like these newest models, specifically Astra, seems to be so much better at spatial intelligence. You know, we've all seen the demos online of, you know, controlling Blender and making great three d images. Now it seems like, you know, you showed Jay in a benchmark that is, like, a meaningful step function improvement on certain tasks. Tasks. Where do you guys think the the pretraining and and post training, I guess, of those models, what has likely changed about OpenAI's approach there that made them so much better compared to models even six months earlier? Like, you know, that's still we're benefiting from their lesson, and we're still extremely good at code, and we're still solving math problems but haven't quite cracked the spatial intelligence needed here.
00:22:56 · S0, Han Ming
[00:22:56] S0: The interesting thing that... That's maybe not entirely intuitive, and maybe I'd be curious to hear you guys all think about, is why specifically it seems like these newest models, specifically Astra, seems to be so much better at spatial intelligence. You know, we've all seen the demos online of, you know, controlling Blender and making great three d images. Now it seems like, you know, you showed Jay in a benchmark that is, like, a meaningful step function improvement on certain tasks. Tasks. Where do you guys think the the pretraining and and post training, I guess, of those models, what has likely changed about OpenAI's approach there that made them so much better compared to models even six months earlier? Like, you know, that's still we're benefiting from their lesson, and we're still extremely good at code, and we're still solving math problems but haven't quite cracked the spatial intelligence needed here. [00:23:36] S1: I think what Astra does incredibly well is its, like, vision capabilities. It was probably pre trained on way more computer used data than ever before. Startup pretrained on so much like CAD data. And it's like, all of these kinds of data probably teach the model, like, similar, like, understanding of, like, the physical world as, like, a lot of robot data might.
00:23:36 · Han Ming, S0
[00:23:36] S1: I think what Astra does incredibly well is its, like, vision capabilities. It was probably pre trained on way more computer used data than ever before. Startup pretrained on so much like CAD data. And it's like, all of these kinds of data probably teach the model, like, similar, like, understanding of, like, the physical world as, like, a lot of robot data might. [00:23:58] S0: So the computer use data is an interesting point because it's not totally intuitive, I think, why computer use data is useful for understanding the physical world and that it, you know, tends to be like a computer where you're clicking around. Like, say more about why you think that is a big unlock because I totally buy they trained way more computer use data than ever before.
00:23:58 · S0, Han Ming
[00:23:58] S0: So the computer use data is an interesting point because it's not totally intuitive, I think, why computer use data is useful for understanding the physical world and that it, you know, tends to be like a computer where you're clicking around. Like, say more about why you think that is a big unlock because I totally buy they trained way more computer use data than ever before. [00:24:13] S1: Right. I think immediately, if you were just training a model for computer use, that would not be able apply for robotics. But if you feed a computer use data into a big model like Astra... And by computer use data, I mean, like, you drag a cursor around on a screen to orbit some CAD object in order to design in Blender. This tells you, like, how to reason about spaces. It at least tells you about, like, top down, left, right, all these concepts that you need to control a robot. Right? And I think that's why this data helps so much for making these LLMs so much better at robot use. I think, like, from the other direction, like, a lot of people in the robotics community were trying more and more different kinds of data as well. Like, egocentric videos kinda went from just teleoperation to to, like, you know, broader range of this data because it's not just robot data that can teach a model how to use a robot. So if we take this to the extreme, it's like, not feed every kind of data, coding, computer use, egocentric, into the same model? I think that's how we get to the most capable, like, robot use agent.