Y Combinator Startup Podcast
00:19:13 · Francois, Vincent
[00:19:13] S4: The biggest insight I think I've had, a change in worldview on how to perform machine learning and how to get the rest of the distance on AGI, was, like, a lot of conversations I've had with François Chollet. And in 2020, maybe even 2018 when he did On the Measure of Intelligence, he talked a lot about transduction versus just wrong, versus program induction. And transduction just means that I'm learning a function theta that maps from x's to y's. And why is that wrong? It's just slow, and it's information inefficient. And so to go from Xs to Ys, I need a lot of pairs of Xs and Ys. And if I have a very small amount, then you need a lot of inductive bias, and what you need to act is a good generator to map from Xs to Ys. So now my theta takes in three N pairs of Xs and Ys and emits the function that maps from X to Ys. And that's what code is. So that's... Like, if you give me a coding interview, even a whiteboard, you say, okay. Here's a pudding problem. Here's some examples. Okay. Cool. Write the function. And then we are emitting an f that will map from x to y. [00:20:17] S3: And and this is quite interesting. Right? Because I think... I mean, this is a lot of the traditional program synthesis literature. Mhmm. And I feel like a lot of reason why those methods didn't work as well was because that inductive bias, as you said. Mhmm. Right? It's... You're shifting the difficulty of the problem from finding that mapping to finding the right inductive biases to map it to this, like, smaller code to then do the mapping. [00:20:37] S4: Mhmm.
00:20:37 · Francois, Vincent
[00:20:37] S4: Mhmm. [00:20:37] S3: But finding that set of inductive biases is super hard. But maybe that's what Astra is buying us. Like, even in Cox Eye, there's a giant shift in the literature from, you know, specifically neurosymbolic methods to just using Astra to write code to map. And so maybe that's a cool [00:20:51] S4: way to think about this.... Neurosymbolic. That is neurosymbolic. We have neurons, and then they're emitting symbols with
00:20:51 · Francois, S0
[00:20:51] S4: way to think about this.... Neurosymbolic. That is neurosymbolic. We have neurons, and then they're emitting symbols with [00:20:55] S0: some code. I think it was actually a pretty good segue to a kind of final topic, which is around... Inspired by this paper also from Philip Isola that he titles the Platonic Representation Hypothesis. Mhmm. Right? Where platonic because it's a reference to Plato's cave and, you know, the idea that, you know, they're... You're seeing shadows of various realities. And I think here, the point he's making is that there's extensive evidence to show that language models and various representations of different datas actually learn these distance mappings between similar things under different training policies that are overlapping. And so, like, the idea there would be that as you train these systems on increasingly large amounts of data, they converge to a curve consistent mapping of the world. And maybe this implies that we would expect language models to get better and better at things over time that would make them useful for new tasks like RoboControl. But maybe I'm... I I imagine this representation hypothesis is actually pretty central to all of your guys' worldview or or your view of your companies. And so maybe why don't you guys tell me a little about how you think about this? And, also, maybe we can use that to make some guesses as to why we think models like Astra seem to be so much better at RoboControl than previous models and where... You know, make some predictions for where we might be going here.
00:22:13 · Jay, Han Ming
[00:22:13] S2: Yeah. I think one way to make this concrete for the robotics models versus language models debate is that the very, very strong language models will have very similar representations of the world with very strong robotics models. And if that's the case, then if you have a really strong language model, you also have a really strong robotics model [00:22:34] S1: Right. [00:22:35] S2: Using... If we believe in the platonic representation hypothesis. And if that's true, that's... Then beta lesson is... That is the best manifestation of the beta lesson. Right? Because you just need one really strong model regardless of architecture, and they would be outperforming any specific models Yeah. That is slightly weaker.
00:22:35 · Jay, S0
[00:22:35] S2: Using... If we believe in the platonic representation hypothesis. And if that's true, that's... Then beta lesson is... That is the best manifestation of the beta lesson. Right? Because you just need one really strong model regardless of architecture, and they would be outperforming any specific models Yeah. That is slightly weaker. [00:22:56] S0: The interesting thing that... That's maybe not entirely intuitive, and maybe I'd be curious to hear you guys all think about, is why specifically it seems like these newest models, specifically Astra, seems to be so much better at spatial intelligence. You know, we've all seen the demos online of, you know, controlling Blender and making great three d images. Now it seems like, you know, you showed Jay in a benchmark that is, like, a meaningful step function improvement on certain tasks. Tasks. Where do you guys think the the pretraining and and post training, I guess, of those models, what has likely changed about OpenAI's approach there that made them so much better compared to models even six months earlier? Like, you know, that's still we're benefiting from their lesson, and we're still extremely good at code, and we're still solving math problems but haven't quite cracked the spatial intelligence needed here.
00:22:56 · S0, Han Ming
[00:22:56] S0: The interesting thing that... That's maybe not entirely intuitive, and maybe I'd be curious to hear you guys all think about, is why specifically it seems like these newest models, specifically Astra, seems to be so much better at spatial intelligence. You know, we've all seen the demos online of, you know, controlling Blender and making great three d images. Now it seems like, you know, you showed Jay in a benchmark that is, like, a meaningful step function improvement on certain tasks. Tasks. Where do you guys think the the pretraining and and post training, I guess, of those models, what has likely changed about OpenAI's approach there that made them so much better compared to models even six months earlier? Like, you know, that's still we're benefiting from their lesson, and we're still extremely good at code, and we're still solving math problems but haven't quite cracked the spatial intelligence needed here. [00:23:36] S1: I think what Astra does incredibly well is its, like, vision capabilities. It was probably pre trained on way more computer used data than ever before. Startup pretrained on so much like CAD data. And it's like, all of these kinds of data probably teach the model, like, similar, like, understanding of, like, the physical world as, like, a lot of robot data might.