Y Combinator Startup Podcast
00:17:17 · S0, Jay, Han Ming
[00:17:17] S0: So here we can see it seems to be approaching in on this guy. Oh, very dramatic. [00:17:22] S2: Yeah. You can look... You can notice that the latency is a bit slow because we are bottlenecked by the latency of Astra. But if you look at the trends, the latency of these models are improving very rapidly. Right. So one thing that we saw is that for Fable class LLMs, their latency is improving by around two x per month, which is very, [00:17:43] S1: very fast. [00:17:44] S0: Very fast. [00:17:44] S2: Yeah. Yeah. If the trends continue, we could get real time control by end of the year. [00:17:50] S0: So now that we saw this task go from end to end, why don't we walk through what that actually entailed? So there's a coding agent in the loop here running this. Yes. What are the steps that the coding agent would have taken? And let's kind of contrast the direct coding agent version versus the Waddle harness in the loop version. [00:18:06] S2: So what just happened was that in a series of turns, Astra received the images from the cameras, and it outputted the end effect of pose for the robot to go to. And it does it over multiple turns to complete the task. This is less of code as policies, but more of tool calls.
00:18:06 · Jay, Vincent
[00:18:06] S2: So what just happened was that in a series of turns, Astra received the images from the cameras, and it outputted the end effect of pose for the robot to go to. And it does it over multiple turns to complete the task. This is less of code as policies, but more of tool calls. [00:18:27] S3: The unscreen part is pretty repetitive, so you could actually automate a lot of that using code. As as well as actually approaching the bottle, picking it up, like, lot of these things are pretty deterministic once you've done a task quite a few times. The interesting part is where you build in the variation into the coder's policy graph. And so we have a few points of variation. This can come from, for example, when you detect an object. Maybe you use a VLM as part of your tool call. Or when something fails, for example, how do you check that it's failed? How do you then potentially do something else depending on the failure? Like, there's kind of more flexible responses. We tend to put a VLM inside a loop to ensure that, know, while while the coding graph is sort of deterministic, there's points of variation that allow it to generalize.
00:19:13 · Francois, Vincent
[00:19:13] S4: The biggest insight I think I've had, a change in worldview on how to perform machine learning and how to get the rest of the distance on AGI, was, like, a lot of conversations I've had with François Chollet. And in 2020, maybe even 2018 when he did On the Measure of Intelligence, he talked a lot about transduction versus just wrong, versus program induction. And transduction just means that I'm learning a function theta that maps from x's to y's. And why is that wrong? It's just slow, and it's information inefficient. And so to go from Xs to Ys, I need a lot of pairs of Xs and Ys. And if I have a very small amount, then you need a lot of inductive bias, and what you need to act is a good generator to map from Xs to Ys. So now my theta takes in three N pairs of Xs and Ys and emits the function that maps from X to Ys. And that's what code is. So that's... Like, if you give me a coding interview, even a whiteboard, you say, okay. Here's a pudding problem. Here's some examples. Okay. Cool. Write the function. And then we are emitting an f that will map from x to y. [00:20:17] S3: And and this is quite interesting. Right? Because I think... I mean, this is a lot of the traditional program synthesis literature. Mhmm. And I feel like a lot of reason why those methods didn't work as well was because that inductive bias, as you said. Mhmm. Right? It's... You're shifting the difficulty of the problem from finding that mapping to finding the right inductive biases to map it to this, like, smaller code to then do the mapping. [00:20:37] S4: Mhmm.
00:20:37 · Francois, Vincent
[00:20:37] S4: Mhmm. [00:20:37] S3: But finding that set of inductive biases is super hard. But maybe that's what Astra is buying us. Like, even in Cox Eye, there's a giant shift in the literature from, you know, specifically neurosymbolic methods to just using Astra to write code to map. And so maybe that's a cool [00:20:51] S4: way to think about this.... Neurosymbolic. That is neurosymbolic. We have neurons, and then they're emitting symbols with
00:20:51 · Francois, S0
[00:20:51] S4: way to think about this.... Neurosymbolic. That is neurosymbolic. We have neurons, and then they're emitting symbols with [00:20:55] S0: some code. I think it was actually a pretty good segue to a kind of final topic, which is around... Inspired by this paper also from Philip Isola that he titles the Platonic Representation Hypothesis. Mhmm. Right? Where platonic because it's a reference to Plato's cave and, you know, the idea that, you know, they're... You're seeing shadows of various realities. And I think here, the point he's making is that there's extensive evidence to show that language models and various representations of different datas actually learn these distance mappings between similar things under different training policies that are overlapping. And so, like, the idea there would be that as you train these systems on increasingly large amounts of data, they converge to a curve consistent mapping of the world. And maybe this implies that we would expect language models to get better and better at things over time that would make them useful for new tasks like RoboControl. But maybe I'm... I I imagine this representation hypothesis is actually pretty central to all of your guys' worldview or or your view of your companies. And so maybe why don't you guys tell me a little about how you think about this? And, also, maybe we can use that to make some guesses as to why we think models like Astra seem to be so much better at RoboControl than previous models and where... You know, make some predictions for where we might be going here.
00:22:13 · Jay, Han Ming
[00:22:13] S2: Yeah. I think one way to make this concrete for the robotics models versus language models debate is that the very, very strong language models will have very similar representations of the world with very strong robotics models. And if that's the case, then if you have a really strong language model, you also have a really strong robotics model [00:22:34] S1: Right. [00:22:35] S2: Using... If we believe in the platonic representation hypothesis. And if that's true, that's... Then beta lesson is... That is the best manifestation of the beta lesson. Right? Because you just need one really strong model regardless of architecture, and they would be outperforming any specific models Yeah. That is slightly weaker.