Y Combinator Startup Podcast
00:23:58 · S0, Han Ming
[00:23:58] S0: So the computer use data is an interesting point because it's not totally intuitive, I think, why computer use data is useful for understanding the physical world and that it, you know, tends to be like a computer where you're clicking around. Like, say more about why you think that is a big unlock because I totally buy they trained way more computer use data than ever before. [00:24:13] S1: Right. I think immediately, if you were just training a model for computer use, that would not be able apply for robotics. But if you feed a computer use data into a big model like Astra... And by computer use data, I mean, like, you drag a cursor around on a screen to orbit some CAD object in order to design in Blender. This tells you, like, how to reason about spaces. It at least tells you about, like, top down, left, right, all these concepts that you need to control a robot. Right? And I think that's why this data helps so much for making these LLMs so much better at robot use. I think, like, from the other direction, like, a lot of people in the robotics community were trying more and more different kinds of data as well. Like, egocentric videos kinda went from just teleoperation to to, like, you know, broader range of this data because it's not just robot data that can teach a model how to use a robot. So if we take this to the extreme, it's like, not feed every kind of data, coding, computer use, egocentric, into the same model? I think that's how we get to the most capable, like, robot use agent.
00:25:11 · Francois, Han Ming
[00:25:11] S4: And it's so funny that, like, you know, if you go to nineteen eighties, Xerox PARC or whatever, like, we literally made the graphical user interface to be more like the physical world so that we could interface with it. And, like, what we ended up doing is building an environment that was actually helpful for robotics to learn how to use the physical world. Right? We we have file systems. We have, like, files. We have folders. We have, you know, the GUIs for, like, SolidWorks and, like, Autodesk to, like, spin around stuff to make it simulate... Similar to the physical world. And then, like, we couldn't get robots to work in the physical world, so we just trained on that. And then now it works on the physical Exactly. [00:25:46] S1: There were incredible papers, I think, from Princeton and also robotics touches on this. It's like, where you just... If you design a right harness where you make the tools that interface with the robot look like computer used tools, like you have a agent drag a cursor around to control where the robot goes, That improves how well the LLM is able to perform on these physical tasks.
00:25:46 · Han Ming, S0, Jay
[00:25:46] S1: There were incredible papers, I think, from Princeton and also robotics touches on this. It's like, where you just... If you design a right harness where you make the tools that interface with the robot look like computer used tools, like you have a agent drag a cursor around to control where the robot goes, That improves how well the LLM is able to perform on these physical tasks. [00:26:05] S0: Where do you guys see... You know, given, like, I think reasonable guesses as to where the base models are gonna continue improving and your guys' own investments in either evaluating these models or building harnesses around them. What do you think is gonna be capabilities that we maybe now see as challenging to do, but that are gonna be increasingly possible or even trivial for a few months from now? Like, you know, six months ago, even the demonstrations that I've seen you guys post on your Twitter, I think, would have been kinda mind blowing to imagine coming from an LM. [00:26:32] S2: There's some consensus within the Frontier Labs and also in the robotics foundation models companies that we will have general purpose robots within the next two years or even earlier. And this is something that society is probably unaware of or even unprepared for. And when we say general purpose robots, we mean something like if you give any natural language instruction, it can do what a competent teenager could do with their bare hands. It's kinda like the ChatGPT moment, but for robotics in terms of capabilities where it can generalize to unseen tasks and unseen environments.
00:26:32 · Jay, S0, Han Ming
[00:26:32] S2: There's some consensus within the Frontier Labs and also in the robotics foundation models companies that we will have general purpose robots within the next two years or even earlier. And this is something that society is probably unaware of or even unprepared for. And when we say general purpose robots, we mean something like if you give any natural language instruction, it can do what a competent teenager could do with their bare hands. It's kinda like the ChatGPT moment, but for robotics in terms of capabilities where it can generalize to unseen tasks and unseen environments. [00:27:06] S0: For you guys, for the Waddle Labs guys, what does that mean for what you guys are building? [00:27:10] S1: I think we're gonna take steps to get there in about two years' time. I think there's still many challenges that are very visible. For example, when I left the online talk about latency, this is a big problem. If you just have Astra being in the loop thinking at every step, this is really slow. It's not gonna be economically, like, useful. So how do you, you know, consolidate kind of a first pass by Astra, like, this this contextual learning into, like, a faster skill or policy that you can then run repeatedly at incredibly high throughput?
00:27:40 · Francois
[00:27:40] S4: I think it's the the mappings to humans will get more and more like this, where, like, the the the optimal thing to do may be to to put a new s a r, state action reward, back into context. It's very quick. But then there needs to be some go to sleep for a while. Almost everything that is intelligent sleeps. Like, tell me an intelligent system that doesn't sleep in some way. And then during sleep, compression happens. The weird thing that happens from your hippocampus and it shortwave ripples to both lobes and like there's weird, you know, paths from memories that were compressed throughout the day to train the weights. And similarly, maybe what the right thing to do is similar to DAgger, data segregation framework in our classic RL, where you're you're going, you're collecting a bunch of data, and then you're somewhat reflecting on it, and then you're using it to update your wait file. And then you have distillation in... From those experiences back into an updated wait file that maybe isn't Astra, but maybe is your own models.
00:28:36 · Vincent, S0
[00:28:36] S3: It sounds very DreamCoder Mhmm. Esque. Mhmm. But a lot of those specific tools, I think, that people used to build... To take DreamCoder, for example. Right? It had this library of skills, and then during the sleep phase, it basically refactored everything into more compact representation and so on. I mean, we're seeing sort of similar things just happen not as rigidly as before in the space of programs, but now it's maybe, like, maybe you wanna refactor traces. Maybe you want to refactor skills. A lot of these robotic coders policy papers, right, they have this growing library of skills, and it's anyone's guess really how you print them, how you organize them, and things like that. So, yeah, I think as as Waddle goes forward, like, how you manage our growing context of skills of deployment data, all of that is going to be quite interesting. [00:29:19] S0: I think with that, I think this was an excellent discussion. Thank you so much, Vincent, Han Ming, and Jay for being here. I'm very excited for all the incredible robotics advancements I think, we're gonna see over the next few years, and I think you're totally right, Jay, that I don't think broader society is totally aware of how much is coming. And it's gonna be a very incredible few years to come. So thank you so much.