Y Combinator Startup Podcast
00:45:23 · Chelsea Finn
[00:45:23] S0: So in robotics, well, so maybe in language models to start off, the data on the web is language data and not all of it high quality, but some of it is really informative and useful. And it is data that in kind of the... That kind of reflects a lot of what you want a model to do. You want it to be able to predict text and to be able to complete text, answer questions, so forth. And there's a lot of questions being answered on the internet and a lot of text that's being completed on the internet. And so in general, with machine learning, you want trained to match test. And so you want the thing that you're gonna be training your model on to be reflective of the thing that you want... You're gonna be asking it to do later on. And so I think the equivalent in robotics is data of robots operating in real world circumstances. And the way that we approach it at Physical Intelligence is to collect data, like to collect robot experience of robots doing all sorts of tasks. You can collect this with teleoperation as, like, kind of to get initial data of robots doing useful things. But in the long run, I think it will also contain a lot of autonomous experience of robots deployed attempting things. And just like how we see in language models, how now a lot of time is spent actually generating data, generating synthetic data by actually running the model and having it think through things. I think a lot of the data in the future in robotics is gonna be the robot attempting to do lots of tasks in lots of real world circumstances. And so, yeah, I think that that's kind of what it looks like. I also think that there are other possible sources of information that's really useful for model training, like videos of people doing things, like YouTube, like web data and captioned images, that tell you, like, this is a kitchen that has a fridge on the right of the sink and so forth. And all of that data, I think, could be really useful for developing a kind of frontier multimodal model that can control robots to do things, reason through how to do a long task, and also control the robot to do those tasks. I think that there's no substitute for the robot experience itself. You can't just like, if you watch a human do something, like if I watch Roger Federer play tennis, it doesn't mean I can't, I can play tennis as well as him, unfortunately.
00:47:44 · Chelsea Finn, S1
And likewise, robots can't like watch a person doing something and then figure out how to do it themselves directly. They really need their experience on their own platform to learn effectively. And I think that, we will need large datasets. I think that that doesn't mean the human video isn't useful. It's useful to watch Roger Federer play tennis, but the, but the actual experience on robot platforms will be a critical component of developing an analogous dataset for robotics. [00:48:13] S1: The next question is, is it possible that general purpose robotics models get democratized via open source the way that large language models did, or will the cost of body data and hardware keep the best models concentrated in a few well resourced labs?
00:48:13 · S1, Chelsea Finn
[00:48:13] S1: The next question is, is it possible that general purpose robotics models get democratized via open source the way that large language models did, or will the cost of body data and hardware keep the best models concentrated in a few well resourced labs? [00:48:28] S0: So I think this is a good question. So I do think the cost of embodied data and hardware will... Could very much make this look different, because I think that it's harder to get data even to distill a model, for example, just readily on the internet. I also think that we've seen kind of pretty large datasets get open source as well, and and pretty powerful models get open sourced. The... I think it's really hard to say exactly what will happen. And so, yeah, I I don't know. The one thing that I will say is that I... Like, with language models, even aside from, like, doc... Democratization like Kimi and and, like, really getting models that, like, perform at the state of the art, even then companies that are focusing a lot on closed source models are also doing a lot of open sourcing. And so, there are, like like like, Gemma, for example, and, the GPT open source and so forth. I think these model... These companies like to support open source because it actually helps build the ecosystem around the things that they're building. And so I imagine there being, I I guess I'm optimistic of that there will be a strong open source community regardless, but I don't know if it will exactly play out exactly the way that language models played out. Okay.
00:48:28 · Chelsea Finn, S1
[00:48:28] S0: So I think this is a good question. So I do think the cost of embodied data and hardware will... Could very much make this look different, because I think that it's harder to get data even to distill a model, for example, just readily on the internet. I also think that we've seen kind of pretty large datasets get open source as well, and and pretty powerful models get open sourced. The... I think it's really hard to say exactly what will happen. And so, yeah, I I don't know. The one thing that I will say is that I... Like, with language models, even aside from, like, doc... Democratization like Kimi and and, like, really getting models that, like, perform at the state of the art, even then companies that are focusing a lot on closed source models are also doing a lot of open sourcing. And so, there are, like like like, Gemma, for example, and, the GPT open source and so forth. I think these model... These companies like to support open source because it actually helps build the ecosystem around the things that they're building. And so I imagine there being, I I guess I'm optimistic of that there will be a strong open source community regardless, but I don't know if it will exactly play out exactly the way that language models played out. Okay. [00:49:43] S1: The next question is, does the model output raw motor commands directly or does it output a target hand position and let a controller solve for the joint angles and what makes that the right level to learn at?
00:49:43 · S1, Chelsea Finn
[00:49:43] S1: The next question is, does the model output raw motor commands directly or does it output a target hand position and let a controller solve for the joint angles and what makes that the right level to learn at? [00:49:56] S0: So the model... All the models that I showed were outputting target joint positions. So like, what is the angle of this joint? What is the angle of this joint and so forth that you want to hit? And then there's a controller, like a PD controller that is trying to then hit that target position for those joints. The model actually is also trained to predict target gripper positions, like where in three d space should my gripper be. And you could also use that as well and back out the joint positions. You could also... Like, another thing you could do is you could go directly to, like, motor torques or to, like, voltages or efforts. There are pros and cons of different options. We have found controlling joints, and controlling in the three d space of the gripper to both work well. And so there's pros and cons. I think the one thing that would be nice about going directly to the voltages is that you could also get a more stiff output or a less stiff output, whereas the controller, if you have a fixed controller, then you're not letting your model control that aspect. And so there's different pros and cons. Think that what we're working seems to work. It doesn't seem to be a bottleneck. And I often like to focus on the things that seem to be bottlenecks versus things that don't seem to be bottlenecks. Okay.
00:49:56 · Chelsea Finn, S1
[00:49:56] S0: So the model... All the models that I showed were outputting target joint positions. So like, what is the angle of this joint? What is the angle of this joint and so forth that you want to hit? And then there's a controller, like a PD controller that is trying to then hit that target position for those joints. The model actually is also trained to predict target gripper positions, like where in three d space should my gripper be. And you could also use that as well and back out the joint positions. You could also... Like, another thing you could do is you could go directly to, like, motor torques or to, like, voltages or efforts. There are pros and cons of different options. We have found controlling joints, and controlling in the three d space of the gripper to both work well. And so there's pros and cons. I think the one thing that would be nice about going directly to the voltages is that you could also get a more stiff output or a less stiff output, whereas the controller, if you have a fixed controller, then you're not letting your model control that aspect. And so there's different pros and cons. Think that what we're working seems to work. It doesn't seem to be a bottleneck. And I often like to focus on the things that seem to be bottlenecks versus things that don't seem to be bottlenecks. Okay. [00:51:14] S1: Next question is, do robots need something like imagination? Well, the ability to picture what should happen next before they can become truly useful.