Y Combinator Startup Podcast
00:42:24 · Chelsea Finn
[00:42:24] S0: So I was not planning to do a PhD. I was always planning to go straight to industry. My parents are engineers and worked in industry. And I kind of thought that the way to have impact was to go to a company so forth. And my dad even told me that he wouldn't hire someone with a PhD. And so I was like, maybe I shouldn't get a PhD if I wouldn't be able to get a job. There's also, he's in a different field as well, civil engineering. And so the, but at the same time, I think that like a PhD is an incredible opportunity and I love my PhD. And obviously it depends a lot on the advisor, depends a lot on like what you would be doing and so forth. But I think that PhD is an incredible opportunity to first learn a lot about how to handle uncertainty, how to pick good problems to work on. And I think that basically, in research, no one kind of even gives you the problem to work on. You have to pick the problem, and you don't know for the problem that you picked if it is achievable to make progress on that problem in a six month time span, two year time span, ten year time span. And so you learn about how to deal with that uncertainty. And then the... So that's really useful. And then it also, it also is an opportunity to, like, do amazing research, and do whatever... In many cases, do it, like, have a lot of freedom to work on the research that you find most exciting. And so I think that today is still, yeah, an amazing opportunity to do to work, to learn about uncertainty. I think that learning about uncertainty is really useful in the startup environments, in being at the frontier of AI because we don't know now, like no one knows what the best route is to make these models more and more powerful. At the same time, there's also a lot of incredible opportunities in industry. I think that the first in terms of what goes into developing, like everything that I showed, it's not just the research. There is a whole like software stack that needs to run on the robot, needs to run reliably. There's obviously everything on the hardware side. And then there's also even the machine learning infrastructure, the data infrastructure, and all that. And so even with like... You don't need a PhD necessarily to do a lot of that engineering work. And then on the research side, there's often opportunities to get involved as well.
00:44:44 · Chelsea Finn, S1
And a lot of research is engineering these days as well. And so I think it kinda depends on... Yeah, it's a very personal decision and what you want. Even today, I think I probably, like, kind of retrospectively would want to do a PhD just to learn about how to handle uncertainty to do research, because I really love being at the frontier and thinking about challenging problems. But there's also a lot of really amazing opportunities in both paths. Okay. [00:45:11] S1: The next question is large language models learn from the internet, but robots don't really have an internet scale dataset of physical experience. What's the robotics equivalent and how do we get it?
00:45:23 · Chelsea Finn
[00:45:23] S0: So in robotics, well, so maybe in language models to start off, the data on the web is language data and not all of it high quality, but some of it is really informative and useful. And it is data that in kind of the... That kind of reflects a lot of what you want a model to do. You want it to be able to predict text and to be able to complete text, answer questions, so forth. And there's a lot of questions being answered on the internet and a lot of text that's being completed on the internet. And so in general, with machine learning, you want trained to match test. And so you want the thing that you're gonna be training your model on to be reflective of the thing that you want... You're gonna be asking it to do later on. And so I think the equivalent in robotics is data of robots operating in real world circumstances. And the way that we approach it at Physical Intelligence is to collect data, like to collect robot experience of robots doing all sorts of tasks. You can collect this with teleoperation as, like, kind of to get initial data of robots doing useful things. But in the long run, I think it will also contain a lot of autonomous experience of robots deployed attempting things. And just like how we see in language models, how now a lot of time is spent actually generating data, generating synthetic data by actually running the model and having it think through things. I think a lot of the data in the future in robotics is gonna be the robot attempting to do lots of tasks in lots of real world circumstances. And so, yeah, I think that that's kind of what it looks like. I also think that there are other possible sources of information that's really useful for model training, like videos of people doing things, like YouTube, like web data and captioned images, that tell you, like, this is a kitchen that has a fridge on the right of the sink and so forth. And all of that data, I think, could be really useful for developing a kind of frontier multimodal model that can control robots to do things, reason through how to do a long task, and also control the robot to do those tasks. I think that there's no substitute for the robot experience itself. You can't just like, if you watch a human do something, like if I watch Roger Federer play tennis, it doesn't mean I can't, I can play tennis as well as him, unfortunately.
00:47:44 · Chelsea Finn, S1
And likewise, robots can't like watch a person doing something and then figure out how to do it themselves directly. They really need their experience on their own platform to learn effectively. And I think that, we will need large datasets. I think that that doesn't mean the human video isn't useful. It's useful to watch Roger Federer play tennis, but the, but the actual experience on robot platforms will be a critical component of developing an analogous dataset for robotics. [00:48:13] S1: The next question is, is it possible that general purpose robotics models get democratized via open source the way that large language models did, or will the cost of body data and hardware keep the best models concentrated in a few well resourced labs?
00:48:13 · S1, Chelsea Finn
[00:48:13] S1: The next question is, is it possible that general purpose robotics models get democratized via open source the way that large language models did, or will the cost of body data and hardware keep the best models concentrated in a few well resourced labs? [00:48:28] S0: So I think this is a good question. So I do think the cost of embodied data and hardware will... Could very much make this look different, because I think that it's harder to get data even to distill a model, for example, just readily on the internet. I also think that we've seen kind of pretty large datasets get open source as well, and and pretty powerful models get open sourced. The... I think it's really hard to say exactly what will happen. And so, yeah, I I don't know. The one thing that I will say is that I... Like, with language models, even aside from, like, doc... Democratization like Kimi and and, like, really getting models that, like, perform at the state of the art, even then companies that are focusing a lot on closed source models are also doing a lot of open sourcing. And so, there are, like like like, Gemma, for example, and, the GPT open source and so forth. I think these model... These companies like to support open source because it actually helps build the ecosystem around the things that they're building. And so I imagine there being, I I guess I'm optimistic of that there will be a strong open source community regardless, but I don't know if it will exactly play out exactly the way that language models played out. Okay.
00:48:28 · Chelsea Finn, S1
[00:48:28] S0: So I think this is a good question. So I do think the cost of embodied data and hardware will... Could very much make this look different, because I think that it's harder to get data even to distill a model, for example, just readily on the internet. I also think that we've seen kind of pretty large datasets get open source as well, and and pretty powerful models get open sourced. The... I think it's really hard to say exactly what will happen. And so, yeah, I I don't know. The one thing that I will say is that I... Like, with language models, even aside from, like, doc... Democratization like Kimi and and, like, really getting models that, like, perform at the state of the art, even then companies that are focusing a lot on closed source models are also doing a lot of open sourcing. And so, there are, like like like, Gemma, for example, and, the GPT open source and so forth. I think these model... These companies like to support open source because it actually helps build the ecosystem around the things that they're building. And so I imagine there being, I I guess I'm optimistic of that there will be a strong open source community regardless, but I don't know if it will exactly play out exactly the way that language models played out. Okay. [00:49:43] S1: The next question is, does the model output raw motor commands directly or does it output a target hand position and let a controller solve for the joint angles and what makes that the right level to learn at?