Y Combinator Startup Podcast
00:40:42 · Chelsea Finn
[00:40:42] S0: So that's a good question. I think that the... I mean, at the very least, I actually think that just starting with the generalist policy and then fine tuning it, even like right off the bat, can be really effective. And we've, And fortunately, a lot of generalist policies, like they're actually really powerful generalist policies that are open source. And so the Pi0 and Pi0.5 models are open source, for example, and we've seen a lot of people, get a lot of use out of those models already. And we're also working with a lot of partners like the tractor company, like Ultra, like Weave, to develop to take our most recent models and get even more juice out of them, even more powerful models for their own applications. So even right off the bat, I think that you can use them. The only scenario in which I actually wouldn't use them is if you're really in a really constrained environment. I've talked to some folks working on surgical robots that are in an Operating Room in the basement with no internet connection and a really bad GPU. And so sometimes it's just really hard to use a larger model, but you still can't do local inference on a workstation with these models. And so I think that, yeah, right away, just taking Pi0.5 or your favorite model and fine tuning it is the way to go. And I think that we'll see lots of these small companies. And I think there's, like, yeah, so much work to do in terms of actually getting these robots to work, with this technology in the real world. Great.
00:42:12 · S1
[00:42:12] S1: The next question is, given how fast robotics is moving in industry, what are the real advantages and drawbacks of doing a PhD today, especially for someone who wants to go into industry afterwards?
00:42:24 · Chelsea Finn
[00:42:24] S0: So I was not planning to do a PhD. I was always planning to go straight to industry. My parents are engineers and worked in industry. And I kind of thought that the way to have impact was to go to a company so forth. And my dad even told me that he wouldn't hire someone with a PhD. And so I was like, maybe I shouldn't get a PhD if I wouldn't be able to get a job. There's also, he's in a different field as well, civil engineering. And so the, but at the same time, I think that like a PhD is an incredible opportunity and I love my PhD. And obviously it depends a lot on the advisor, depends a lot on like what you would be doing and so forth. But I think that PhD is an incredible opportunity to first learn a lot about how to handle uncertainty, how to pick good problems to work on. And I think that basically, in research, no one kind of even gives you the problem to work on. You have to pick the problem, and you don't know for the problem that you picked if it is achievable to make progress on that problem in a six month time span, two year time span, ten year time span. And so you learn about how to deal with that uncertainty. And then the... So that's really useful. And then it also, it also is an opportunity to, like, do amazing research, and do whatever... In many cases, do it, like, have a lot of freedom to work on the research that you find most exciting. And so I think that today is still, yeah, an amazing opportunity to do to work, to learn about uncertainty. I think that learning about uncertainty is really useful in the startup environments, in being at the frontier of AI because we don't know now, like no one knows what the best route is to make these models more and more powerful. At the same time, there's also a lot of incredible opportunities in industry. I think that the first in terms of what goes into developing, like everything that I showed, it's not just the research. There is a whole like software stack that needs to run on the robot, needs to run reliably. There's obviously everything on the hardware side. And then there's also even the machine learning infrastructure, the data infrastructure, and all that. And so even with like... You don't need a PhD necessarily to do a lot of that engineering work. And then on the research side, there's often opportunities to get involved as well.
00:44:44 · Chelsea Finn, S1
And a lot of research is engineering these days as well. And so I think it kinda depends on... Yeah, it's a very personal decision and what you want. Even today, I think I probably, like, kind of retrospectively would want to do a PhD just to learn about how to handle uncertainty to do research, because I really love being at the frontier and thinking about challenging problems. But there's also a lot of really amazing opportunities in both paths. Okay. [00:45:11] S1: The next question is large language models learn from the internet, but robots don't really have an internet scale dataset of physical experience. What's the robotics equivalent and how do we get it?
00:45:23 · Chelsea Finn
[00:45:23] S0: So in robotics, well, so maybe in language models to start off, the data on the web is language data and not all of it high quality, but some of it is really informative and useful. And it is data that in kind of the... That kind of reflects a lot of what you want a model to do. You want it to be able to predict text and to be able to complete text, answer questions, so forth. And there's a lot of questions being answered on the internet and a lot of text that's being completed on the internet. And so in general, with machine learning, you want trained to match test. And so you want the thing that you're gonna be training your model on to be reflective of the thing that you want... You're gonna be asking it to do later on. And so I think the equivalent in robotics is data of robots operating in real world circumstances. And the way that we approach it at Physical Intelligence is to collect data, like to collect robot experience of robots doing all sorts of tasks. You can collect this with teleoperation as, like, kind of to get initial data of robots doing useful things. But in the long run, I think it will also contain a lot of autonomous experience of robots deployed attempting things. And just like how we see in language models, how now a lot of time is spent actually generating data, generating synthetic data by actually running the model and having it think through things. I think a lot of the data in the future in robotics is gonna be the robot attempting to do lots of tasks in lots of real world circumstances. And so, yeah, I think that that's kind of what it looks like. I also think that there are other possible sources of information that's really useful for model training, like videos of people doing things, like YouTube, like web data and captioned images, that tell you, like, this is a kitchen that has a fridge on the right of the sink and so forth. And all of that data, I think, could be really useful for developing a kind of frontier multimodal model that can control robots to do things, reason through how to do a long task, and also control the robot to do those tasks. I think that there's no substitute for the robot experience itself. You can't just like, if you watch a human do something, like if I watch Roger Federer play tennis, it doesn't mean I can't, I can play tennis as well as him, unfortunately.
00:47:44 · Chelsea Finn, S1
And likewise, robots can't like watch a person doing something and then figure out how to do it themselves directly. They really need their experience on their own platform to learn effectively. And I think that, we will need large datasets. I think that that doesn't mean the human video isn't useful. It's useful to watch Roger Federer play tennis, but the, but the actual experience on robot platforms will be a critical component of developing an analogous dataset for robotics. [00:48:13] S1: The next question is, is it possible that general purpose robotics models get democratized via open source the way that large language models did, or will the cost of body data and hardware keep the best models concentrated in a few well resourced labs?