00:28:17 · Karol Hausman
We thought that this would be a path to bringing in a lot of prior knowledge that robots didn't experience firsthand that we learned from the Internet into the robotics world. And that would kind of solve this problem of you having to collect so much data to understand how the world work... How the world works because that that understanding is already embedded in large language models. So if you figure out a way to combine these two, that should really, really help. And we had this one particular experiment where we we were testing how much transfer do you get from this Internet scale knowledge to robotic behavior. And usually when you when you work on robots, work on this very specific task, and then you test that specific task. So you're very rarely surprised. It's it's mostly disappointing because, like, you really want this task to work. You've been working on this task for a very long time, and then it usually doesn't work. But if you work hard enough, eventually you get it to work. But very rarely you you are in a situation where you didn't expect the task to work or, like, it's not the task that you're working on and it works. So very early, you're positively surprised. So we set up a few of these experiments trying to test how much knowledge transfers, and one of them was this experiment with a Coke can in front of a robot and three pictures of different celebrities in front of it. And the prompt we gave to the model we built that combined LLM and and robot models was put the Coke can on picture of Taylor Swift. And one of the pictures was Taylor Swift, and it's... You can see this video on the Internet. That's a pretty pathetic demonstration of what robots could do at the time. But the robot picked it up and then slowly moved it towards Taylor Swift. And that was another one of these moments of huge, huge excitement, even though if you watch the video, it's totally unimpressive because this was... The robot models had never had the chance or had never had any of Taylor Swift in their data. It had to understand the concept of Taylor Swift, connect it to the image of Taylor Swift, and then connect it to the right motion that would move Coke and to the picture of Taylor Swift, all from Internet data.
00:30:24 · Karol Hausman, Mario Gabriele
So that was the moment where it clicked for us that that actually work, where you can bring in a lot of prior knowledge from LLMs, from from the Internet, and connect it to robot motions. And even though the demonstration itself was very unimpressive, just the fact that you can combine these two knowledge sources, that was really, really, really impressive for us, and it felt like it opened another door. [00:30:47] S1: Yeah. It feels like... I don't know if you're familiar. I'm sure you are. I can't believe I'm I'm asking if you're familiar. You know much more about this than I do. But there was some experiment in, like, the seventies, I think, called, like, SHRDLU or something like that, where the entire approach was to try and teach AI all the rules of the world programmatically and to say, you know, you know, here is the category of what a bird is. It has these characteristics, and then here's the category of a bat. It has some overlapping character, you know, all of these sorts of things. But but, essentially, know, people are familiar with this concept now because of the way we use LLMs, but it was almost as if the robot suddenly inherited all of these rules and, you know, pieces of knowledge from tying them up with the LLMs that allows the the Taylor Swift demo to happen.
00:31:36 · Karol Hausman, Mario Gabriele
[00:31:36] S0: Yeah. That's that's exactly right. And I think there's maybe two points there. One is that we made the same mistake in robotics. We wanted to write all of these rules. We thought that if we... If only we had enough of these rules, the robots would be able to follow them and do the right thing. But kind of like we said about the inner game of tennis, you know, you can't just write all of the rules. You kind of have to do it. And there is some underlying structure, but you can't just fully put your finger on it of what it is. You need to learn it from data. And I think people thought about it similarly in in language as well. They thought that, you know, if only we could write all of the rules Yes. That would be enough. But it turns out that there is, you know, trillions or billions of these rules, and sometimes we can't fully even express them in language. You just need to learn them. And if you do, then you would be able to follow them even though you still don't understand what fully... What what they are. So I think we're learning this lesson over and over again. [00:32:33] S1: And so, you know, you have this almost third eureka moment for yourself where the technology has, you know, taken yet another jump. By that point, it sounds like you have most of the the physical intelligence cofounders around the table. But how did you sort of pull the last few members aboard and decide to make that leap?
00:32:54 · Karol Hausman
[00:32:54] S0: Yeah. I think at that point, it started becoming clear that it could be possible. And if you worked on something for so long, at that point that was basically my entire adult life, fifteen years or so. And for a long time you thought that there was no solution to this problem. There were some times where it felt a little bit more tangible, but it never felt like it could actually be solved. And then you have this moment where for the first time you see the light at the end of the tunnel. Like maybe, just maybe, if we do everything right, if we combine it with Internet knowledge, if we scale it up, if we if we do all of the pieces that need to be done, it might work. If you see that light at the end of the tunnel, you you can't unsee it. You you want to do something about it. And at that point, it became clear that the way to to accomplish this is to create an organization whose sole purpose is to solve physical intelligence, is to solve this problem. It can be solved as, like, priority number 20 in another organization. The only reason to to exist for this organization would be to solve physical intelligence. So then the next question is how do we actually do it and what are the what are the the conditions to to make it happen? And what became clear immediately is you need to have truly the best people, truly the best researchers to make it happen. Because the second best team in those areas doesn't doesn't really work with a company like that. You need to have the top top people in the field. So that one actually wasn't that hard because we were already working with each other for a very long time. So I happened to be friends with the best people in the field already. So it was just a matter of making sure that we all want to do it and we're ready for it. The other condition was that you need to get access to a lot of funding. It would require very long term bet and investors that are fully aligned with this taking some time and this starting as research company and not being oriented around revenue or around the short term revenue. So it was the second requirement, and the third requirement was just like build a very very... Build an incredible company with the right focus, with the right people, with the expertise in all the other areas that we didn't have expertise in, hardware operations and and things like this.
00:35:13 · Karol Hausman, Mario Gabriele
I spend most of my time figuring out these two points. How do we... How can we find the right investors that can that can really help us in this in this adventure, and they're fully aligned with how we want to do this? And then how can we find the the right people to to fill all the missing pieces? [00:35:32] S1: How did Lachy Groom join the crew? Because he's obviously had a very impressive career as an operator, as an investor, but isn't someone who one would automatically think, you know, this is someone who's gonna devote the next chapter of their life to robotics. So I didn't know Lachy before we started thinking about starting a company.
00:35:52 · Karol Hausman
[00:35:52] S0: But then... And he should tell the story rather than me, but I believe what was what was happening is that he was in... He's been investing in that point for a few years. And as long as he's been investing, he always thought that he doesn't wanna be investing forever. He wants to find something else that he can fully devote himself to and he wants to build things. And the one area he was always fascinated by was robotics. And he was seeing over the past few years that there was a lot of innovations happening there. There was some kind of moment. Robotics was having its moment. And particularly, he was impressed. He was seeing over and over papers coming from Chelsea Finn and Sergey Levine's lab, as well as our papers from Google. So he he talked to a lot of a lot of his friends that if ever Chelsea, Sergey, or Karol or anybody from those teams are thinking of starting a company, please connect me to them. I would at least want to invest or at least talk to them. So that's when we got introduced by by a friend, and we pitched to Lachy as a way to to to have him invest. And at the end of that pitch, it became clear that he would like to do much more than just invest. This is the opportunity he's been looking for for all the years he's been investing, and he really wanted to to to dive in. So then the next step was us trying to get to know each other as quickly as possible, spend as much time together as as we could. And and then as we were doing that, it also became clear that this is the person that that I've been looking for that could help us on on the on the commercial side, on fundraising side, on operation side, kind of one