00:24:46 · Mario Gabriele, Karol Hausman
[00:24:46] S1: That's such good stories there. The first one is almost like, you know, when a wizard is accepted to Hogwarts or something. It's like you finally found your, you know, collection of of magical people and and the magic that they were doing. The the Sergey story is also so interesting to me because you seem to have gone from skeptical to convinced very, very fast. Like, what was the cycle on on that? Was that like literally the same day, the same week? [00:25:13] S0: I think it was within that lecture. Wow. I think like in the middle of that talk, was like, yep. Makes sense. Is 100% it. What... Whatever I was doing is wrong. This is the right way. I want to do everything I can to work on on this topic. And I I I have feelings like this. I think every researcher has has a feeling like this every now and then, where they just see something and it clicks. And at that point, that's a very powerful feeling. You kind of want to drop everything you've been doing, and you just you just feel like you you found something that that is much closer to truth than what you thought before. I think we all had a similar moment afterwards when we're working on combining large language models with with robot learning. And that was, I think, one more of these... Another of these powerful moments where, like, things start to click. [00:26:05] S1: That was the Taylor Swift demo? Is that the one... That's right. Yeah. Maybe you could tell that story because that is a fascinating one.
00:26:11 · Karol Hausman
[00:26:11] S0: Yeah. So what happened afterwards is I started working with Sergey, there was a few of us who were really getting into into that field. It kinda felt like we were a small group of renegades within the robotics community because there were a lot of problems with deep learning at the time. Was. And some of these problems remain. Like, it was not interpretable. There was no way of really making it modular. It seemed like nobody fully understood how it worked. It wasn't very sample efficient. And there were all of these problems. So, like, at that time, it was extremely unpopular to write any deep learning in robotics paper. It And was kinda like between these two worlds of of machine learning world where any robotics paper seemed kind of like the wrong paper for that venue. The robotics world that never fully embraced or didn't embrace deep learning at that time, but it was still really cool to work with a small group of people and and push these methods forward. So then afterwards, the only place that was really embracing that view was Google Brain. So I decided to join Google Brain right out of my PhD, continue working with Sergey and others on those sort of methods with the basic premise being that the way to to really get it to work is to scale it up. So we're scaling it up. We're trying to figure out how to really get it to work at scale. But again, it started feeling like there is so much complexity in the world that if robots need to... The only way for robots to learn all of this complexity is to experience all of it firsthand, it'll be very difficult to scale. It will be very difficult to to have them learn about logic and, you know, how to break down a task if they had to do it all by themselves. Yes. So then I think around 2022, I I would need to look back to to see what year it was exactly, we started combining it together with Brian Ichter, my my other cofounder. We started combining these robotic methods with large language models. This was before the chat GPT moment. This is where people just started experimenting with with large language models and kind of understanding them better and better. There was this one particular demo. So we we were really excited about this because there was one...
00:28:17 · Karol Hausman
We thought that this would be a path to bringing in a lot of prior knowledge that robots didn't experience firsthand that we learned from the Internet into the robotics world. And that would kind of solve this problem of you having to collect so much data to understand how the world work... How the world works because that that understanding is already embedded in large language models. So if you figure out a way to combine these two, that should really, really help. And we had this one particular experiment where we we were testing how much transfer do you get from this Internet scale knowledge to robotic behavior. And usually when you when you work on robots, work on this very specific task, and then you test that specific task. So you're very rarely surprised. It's it's mostly disappointing because, like, you really want this task to work. You've been working on this task for a very long time, and then it usually doesn't work. But if you work hard enough, eventually you get it to work. But very rarely you you are in a situation where you didn't expect the task to work or, like, it's not the task that you're working on and it works. So very early, you're positively surprised. So we set up a few of these experiments trying to test how much knowledge transfers, and one of them was this experiment with a Coke can in front of a robot and three pictures of different celebrities in front of it. And the prompt we gave to the model we built that combined LLM and and robot models was put the Coke can on picture of Taylor Swift. And one of the pictures was Taylor Swift, and it's... You can see this video on the Internet. That's a pretty pathetic demonstration of what robots could do at the time. But the robot picked it up and then slowly moved it towards Taylor Swift. And that was another one of these moments of huge, huge excitement, even though if you watch the video, it's totally unimpressive because this was... The robot models had never had the chance or had never had any of Taylor Swift in their data. It had to understand the concept of Taylor Swift, connect it to the image of Taylor Swift, and then connect it to the right motion that would move Coke and to the picture of Taylor Swift, all from Internet data.
00:30:24 · Karol Hausman, Mario Gabriele
So that was the moment where it clicked for us that that actually work, where you can bring in a lot of prior knowledge from LLMs, from from the Internet, and connect it to robot motions. And even though the demonstration itself was very unimpressive, just the fact that you can combine these two knowledge sources, that was really, really, really impressive for us, and it felt like it opened another door. [00:30:47] S1: Yeah. It feels like... I don't know if you're familiar. I'm sure you are. I can't believe I'm I'm asking if you're familiar. You know much more about this than I do. But there was some experiment in, like, the seventies, I think, called, like, SHRDLU or something like that, where the entire approach was to try and teach AI all the rules of the world programmatically and to say, you know, you know, here is the category of what a bird is. It has these characteristics, and then here's the category of a bat. It has some overlapping character, you know, all of these sorts of things. But but, essentially, know, people are familiar with this concept now because of the way we use LLMs, but it was almost as if the robot suddenly inherited all of these rules and, you know, pieces of knowledge from tying them up with the LLMs that allows the the Taylor Swift demo to happen.
00:31:36 · Karol Hausman, Mario Gabriele
[00:31:36] S0: Yeah. That's that's exactly right. And I think there's maybe two points there. One is that we made the same mistake in robotics. We wanted to write all of these rules. We thought that if we... If only we had enough of these rules, the robots would be able to follow them and do the right thing. But kind of like we said about the inner game of tennis, you know, you can't just write all of the rules. You kind of have to do it. And there is some underlying structure, but you can't just fully put your finger on it of what it is. You need to learn it from data. And I think people thought about it similarly in in language as well. They thought that, you know, if only we could write all of the rules Yes. That would be enough. But it turns out that there is, you know, trillions or billions of these rules, and sometimes we can't fully even express them in language. You just need to learn them. And if you do, then you would be able to follow them even though you still don't understand what fully... What what they are. So I think we're learning this lesson over and over again. [00:32:33] S1: And so, you know, you have this almost third eureka moment for yourself where the technology has, you know, taken yet another jump. By that point, it sounds like you have most of the the physical intelligence cofounders around the table. But how did you sort of pull the last few members aboard and decide to make that leap?
00:32:54 · Karol Hausman
[00:32:54] S0: Yeah. I think at that point, it started becoming clear that it could be possible. And if you worked on something for so long, at that point that was basically my entire adult life, fifteen years or so. And for a long time you thought that there was no solution to this problem. There were some times where it felt a little bit more tangible, but it never felt like it could actually be solved. And then you have this moment where for the first time you see the light at the end of the tunnel. Like maybe, just maybe, if we do everything right, if we combine it with Internet knowledge, if we scale it up, if we if we do all of the pieces that need to be done, it might work. If you see that light at the end of the tunnel, you you can't unsee it. You you want to do something about it. And at that point, it became clear that the way to to accomplish this is to create an organization whose sole purpose is to solve physical intelligence, is to solve this problem. It can be solved as, like, priority number 20 in another organization. The only reason to to exist for this organization would be to solve physical intelligence. So then the next question is how do we actually do it and what are the what are the the conditions to to make it happen? And what became clear immediately is you need to have truly the best people, truly the best researchers to make it happen. Because the second best team in those areas doesn't doesn't really work with a company like that. You need to have the top top people in the field. So that one actually wasn't that hard because we were already working with each other for a very long time. So I happened to be friends with the best people in the field already. So it was just a matter of making sure that we all want to do it and we're ready for it. The other condition was that you need to get access to a lot of funding. It would require very long term bet and investors that are fully aligned with this taking some time and this starting as research company and not being oriented around revenue or around the short term revenue. So it was the second requirement, and the third requirement was just like build a very very... Build an incredible company with the right focus, with the right people, with the expertise in all the other areas that we didn't have expertise in, hardware operations and and things like this.