00:21:09 · Karol Hausman
So I immediately left the room, knocked because there's basically no one there. And I knocked at the at the at the room next door and asked if I could work there. And, basically, I was completely unqualified. Had no idea how to program these robots. I was not familiar with any of the tools, but the person who was there, it was another one of those people that gave me a chance and and asked me to just come in on Monday and see what I could do. Wow. So I got involved, got a job at that at that lab. That's how I got into finally working on intelligent robots. So, yeah, I I think this is like yet another person that I'm really grateful to. And then afterwards, I I moved to The US, got into a PhD program studying studying intelligent robots. And at that point, I was so stoked that, like, I finally get to do this and I finally found a small group of people that work on those things. It didn't matter that, you know, it doesn't work that well. I was so excited that I finally get to, you know, meet the people that are that are working on this. I can learn from them. I can I can push it forward, and and I can finally work on the thing that I always wanted to work on? So I did this throughout my PhD, and I had another moment like this during my PhD, this kind of defining moment where I was working on a certain set of problems where it was referred to as as active perception. And I can I can go a little bit deeper into this if you're interested? But at that point, it kinda felt like there are ways to write... To to to write papers, to to progress in your PhD, but there isn't anything that felt like it could actually solve this problem. It it all felt like the world is too complex. You can push it to some extent, this technology, but you you won't really get to the finish line. They're not they're not kind of adding up. You don't really see a path out of it. And there was another moment where a poster candidate stopped by our lab. And at that time, when a poster candidate candidate stops by, you are supposed as a PhD student, show them your work and kind of talk with them as part of the interview. So I showed to that post that candidate stuff that I was working on, and he listened to all of it. And then at the end of it, he said that I should drop all of this and switch to deep learning. That's the way to to solve the problem that I was actually working on.
00:23:27 · Karol Hausman
At that point, I just thought to myself that he had no idea what he's talking about, and he probably didn't listen to me. Because at that point, I think I spent already two years or something like this working on the topic I was working on. But then later that day, I went to his lecture where he showed what what he was working on. And that was another one of those moments where it was extremely eye opening because for the first time, saw something that, like, could actually work. Where it was less of like a, you know, here's just a little paper over there and a little paper over there, but they don't add up. But it's something that that kind of brought all the pieces together. And, yeah, that was another one of those moments, like the the similar one to that... To the one I've experienced in Munich where at that point, I decided to to drop my PhD topic, change it completely, start collaborating with that person, and do everything I can to push deep learning in robotics. And that person was Sergey Levine, who is now my cofounder at Physical Intelligence and one of the pioneers of of deep learning in in robotics. And, yeah, I I never looked back after that. But to kind of... This is a long way of answering your question, but it didn't feel lonely in that, you know, I was just really excited to be working on that problem.
00:24:46 · Mario Gabriele, Karol Hausman
[00:24:46] S1: That's such good stories there. The first one is almost like, you know, when a wizard is accepted to Hogwarts or something. It's like you finally found your, you know, collection of of magical people and and the magic that they were doing. The the Sergey story is also so interesting to me because you seem to have gone from skeptical to convinced very, very fast. Like, what was the cycle on on that? Was that like literally the same day, the same week? [00:25:13] S0: I think it was within that lecture. Wow. I think like in the middle of that talk, was like, yep. Makes sense. Is 100% it. What... Whatever I was doing is wrong. This is the right way. I want to do everything I can to work on on this topic. And I I I have feelings like this. I think every researcher has has a feeling like this every now and then, where they just see something and it clicks. And at that point, that's a very powerful feeling. You kind of want to drop everything you've been doing, and you just you just feel like you you found something that that is much closer to truth than what you thought before. I think we all had a similar moment afterwards when we're working on combining large language models with with robot learning. And that was, I think, one more of these... Another of these powerful moments where, like, things start to click. [00:26:05] S1: That was the Taylor Swift demo? Is that the one... That's right. Yeah. Maybe you could tell that story because that is a fascinating one.
00:26:11 · Karol Hausman
[00:26:11] S0: Yeah. So what happened afterwards is I started working with Sergey, there was a few of us who were really getting into into that field. It kinda felt like we were a small group of renegades within the robotics community because there were a lot of problems with deep learning at the time. Was. And some of these problems remain. Like, it was not interpretable. There was no way of really making it modular. It seemed like nobody fully understood how it worked. It wasn't very sample efficient. And there were all of these problems. So, like, at that time, it was extremely unpopular to write any deep learning in robotics paper. It And was kinda like between these two worlds of of machine learning world where any robotics paper seemed kind of like the wrong paper for that venue. The robotics world that never fully embraced or didn't embrace deep learning at that time, but it was still really cool to work with a small group of people and and push these methods forward. So then afterwards, the only place that was really embracing that view was Google Brain. So I decided to join Google Brain right out of my PhD, continue working with Sergey and others on those sort of methods with the basic premise being that the way to to really get it to work is to scale it up. So we're scaling it up. We're trying to figure out how to really get it to work at scale. But again, it started feeling like there is so much complexity in the world that if robots need to... The only way for robots to learn all of this complexity is to experience all of it firsthand, it'll be very difficult to scale. It will be very difficult to to have them learn about logic and, you know, how to break down a task if they had to do it all by themselves. Yes. So then I think around 2022, I I would need to look back to to see what year it was exactly, we started combining it together with Brian Ichter, my my other cofounder. We started combining these robotic methods with large language models. This was before the chat GPT moment. This is where people just started experimenting with with large language models and kind of understanding them better and better. There was this one particular demo. So we we were really excited about this because there was one...
00:28:17 · Karol Hausman
We thought that this would be a path to bringing in a lot of prior knowledge that robots didn't experience firsthand that we learned from the Internet into the robotics world. And that would kind of solve this problem of you having to collect so much data to understand how the world work... How the world works because that that understanding is already embedded in large language models. So if you figure out a way to combine these two, that should really, really help. And we had this one particular experiment where we we were testing how much transfer do you get from this Internet scale knowledge to robotic behavior. And usually when you when you work on robots, work on this very specific task, and then you test that specific task. So you're very rarely surprised. It's it's mostly disappointing because, like, you really want this task to work. You've been working on this task for a very long time, and then it usually doesn't work. But if you work hard enough, eventually you get it to work. But very rarely you you are in a situation where you didn't expect the task to work or, like, it's not the task that you're working on and it works. So very early, you're positively surprised. So we set up a few of these experiments trying to test how much knowledge transfers, and one of them was this experiment with a Coke can in front of a robot and three pictures of different celebrities in front of it. And the prompt we gave to the model we built that combined LLM and and robot models was put the Coke can on picture of Taylor Swift. And one of the pictures was Taylor Swift, and it's... You can see this video on the Internet. That's a pretty pathetic demonstration of what robots could do at the time. But the robot picked it up and then slowly moved it towards Taylor Swift. And that was another one of these moments of huge, huge excitement, even though if you watch the video, it's totally unimpressive because this was... The robot models had never had the chance or had never had any of Taylor Swift in their data. It had to understand the concept of Taylor Swift, connect it to the image of Taylor Swift, and then connect it to the right motion that would move Coke and to the picture of Taylor Swift, all from Internet data.
00:30:24 · Karol Hausman, Mario Gabriele
So that was the moment where it clicked for us that that actually work, where you can bring in a lot of prior knowledge from LLMs, from from the Internet, and connect it to robot motions. And even though the demonstration itself was very unimpressive, just the fact that you can combine these two knowledge sources, that was really, really, really impressive for us, and it felt like it opened another door. [00:30:47] S1: Yeah. It feels like... I don't know if you're familiar. I'm sure you are. I can't believe I'm I'm asking if you're familiar. You know much more about this than I do. But there was some experiment in, like, the seventies, I think, called, like, SHRDLU or something like that, where the entire approach was to try and teach AI all the rules of the world programmatically and to say, you know, you know, here is the category of what a bird is. It has these characteristics, and then here's the category of a bat. It has some overlapping character, you know, all of these sorts of things. But but, essentially, know, people are familiar with this concept now because of the way we use LLMs, but it was almost as if the robot suddenly inherited all of these rules and, you know, pieces of knowledge from tying them up with the LLMs that allows the the Taylor Swift demo to happen.