00:18:15 · Mario Gabriele
[00:18:15] S1: world of AI, how we how we thought that the machines should think or how they should learn things versus how they actually learn them. You... I think in 2023, say that you had a moment where for the first time you really felt like you could see a bit of the future of the robotics industry, and things were really clicking into place. But you had really been in that field for quite a while already, it strikes me. So I wondered what the period before that moment was like. Like, was that a quite a lonely time to be working in a space where you don't know how it's gonna play out?
00:18:50 · Karol Hausman
[00:18:50] S0: So maybe I can go back and tell you a little bit more about the journey. I would love that. So you understand that that kind of the feeling. After I moved to Warsaw, as I as I mentioned, it it was very disappointing to see the state of robotics. And it... For a long time, it felt like no one was working on the intelligent robots that I really wanted to work on. And I remember at the time, I was searching for for really anything, anybody who who is doing something that that is a little bit closer to what I imagined robots to be. And for a long time, it felt like no one was. Then I remember I I finally found someone. I found a professor in Germany that was that was writing papers that seemed a little bit closer to what I would imagine robots to be. And there was this this master's program in in Munich called robotics, cognition, and intelligence. And I thought if I if I apply for this program and get in and they don't do intelligent robots, then I'm basically lost. Like, if the program with with that title doesn't do anything that I that I imagined, then something is very, very wrong. So I applied and and moved to Munich. And I remember the first day of classes, I went to one of the the the... One of the very first classes, and I got stuck in a in a metro somewhere and then was super light, so I missed the entire class. And I and I showed up. No one was there. The class was over at that point. But I saw a janitor, and I was looking for a for a bathroom. So I asked him where the bathroom is. He pointed me to the to the 3rd Floor of the building. I remember it very vividly. And as I went upstairs, I remember seeing these big, big doors, and there was some mechanical sound coming from behind them. I don't know why, but I just decided to, like, open them and see what's going on there, that the sound was very intriguing. And as I open these doors, I see these two basically humanoid robots doing... One of them was making popcorn and the other one I believe was spreading butter on a toast. And that was the moment where, you know, this is something I've been searching for at this point for for years. So that was the moment that that was probably one of the most powerful moments I I had during that journey where I finally found it. There's robots that that look intelligent, that do something that is very human like. There are people who work on this.
00:21:09 · Karol Hausman
So I immediately left the room, knocked because there's basically no one there. And I knocked at the at the at the room next door and asked if I could work there. And, basically, I was completely unqualified. Had no idea how to program these robots. I was not familiar with any of the tools, but the person who was there, it was another one of those people that gave me a chance and and asked me to just come in on Monday and see what I could do. Wow. So I got involved, got a job at that at that lab. That's how I got into finally working on intelligent robots. So, yeah, I I think this is like yet another person that I'm really grateful to. And then afterwards, I I moved to The US, got into a PhD program studying studying intelligent robots. And at that point, I was so stoked that, like, I finally get to do this and I finally found a small group of people that work on those things. It didn't matter that, you know, it doesn't work that well. I was so excited that I finally get to, you know, meet the people that are that are working on this. I can learn from them. I can I can push it forward, and and I can finally work on the thing that I always wanted to work on? So I did this throughout my PhD, and I had another moment like this during my PhD, this kind of defining moment where I was working on a certain set of problems where it was referred to as as active perception. And I can I can go a little bit deeper into this if you're interested? But at that point, it kinda felt like there are ways to write... To to to write papers, to to progress in your PhD, but there isn't anything that felt like it could actually solve this problem. It it all felt like the world is too complex. You can push it to some extent, this technology, but you you won't really get to the finish line. They're not they're not kind of adding up. You don't really see a path out of it. And there was another moment where a poster candidate stopped by our lab. And at that time, when a poster candidate candidate stops by, you are supposed as a PhD student, show them your work and kind of talk with them as part of the interview. So I showed to that post that candidate stuff that I was working on, and he listened to all of it. And then at the end of it, he said that I should drop all of this and switch to deep learning. That's the way to to solve the problem that I was actually working on.
00:23:27 · Karol Hausman
At that point, I just thought to myself that he had no idea what he's talking about, and he probably didn't listen to me. Because at that point, I think I spent already two years or something like this working on the topic I was working on. But then later that day, I went to his lecture where he showed what what he was working on. And that was another one of those moments where it was extremely eye opening because for the first time, saw something that, like, could actually work. Where it was less of like a, you know, here's just a little paper over there and a little paper over there, but they don't add up. But it's something that that kind of brought all the pieces together. And, yeah, that was another one of those moments, like the the similar one to that... To the one I've experienced in Munich where at that point, I decided to to drop my PhD topic, change it completely, start collaborating with that person, and do everything I can to push deep learning in robotics. And that person was Sergey Levine, who is now my cofounder at Physical Intelligence and one of the pioneers of of deep learning in in robotics. And, yeah, I I never looked back after that. But to kind of... This is a long way of answering your question, but it didn't feel lonely in that, you know, I was just really excited to be working on that problem.
00:24:46 · Mario Gabriele, Karol Hausman
[00:24:46] S1: That's such good stories there. The first one is almost like, you know, when a wizard is accepted to Hogwarts or something. It's like you finally found your, you know, collection of of magical people and and the magic that they were doing. The the Sergey story is also so interesting to me because you seem to have gone from skeptical to convinced very, very fast. Like, what was the cycle on on that? Was that like literally the same day, the same week? [00:25:13] S0: I think it was within that lecture. Wow. I think like in the middle of that talk, was like, yep. Makes sense. Is 100% it. What... Whatever I was doing is wrong. This is the right way. I want to do everything I can to work on on this topic. And I I I have feelings like this. I think every researcher has has a feeling like this every now and then, where they just see something and it clicks. And at that point, that's a very powerful feeling. You kind of want to drop everything you've been doing, and you just you just feel like you you found something that that is much closer to truth than what you thought before. I think we all had a similar moment afterwards when we're working on combining large language models with with robot learning. And that was, I think, one more of these... Another of these powerful moments where, like, things start to click. [00:26:05] S1: That was the Taylor Swift demo? Is that the one... That's right. Yeah. Maybe you could tell that story because that is a fascinating one.
00:26:11 · Karol Hausman
[00:26:11] S0: Yeah. So what happened afterwards is I started working with Sergey, there was a few of us who were really getting into into that field. It kinda felt like we were a small group of renegades within the robotics community because there were a lot of problems with deep learning at the time. Was. And some of these problems remain. Like, it was not interpretable. There was no way of really making it modular. It seemed like nobody fully understood how it worked. It wasn't very sample efficient. And there were all of these problems. So, like, at that time, it was extremely unpopular to write any deep learning in robotics paper. It And was kinda like between these two worlds of of machine learning world where any robotics paper seemed kind of like the wrong paper for that venue. The robotics world that never fully embraced or didn't embrace deep learning at that time, but it was still really cool to work with a small group of people and and push these methods forward. So then afterwards, the only place that was really embracing that view was Google Brain. So I decided to join Google Brain right out of my PhD, continue working with Sergey and others on those sort of methods with the basic premise being that the way to to really get it to work is to scale it up. So we're scaling it up. We're trying to figure out how to really get it to work at scale. But again, it started feeling like there is so much complexity in the world that if robots need to... The only way for robots to learn all of this complexity is to experience all of it firsthand, it'll be very difficult to scale. It will be very difficult to to have them learn about logic and, you know, how to break down a task if they had to do it all by themselves. Yes. So then I think around 2022, I I would need to look back to to see what year it was exactly, we started combining it together with Brian Ichter, my my other cofounder. We started combining these robotic methods with large language models. This was before the chat GPT moment. This is where people just started experimenting with with large language models and kind of understanding them better and better. There was this one particular demo. So we we were really excited about this because there was one...