Going Direct Conversations
00:18:08 · Jordan Wolfe, Chris Paxton
[00:18:08] S2: Yeah. Yeah. What's your thought, Chris, on the teleoperation side? [00:18:13] S0: I mean, I think that, yeah, like you're saying, that's the gold standard, getting real robot data. And I think the gold standard is actually probably, the best data is probably the data the robot collects itself. It's probably like reinforcement learning kind of data. But that's pretty rare still. But the robots are... Teleoperation data is great as long as you can actually get it, right? But while scaling, it's hard. And so that was actually the next question was kind of reinforcement learning. [00:18:45] S2: So Chris, can you kind of explain that for folks that might not be so understanding of what that actually means in the term of robotics space?
00:18:53 · Chris Paxton
[00:18:53] S0: Yeah, sure. Yeah, so I apparently even wrote a book chapter on this for what it's worth. So reinforcement learning is... What's the best way to describe this? I really should have a better answer for you off the top of my head. But it's basically the idea that the robot needs to be able to teach itself from its own experiences. So the way that you or I learn to perform a task is not by watching. I don't watch... If I wanted to learn how to... hit a tennis ball. I think that's like to serve a tennis ball or something like this. Like I wouldn't do that by just watching a bunch of YouTube videos. I would do that by tossing a tennis ball and trying to hit it, right? And then do practicing for days or weeks or months or years depending on how good I wanted to get, right? And that's what reinforcement learning is in general. It is saying that the robot is going to just like sit there and take in information about the world and it's going to And it's going to make a decision and try something, and if it fails, it's going to take that and think about it and do better next time, hopefully. Or if it succeeds, it's going to be like, okay, that was good. Let's try again a little bit. Let's try to do more like that. And the reason that I'm saying it's the best is because it's the same reason that it's best for you or me. If you want to learn to do something, You'll have your own embodiment and your own version, your own thoughts about how to do that, and you're never going to really get that from watching someone. But you'll do pretty good from watching someone. You'll learn a lot from watching someone, but you still at some point always need to do it yourself.
00:20:27 · Jordan Wolfe, Chris Paxton
[00:20:27] S2: Right? And so that's a good segue into kind of another aspect, which is kind of one I really wanted to better understand, which is learning from kind of human video. So the concept of can you collect, you know, you collect data, you label it, you provide contacts on and on and on. You have your own data schema. So Mike, what's kind of your perspective on, because of just what Chris said around like you actually have to do the thing to learn the thing. At some point, at some point. [00:20:55] S0: You can get pretty far without it, right? Let's dig into that because this is like a very abstract thing for folks that [00:21:01] S2: aren't in the robotic space. So how useful is kind of human video for training robots, Mike? I mean, do you think, is that a critical piece right now? And will it be a critical piece going forward? Or is it going to be the robot physically doing like physically watching someone and not video.
00:21:19 · Michael Cho
[00:21:19] S1: Yeah, I think it's a critical piece, but you, it can't be the only piece. I think, like I said just now, there are three types of data. You don't need all three types is my view. You need to co-train on all three types. Video being one of them. Video is great because it's so easy to collect video. We already have a lot of it on YouTube, let's say. You can also quite easily just collect ecosystem video, that sort of stuff. The issue, well, I mean, the great thing about video is because it's so easy to collect, you get a lot of diversity of environments and tasks. And you want the diversity, right, to so-called hopefully allow the model to see more things and then generalize better ultimately. But I also think that there's a pretty low threshold to what, like the performance of a model that you only train from video. Because these videos doesn't come with action labels. Unlike, let's say, teleoperation, it's hard to think about it. So you always have to use some techniques to infer actions. But those actions are always, you know, like approximation. Not only that, because every embodiment is different. Imagine you have an ecosystem video of some humans doing something. The thing is, all of us have hands, but all of our hands look similar, but they're actually also distinctly different. And I tend to think that in robotics, all the minor detail matters a lot. Millimeters, you're talking about 50 milliseconds latency. could be the difference between you falling down and standing up, you know? And so when you actually apply in the real physical, because the real physical world, like physics is just brutal, like gravity is brutal. Like if you just miss by that little bit, it's the difference between you're able to do it and not doing it. And so I think video is great, again, to give you that whole very broad concept, okay, what is playing tennis, what is playing basketball, things like that, right? But if you actually want to play basketball, I think you can't just watch basketball. Yeah, something, yeah.
00:23:17 · Jordan Wolfe
[00:23:17] S2: And so Chris, when you think about what Mike just said, which is obviously capturing video in the diversity of environments is actually a very easy thing to do, obviously, with all the devices that we have today. Like as a roboticist, what needs, If you were to start collecting a bunch of this data, what would need to be built on top of it for actually to be useful for someone like you to actually train the robots that you're working on? So like Michael was saying, one of the things that you need to do is kind of the robot needs to be able to understand the actions that are being performed, right?
00:23:51 · Chris Paxton
[00:23:51] S0: So what that means is a lot of these methods that you'll see, and we've talked about a couple, we've had a bunch of these on our podcast, right? There's a few ways that you can do it. One of them is like, actually like predicting like this is where my hand this is where the hand is in the video and like but obviously that's super limiting because most videos that we take like on youtube don't have like clear videos clear pictures of your hands and so on so like ideally things get less structured and you can kind of throw more data into the mixture but then you have to figure out how to actually train on all kinds of heterogeneous data and that is different missing field it's a complicated and open research area in my opinion honestly A lot of the things that you see companies doing, like Tesla, for example, is just like they actually seem to have just like a setup where you can very clearly see what the robot is doing, right? Which is not, sorry, not what the robot's doing, what the human's doing. Both, right? Because it needs to know what it's doing to be able to learn off of it. But that's obviously not, that wouldn't be the end goal if you wanted to, if you really want to train off all of YouTube or something. We don't have that yet. That's not something that we're, that currently people are doing. They're useful. So it's like a different kind of video, I think, than you're usually thinking of. And this is a real limitation.