Going Direct Conversations
00:23:17 · Jordan Wolfe
[00:23:17] S2: And so Chris, when you think about what Mike just said, which is obviously capturing video in the diversity of environments is actually a very easy thing to do, obviously, with all the devices that we have today. Like as a roboticist, what needs, If you were to start collecting a bunch of this data, what would need to be built on top of it for actually to be useful for someone like you to actually train the robots that you're working on? So like Michael was saying, one of the things that you need to do is kind of the robot needs to be able to understand the actions that are being performed, right?
00:23:51 · Chris Paxton
[00:23:51] S0: So what that means is a lot of these methods that you'll see, and we've talked about a couple, we've had a bunch of these on our podcast, right? There's a few ways that you can do it. One of them is like, actually like predicting like this is where my hand this is where the hand is in the video and like but obviously that's super limiting because most videos that we take like on youtube don't have like clear videos clear pictures of your hands and so on so like ideally things get less structured and you can kind of throw more data into the mixture but then you have to figure out how to actually train on all kinds of heterogeneous data and that is different missing field it's a complicated and open research area in my opinion honestly A lot of the things that you see companies doing, like Tesla, for example, is just like they actually seem to have just like a setup where you can very clearly see what the robot is doing, right? Which is not, sorry, not what the robot's doing, what the human's doing. Both, right? Because it needs to know what it's doing to be able to learn off of it. But that's obviously not, that wouldn't be the end goal if you wanted to, if you really want to train off all of YouTube or something. We don't have that yet. That's not something that we're, that currently people are doing. They're useful. So it's like a different kind of video, I think, than you're usually thinking of. And this is a real limitation.
00:25:11 · Jordan Wolfe, Chris Paxton
[00:25:11] S2: So when you say a different type of video, are you talking about like, are you talking about, let's use an example, right? So let's use the example of like bartending, for instance, right? So are you talking about like a dedicated video to like slicing a lime or preparing a cocktail and like having very intentional data to like... [00:25:34] S0: Usually, yeah, that's right. But also, usually, depending on how you're doing it a lot of the time, that means that you need to be able to estimate where the hands are in the video, which means that it's not just a video of a bartender doing a lime. It's like a video from the bartender's point of view of their hands slicing a lime or whatever. So that means that because to be able to know what the robot should do, I need to know where this finger is a lot of the time. Otherwise, because that's what the robot's going to have to do itself later. So a lot of learning from human video stuff ends up getting very complicated like this. Now, of course, that's not all of it. There's quite a lot of different ways that people have learned from human video. But yeah.
00:26:12 · Michael Cho
[00:26:12] S1: Yeah, I should also just add actually video alone is not going to get us there also to human performance because like you can imagine if you take an egocentric video, there's a lot of occlusion. Let's say when I pick up a box, you cannot see the rest of my big fingers because the box is covering the fingers, but I'm actually touching the box, right? So we humans take it for granted. We have very, very good touch sensors all over our body, right? Especially the fingers, dips and all that. So we actually are organizing this robotic origami competition. We announced it a couple of weeks ago. And we actually are co-organizing that competition together with some folks from the Japan Origami Association. So these are like the real human experts doing origami. So we actually invited like a grandmaster of the origami association to this robotic conference. And then next to that, the old man doing origami is is a robot doing origami and the robot absolutely like it's very impressive but nothing compared to the old man but what i want to highlight though is i was like standing in front of this like grandmaster for like 10 minutes just observing what he's doing half the time he's talking to people around him he's not even looking it's all touch Yeah. Yeah. Like, you know, making his paper crane on that. So, so I think, so, so there's also a very big hardware thing that we, we just don't have good tough senses, right? Anywhere near as good as ours. Sure. In terms of like, you know, like fatality at the cost that it's, that makes sense. So that's also another one. And I think you're understanding a little bit how good the robot part was.
00:27:46 · Chris Paxton, Jordan Wolfe
[00:27:46] S0: The robot guy, I walked by this booth, obviously, at this conference with lots of top researchers. And a lot of them were saying, this is one of the coolest robot manipulation things I've seen. And he's just getting blown away by the old man next door. Right. So like, that's just like, I want to just be clear. It was actually very good. The robot part was very good. It was just, yeah. [00:28:08] S2: Yeah. Yeah. We take it for granted how like, how unbelievable our hands are, right? Like just the dexterity and like what you can manipulate and how you can manipulate. It's pretty, it's pretty remarkable. All right, let's go to kind of one last one, which is sim to reel or reel to sim. So Chris, maybe kind of break that down for us, what that means and kind of your perspective on the level of use, like usability in training and training.
00:28:40 · Chris Paxton
[00:28:40] S0: Yeah, so this is actually something that I worked on for a very long time, like six plus years, I think. But so robotics. So yeah, SynthReal means that we train the robot how to do something in a simulation. And then it also has to work in the real world, right? So I take the robot, and maybe I create a digital twin of the task that I want it to perform. If it's an industrial task or an industrial kidding task, maybe I would just spawn all the objects that I need to place on a table and have a copy of the robot and have it sit and have and then maybe I create a whole bunch like a million of these robots in simulation I have them all practice picking and placing the object and into into the target and whatever that right and I use that data like I simulate cameras or whatever sensors that I may have access to and then I train a policy, train a neural network on that simulated data. And the sim-to-real part, that's the sim part. The sim-to-real part is then I have to set that up in such a way that I can then take that policy and use it in the real world. So this is the way that a lot of things work right now, particularly when it comes to all the dancing robot videos that you see are sim-to-real. all of them are trained this way. But there's a lot of other stuff like central manipulation is another, is an active research area, so.