00:43:41 · Pranav, Lawrence Zhu, Chris Paxton, Michael Cho
[00:43:41] S2: If Yeah. Seven meters. [00:43:42] S1: I think we measured. Yeah. So, like, even though we have all those, like, retargeting, we have the, dense human demonstration, I would say this embodiment gap is still, like, leading our robot to somehow, like, drift away from what we've been demonstrated. And also because there's, like, I I would say it's mostly because of the perspective gap. So for long horizon, navigation, we probably need much more data. The... We control this to be like one hour of data collection, and I think that's why the performance drop come in the navigation phase. [00:44:13] S2: Yep. Another one that's a little bit funny to our specific setup is that, as you saw, like, our room is, like, very, like, lab. Like, it's, like, a very lab environment and doesn't have too many features. So, like, as you go around this this trajectory, a lot of the features can be very similar, which we try to add, like, little, like, things on the floor, like the TV and stuff like that so that, like, it can start to really see the path from observation. But, ultimately [00:44:38] S1: Obviously, it works better in a more human environment. Exactly. [00:44:41] S0: Yossi's literally show, like, a green green room. [00:44:43] S3: Yeah. Exactly. Exactly. I I actually wanted to ask, like, is that actually a green screen? Like, is that why that's that color? They built this to that big background. For...
00:44:43 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:44:43] S3: Yeah. Exactly. Exactly. I I actually wanted to ask, like, is that actually a green screen? Like, is that why that's that color? They built this to that big background. For... [00:44:52] S1: I think they built this room for a sim to rail transfer Yeah. Like, two, three years ago. So they built this room as similar to sim as possible. Yeah. They Yeah. Hopefully, we're repainting it by the end of the month. Okay. [00:45:05] S3: That's good. Yeah. There were definitely some moments when you were, like, picking up the Yeah. Exactly. Yeah. First time when I was like, hang on. Is this the sim... [00:45:12] S0: This looks... [00:45:13] S3: Everything looks a little worse. [00:45:14] S0: Making the navigation any any easier. Like, it's like, know, everywhere you look the same. Okay. Okay. But, yeah, I'm... I I think if you can circle back... Yeah. So I I think it's interesting what you've mentioned, because it's... It seems to me that just out the task, you know, the the charts that you have... So what you're saying is that the grocery shopping and [00:45:33] S3: But that actually is the same. Sorry. [00:45:34] S0: Yeah. The... It it seems that the the the navigation task never dropped the success rate, but it's also be... You're saying is primarily because perhaps the navigation is really short short horizon. [00:45:45] S1: Yeah. For the the one... Yep. Sorry. For, you mean for the navigation part, for the, this one?
00:45:45 · Lawrence Zhu, Chris Paxton
[00:45:45] S1: Yeah. For the the one... Yep. Sorry. For, you mean for the navigation part, for the, this one? [00:45:53] S0: Yeah. It's it's... No. Yeah. So if you see the charts, right? So what was curious to me was that, for example, handover wine, grocery shopping, like when you go to navigation, literally the success rate didn't drop. That means navigation was basically a 100%. Right? Yeah. So... Perfect. But then, you know, in the case of surf utensils... So that's why I was curious, like, why when you... Like, that navigation suck and then the other one. But but I think what your explanation, which is surf utensils, it's actually a pretty long horizon navigation task. So Yeah. And that's why it's kind of failing still. [00:46:21] S1: And I think for both handover wine and grocery shopping, we basically... At the start of navigation, I will say the... There are... Of course, as Pranav said, we have dense visual feature that helps navigation. But I think another another, like, justification is that the goal was relatively thin at the early stage of navigation. Unlike, like, servo utensil and servo quasong, you have, like, it's pretty much a 135 degree turn followed by a 90 degree turn. So you don't really see your goal points. The... During your policy inference, you have to, like, figuring out what's the correct future viewpoints from, like, very limited visual features. [00:47:02] S0: Right. And and I suppose if this environment becomes a lot more dynamic, do you think the navigation will start to suffer?
00:47:02 · Chris Paxton, Lawrence Zhu, Pranav
[00:47:02] S0: Right. And and I suppose if this environment becomes a lot more dynamic, do you think the navigation will start to suffer? [00:47:13] S1: I think it depends on whether it is included in the demonstration. Right? Because, yeah, we're basically a behavior cloning pipeline. And we actually tried in the handoverwind task when the human recipient actually moves around. It... It's just gonna keep following until, like, it reaches out to the distance that's close enough to human that, like, demo... That that is demoed in the human data that you should perform handover one. It actually generalized really well for the handover one task. I think it just extracts the feature of, like, a human shape and then some sort of a lag in the screen, and then it just learns the general pattern to just go to the human, I think. [00:47:56] S2: Yeah. It was fun because we once had, like, a couple of people who were, like, visiting our lab, and they were interested in, like, seeing a demonstration. So we're like, oh, we'll do, like, handover wine. And so there's, like, a seven year old kid who was just standing there, and then, like, we're like, oh, is it gonna generalize to him? And I was actually able to hand over the wine to him. Like [00:48:12] S1: Yeah. Because, yeah, the fish sandwich on the mug. [00:48:17] S0: Yeah. Too too too too young to drink. Yeah. Yeah. [00:48:21] S1: We were a little bit scared because, apparently, like, the visual looked different.
00:48:21 · Lawrence Zhu, Chris Paxton
[00:48:21] S1: We were a little bit scared because, apparently, like, the visual looked different. [00:48:27] S0: Yeah. That's fascinating. I mean, I'm I'm curiously interested in in... Because navigation is is a very different thing from manipulation. And especially real world, like, in the wild navigation, like, I think the tough part about it is, like, you have, you know, autonomous agents, humans, you know, usually also walking around. So [00:48:46] S1: Exactly. How [00:48:47] S0: they move around you. It's it's a... So yeah but I would say certainly though like what is is also very convincing here is just in the manipulation part of this pipeline like it definitely having the human data just just just just augment by default a lot more. I I... It seems across all the tasks, if it's just manipulation before it starts navigating, it's it's all have a have a significant jump. Yeah. Yeah. Yeah. So how do you guys think, like, long term having, like, this, like, hopefully, cheapest smart glasses or whatever, eco centric video data collection things... Devices will will help improve the performance of of this? Yeah. [00:49:30] S1: Yeah. Pranav, do you wanna Yeah. Talk about
00:49:30 · Lawrence Zhu, Pranav
[00:49:30] S1: Yeah. Pranav, do you wanna Yeah. Talk about [00:49:33] S2: I think as we have, like, smart glasses, like, they become more popular, everyone starts wearing them and everything. They're starting to, like, walk around with them. I think it'll be very... It'll be much more easy to scale up data. Right? Because right now, if you wanna have a robot policy doing something, you go to the robot, collect tons and tons of data, and train a model, and then deploy on the robot. But then if you have all of this data readily available, then it's much more easy to then train a policy on a robot and then transfer it from human to robot and then have your robot deploy in a setting that maybe humans are already in. [00:50:13] S1: Yeah. I would say the real... Like, the true value of these sort of, like, wearable data collection devices is, like, in our three d space, right, it's basically shared in between the human and robot. So basically the scene representation of whatever you're looking at is actually there's no difference so what this work is showing is we put everything into like a two d image so there will be some sort of appearance gap some sort of perspective gap. But I believe in the future, when the second generation of these glasses came out, when we have like depth destination and when we have like three d vision, then that is pretty much... There's no domain gap between the human observation and the robot observation, ideally. So I would say that will pretty much help us to scale robot learning a lot.