00:41:39 · Lawrence Zhu, Chris Paxton
[00:41:39] S1: Exactly. Yeah. [00:41:40] S0: Yeah. There's a little bit of navigation, however small that is. I actually do do want to... If you can cycle back a few slides, you you have all these, like Yeah. Charts about the different success rate. Yep. I noticed they they do have maybe different... Yeah. Before this. Mhmm. I think all the different tasks that you have, you were comparing... Yeah. It's it's an earlier slide. [00:42:09] S1: The big one. Right? [00:42:10] S0: Yeah. This one. This one. Yeah. Yep. Could you comment a little bit more, like, what... Like like, why is grocery shopping, you know, like, you know, seems pretty good and then, you know, stuff utensils. Like, I... I'm curious, like, what's your intuition about, like, the relative, like, success rate between these?
00:42:10 · Chris Paxton, Lawrence Zhu, Pranav
[00:42:10] S0: Yeah. This one. This one. Yeah. Yep. Could you comment a little bit more, like, what... Like like, why is grocery shopping, you know, like, you know, seems pretty good and then, you know, stuff utensils. Like, I... I'm curious, like, what's your intuition about, like, the relative, like, success rate between these? [00:42:26] S1: That's really... Yeah. So I think for the for the grocery shopping task of course I think it really benefits from the face identification because I would say you can see the bimanual grasp at the beginning right so it's of course first benefiting from the co training from human and robot data because we have like much denser trajectory of like how you should grasp that, I think it's orange juice or it's a chip bag at the same time, but at the same time, I think having a stationary base also helps with the performance here. Because with the Mobile Robo teleaudit with the mobile based on Mobile ALOHA, sometimes I have like unintended base drift that goes back and while the gripper is still approaching the chips. Yeah. And for the first two tasks, serve utensils and serve quasum, they are actually trained with the same model, as I mentioned before. So we expect them to have a little bit lower success rate because they might confuse in between the two demonstrations. But I think one of the key highlight we should notify here is, like, the performance drop actually come from navigation because it's a super long horizon navigation path. I think the robot travels around seven meters per now. [00:43:41] S2: If Yeah. Seven meters.
00:43:41 · Pranav, Lawrence Zhu, Chris Paxton, Michael Cho
[00:43:41] S2: If Yeah. Seven meters. [00:43:42] S1: I think we measured. Yeah. So, like, even though we have all those, like, retargeting, we have the, dense human demonstration, I would say this embodiment gap is still, like, leading our robot to somehow, like, drift away from what we've been demonstrated. And also because there's, like, I I would say it's mostly because of the perspective gap. So for long horizon, navigation, we probably need much more data. The... We control this to be like one hour of data collection, and I think that's why the performance drop come in the navigation phase. [00:44:13] S2: Yep. Another one that's a little bit funny to our specific setup is that, as you saw, like, our room is, like, very, like, lab. Like, it's, like, a very lab environment and doesn't have too many features. So, like, as you go around this this trajectory, a lot of the features can be very similar, which we try to add, like, little, like, things on the floor, like the TV and stuff like that so that, like, it can start to really see the path from observation. But, ultimately [00:44:38] S1: Obviously, it works better in a more human environment. Exactly. [00:44:41] S0: Yossi's literally show, like, a green green room. [00:44:43] S3: Yeah. Exactly. Exactly. I I actually wanted to ask, like, is that actually a green screen? Like, is that why that's that color? They built this to that big background. For...
00:44:43 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:44:43] S3: Yeah. Exactly. Exactly. I I actually wanted to ask, like, is that actually a green screen? Like, is that why that's that color? They built this to that big background. For... [00:44:52] S1: I think they built this room for a sim to rail transfer Yeah. Like, two, three years ago. So they built this room as similar to sim as possible. Yeah. They Yeah. Hopefully, we're repainting it by the end of the month. Okay. [00:45:05] S3: That's good. Yeah. There were definitely some moments when you were, like, picking up the Yeah. Exactly. Yeah. First time when I was like, hang on. Is this the sim... [00:45:12] S0: This looks... [00:45:13] S3: Everything looks a little worse. [00:45:14] S0: Making the navigation any any easier. Like, it's like, know, everywhere you look the same. Okay. Okay. But, yeah, I'm... I I think if you can circle back... Yeah. So I I think it's interesting what you've mentioned, because it's... It seems to me that just out the task, you know, the the charts that you have... So what you're saying is that the grocery shopping and [00:45:33] S3: But that actually is the same. Sorry. [00:45:34] S0: Yeah. The... It it seems that the the the navigation task never dropped the success rate, but it's also be... You're saying is primarily because perhaps the navigation is really short short horizon. [00:45:45] S1: Yeah. For the the one... Yep. Sorry. For, you mean for the navigation part, for the, this one?
00:45:45 · Lawrence Zhu, Chris Paxton
[00:45:45] S1: Yeah. For the the one... Yep. Sorry. For, you mean for the navigation part, for the, this one? [00:45:53] S0: Yeah. It's it's... No. Yeah. So if you see the charts, right? So what was curious to me was that, for example, handover wine, grocery shopping, like when you go to navigation, literally the success rate didn't drop. That means navigation was basically a 100%. Right? Yeah. So... Perfect. But then, you know, in the case of surf utensils... So that's why I was curious, like, why when you... Like, that navigation suck and then the other one. But but I think what your explanation, which is surf utensils, it's actually a pretty long horizon navigation task. So Yeah. And that's why it's kind of failing still. [00:46:21] S1: And I think for both handover wine and grocery shopping, we basically... At the start of navigation, I will say the... There are... Of course, as Pranav said, we have dense visual feature that helps navigation. But I think another another, like, justification is that the goal was relatively thin at the early stage of navigation. Unlike, like, servo utensil and servo quasong, you have, like, it's pretty much a 135 degree turn followed by a 90 degree turn. So you don't really see your goal points. The... During your policy inference, you have to, like, figuring out what's the correct future viewpoints from, like, very limited visual features. [00:47:02] S0: Right. And and I suppose if this environment becomes a lot more dynamic, do you think the navigation will start to suffer?
00:47:02 · Chris Paxton, Lawrence Zhu, Pranav
[00:47:02] S0: Right. And and I suppose if this environment becomes a lot more dynamic, do you think the navigation will start to suffer? [00:47:13] S1: I think it depends on whether it is included in the demonstration. Right? Because, yeah, we're basically a behavior cloning pipeline. And we actually tried in the handoverwind task when the human recipient actually moves around. It... It's just gonna keep following until, like, it reaches out to the distance that's close enough to human that, like, demo... That that is demoed in the human data that you should perform handover one. It actually generalized really well for the handover one task. I think it just extracts the feature of, like, a human shape and then some sort of a lag in the screen, and then it just learns the general pattern to just go to the human, I think. [00:47:56] S2: Yeah. It was fun because we once had, like, a couple of people who were, like, visiting our lab, and they were interested in, like, seeing a demonstration. So we're like, oh, we'll do, like, handover wine. And so there's, like, a seven year old kid who was just standing there, and then, like, we're like, oh, is it gonna generalize to him? And I was actually able to hand over the wine to him. Like [00:48:12] S1: Yeah. Because, yeah, the fish sandwich on the mug. [00:48:17] S0: Yeah. Too too too too young to drink. Yeah. Yeah. [00:48:21] S1: We were a little bit scared because, apparently, like, the visual looked different.