00:30:17 · Michael Cho, Lawrence Zhu, Pranav, Chris Paxton
[00:30:17] S3: Yeah. And this is just one hour of data. So you can map [00:30:19] S1: Oh, half an hour. Half an hour. Hour. Yep. Alright. Cool. And for scene generalization, we actually compared with the... Our baseline was not collected in the new scene. So we're basically trying to show in this case is the first grasping. Right? So we take the wine glass. That's a subtask. It's actually failing without human data, so that's something we wanna test in this case. [00:30:41] S2: Yeah. There's a huge observation difference in that situation. So, like, when you have robot only data and stuff, it doesn't see this new scene, and so it has no idea what to do. It constantly missed the wine glass. It constantly wasn't able to actually grab it, unlike EMMA, which was actually able to grab it and then actually navigate to the person and hand it over, which was pretty impressive. [00:31:02] S1: So for manipulation alone, it's, like, not solvable by, like, direct transferring to a new scene. Even the only thing that changed is the background. So... Yep. And... [00:31:12] S0: But I'm I'm trying to get an intuition why there's such a stark difference. Because what you're saying is that just by having Emma, which is the additional... This... Another embodiment Mhmm. Effectively, because the human body is in other embodiment. But just in the first, very first time, which is purely manipulation because you haven't navigate yet. Yeah. And yet it general... It's it's generalized so well to a different scene. I'm I'm trying to get... Yeah.
00:31:12 · Chris Paxton, Michael Cho, Lawrence Zhu
[00:31:12] S0: But I'm I'm trying to get an intuition why there's such a stark difference. Because what you're saying is that just by having Emma, which is the additional... This... Another embodiment Mhmm. Effectively, because the human body is in other embodiment. But just in the first, very first time, which is purely manipulation because you haven't navigate yet. Yeah. And yet it general... It's it's generalized so well to a different scene. I'm I'm trying to get... Yeah. [00:31:34] S3: There's thirty minutes of new human data in the new scene, but there's no new robot data in the new scene. [00:31:39] S1: Oh, I see. [00:31:40] S0: I see. See. Yeah. Oh, wow. [00:31:46] S1: Sorry. So in this case, the second and third subtask for the baseline actually doesn't matter that much because there's no data. But for our, we'll basically show for the first subtask without the human data collected, it's actually not able to, like, direct transfer even with some sort of perception overfit. So we kinda ablate this out. Yep. [00:32:04] S0: I see. So it's not like an unseen environment. Basically, it's it's seen. [00:32:08] S1: Yeah. It's transparent. [00:32:09] S0: But it's very easily seen. It's very easily seen because it's human doing it. Half an hour is done. Okay. Got it. Yep. Okay. So this [00:32:14] S3: can So, like, that's... So it's so it's a... It's an argument that, I guess, you can scale it easily. Not that Yeah. Yeah. Not that you don't need to scale it, I guess. Is that right? [00:32:24] S0: Okay. Okay.
00:32:24 · Chris Paxton, Lawrence Zhu, Pranav
[00:32:24] S0: Okay. Okay. [00:32:24] S1: Yep. Cool. So for the handover wine task within the lab environment, so the original one, we performed the scaling, experiment where we basically try, to, collect data from fifteen minutes all the way to, like, sixteen minutes of, like, human and robot data. These are so all of our baseline mobile are co trained on the same set of like 50, static robot demos to ensure like fair comparison and in this case we can see like the scaling effect actually goes like with like a 30% success rate when we actually collide starts to scale human data with sixty minutes of data collection that will be I think maybe 200 of human devils and for robot tally ops that will be like around 60 to like 50 to 60 [00:33:07] S0: but this is purely just manipulation. Right? This is... Doesn't... Handover wine is purely just a... Is it... Or does it include the navigation part too? [00:33:15] S1: It's a mobile task. Yes. [00:33:17] S2: Yeah. So you you basically pick up the wine glass from the table, then you turn around and then drive to the person and then hand over the wine to the person. [00:33:25] S1: Yep. So this reported here is full task success rate. [00:33:28] S0: Mhmm. Okay.
00:33:29 · Lawrence Zhu
[00:33:29] S1: Cool. And we also performed the ablation study of the proposed action retargeting and phase switch. So, of course, without face switch, what we observe is, like, it will start to move at the precise manipulation phase. Like, for example, in grasping that wine glass, it will, like, start turning when it's, like, not even, the gripper's not even, like, cloped close to the wine glass. And for ammo without action retargeting, it usually misses the human because they're actually executing a human trajectory with differential drive base is actually going to lead the robot to the wrong direction, and that's going to miss the human recipient in the COW. So this is our added experiment. It also allows co training with mobile robot data to achieve coupled arm based movement. And we, let's say if we have the key assumption before, was like we split all the demonstration between, like a, from a human full body motion into navigation manipulation task, then that implicitly is like kind of limiting us from performing like coupled arm based movements. And so in that case, we will just collect a very small number of mobile robot data. And by co training this, this is... We can actually achieve coupled base arm movement. This is me collecting data again. We show that it's super painful to collect. And if you can see there, I have to kneel down every single time. It's because there's a kinematic collision. There's actually a collision between my arm and the leader arm over there. So I have to like go down to grab the back of the chair this is a push chair task what we try to show because for this task you have to actively adjust the backrest of the chair when pushing it yeah So what we show here is we achieve this arm based coordination by co training mobile robot data with mobile human data. And for EMMA, can see actually with the added human data, something that really surprises us is the grasp accuracy got improved a lot. So, yeah, it almost like it was a clip right at the back backrest of the chair, for, like, more than half of the time, and then it never fails something like this. Yeah. So this is like
00:35:39 · Michael Cho, Chris Paxton, Lawrence Zhu, Pranav
[00:35:39] S3: This is a failure that Yeah. That... Or... Okay. [00:35:41] S0: But is is this, you know, under your face, like, detector, is this considered manipulation? Like, the whole thing is manipulation? [00:35:48] S1: No. This... Sorry. For this model, we removed the face switch because we don't need it anymore. Right? We can just perform the navigation manipulation simultaneously because we have mobile robot data. [00:35:59] S3: Because it's super interesting that the human data would improve the grasp accuracy because that's actually kind of the exact opposite of what I would expect. [00:36:05] S1: Yeah. [00:36:05] S3: Why is that? [00:36:06] S1: It's... I would say it's more like we added a lot of demonstrations of Okay. When actually... When you actually see the end effector is cotrained. Right? So the end effector of a human hand, like, how how it approaches the backrest of the chair is actually densely demonstrated in our human data. For this task, we only have, like... We're starting from ten minutes of mobile robot data, so that will be around, like, 17 demos, if I remember correctly. [00:36:34] S3: Okay. [00:36:34] S1: And we start adding twenty minutes of human data. That will be around, like, 80 demonstrations per now. Is that, like, good estimation? [00:36:40] S2: Yeah. Yeah. Yeah. [00:36:41] S1: I think [00:36:41] S2: that's So, yeah, essentially all [00:36:43] S1: the data... You go ahead.
00:36:43 · Lawrence Zhu, Pranav, Chris Paxton
[00:36:43] S1: the data... You go ahead. [00:36:44] S2: Yeah. Yeah. Essentially, like, think what you're saying is that, like, basically, it's about the motion. Right? So, like, the way that people do it, like, [00:36:51] S0: you get a lot of [00:36:51] S2: that density in the motion of how we grab onto it. And so that's why it's working better with, like, the grabbing ability.