00:31:12 · Chris Paxton, Michael Cho, Lawrence Zhu
[00:31:12] S0: But I'm I'm trying to get an intuition why there's such a stark difference. Because what you're saying is that just by having Emma, which is the additional... This... Another embodiment Mhmm. Effectively, because the human body is in other embodiment. But just in the first, very first time, which is purely manipulation because you haven't navigate yet. Yeah. And yet it general... It's it's generalized so well to a different scene. I'm I'm trying to get... Yeah. [00:31:34] S3: There's thirty minutes of new human data in the new scene, but there's no new robot data in the new scene. [00:31:39] S1: Oh, I see. [00:31:40] S0: I see. See. Yeah. Oh, wow. [00:31:46] S1: Sorry. So in this case, the second and third subtask for the baseline actually doesn't matter that much because there's no data. But for our, we'll basically show for the first subtask without the human data collected, it's actually not able to, like, direct transfer even with some sort of perception overfit. So we kinda ablate this out. Yep. [00:32:04] S0: I see. So it's not like an unseen environment. Basically, it's it's seen. [00:32:08] S1: Yeah. It's transparent. [00:32:09] S0: But it's very easily seen. It's very easily seen because it's human doing it. Half an hour is done. Okay. Got it. Yep. Okay. So this [00:32:14] S3: can So, like, that's... So it's so it's a... It's an argument that, I guess, you can scale it easily. Not that Yeah. Yeah. Not that you don't need to scale it, I guess. Is that right? [00:32:24] S0: Okay. Okay.
00:32:24 · Chris Paxton, Lawrence Zhu, Pranav
[00:32:24] S0: Okay. Okay. [00:32:24] S1: Yep. Cool. So for the handover wine task within the lab environment, so the original one, we performed the scaling, experiment where we basically try, to, collect data from fifteen minutes all the way to, like, sixteen minutes of, like, human and robot data. These are so all of our baseline mobile are co trained on the same set of like 50, static robot demos to ensure like fair comparison and in this case we can see like the scaling effect actually goes like with like a 30% success rate when we actually collide starts to scale human data with sixty minutes of data collection that will be I think maybe 200 of human devils and for robot tally ops that will be like around 60 to like 50 to 60 [00:33:07] S0: but this is purely just manipulation. Right? This is... Doesn't... Handover wine is purely just a... Is it... Or does it include the navigation part too? [00:33:15] S1: It's a mobile task. Yes. [00:33:17] S2: Yeah. So you you basically pick up the wine glass from the table, then you turn around and then drive to the person and then hand over the wine to the person. [00:33:25] S1: Yep. So this reported here is full task success rate. [00:33:28] S0: Mhmm. Okay.
00:33:29 · Lawrence Zhu
[00:33:29] S1: Cool. And we also performed the ablation study of the proposed action retargeting and phase switch. So, of course, without face switch, what we observe is, like, it will start to move at the precise manipulation phase. Like, for example, in grasping that wine glass, it will, like, start turning when it's, like, not even, the gripper's not even, like, cloped close to the wine glass. And for ammo without action retargeting, it usually misses the human because they're actually executing a human trajectory with differential drive base is actually going to lead the robot to the wrong direction, and that's going to miss the human recipient in the COW. So this is our added experiment. It also allows co training with mobile robot data to achieve coupled arm based movement. And we, let's say if we have the key assumption before, was like we split all the demonstration between, like a, from a human full body motion into navigation manipulation task, then that implicitly is like kind of limiting us from performing like coupled arm based movements. And so in that case, we will just collect a very small number of mobile robot data. And by co training this, this is... We can actually achieve coupled base arm movement. This is me collecting data again. We show that it's super painful to collect. And if you can see there, I have to kneel down every single time. It's because there's a kinematic collision. There's actually a collision between my arm and the leader arm over there. So I have to like go down to grab the back of the chair this is a push chair task what we try to show because for this task you have to actively adjust the backrest of the chair when pushing it yeah So what we show here is we achieve this arm based coordination by co training mobile robot data with mobile human data. And for EMMA, can see actually with the added human data, something that really surprises us is the grasp accuracy got improved a lot. So, yeah, it almost like it was a clip right at the back backrest of the chair, for, like, more than half of the time, and then it never fails something like this. Yeah. So this is like
00:35:39 · Michael Cho, Chris Paxton, Lawrence Zhu, Pranav
[00:35:39] S3: This is a failure that Yeah. That... Or... Okay. [00:35:41] S0: But is is this, you know, under your face, like, detector, is this considered manipulation? Like, the whole thing is manipulation? [00:35:48] S1: No. This... Sorry. For this model, we removed the face switch because we don't need it anymore. Right? We can just perform the navigation manipulation simultaneously because we have mobile robot data. [00:35:59] S3: Because it's super interesting that the human data would improve the grasp accuracy because that's actually kind of the exact opposite of what I would expect. [00:36:05] S1: Yeah. [00:36:05] S3: Why is that? [00:36:06] S1: It's... I would say it's more like we added a lot of demonstrations of Okay. When actually... When you actually see the end effector is cotrained. Right? So the end effector of a human hand, like, how how it approaches the backrest of the chair is actually densely demonstrated in our human data. For this task, we only have, like... We're starting from ten minutes of mobile robot data, so that will be around, like, 17 demos, if I remember correctly. [00:36:34] S3: Okay. [00:36:34] S1: And we start adding twenty minutes of human data. That will be around, like, 80 demonstrations per now. Is that, like, good estimation? [00:36:40] S2: Yeah. Yeah. Yeah. [00:36:41] S1: I think [00:36:41] S2: that's So, yeah, essentially all [00:36:43] S1: the data... You go ahead.
00:36:43 · Lawrence Zhu, Pranav, Chris Paxton
[00:36:43] S1: the data... You go ahead. [00:36:44] S2: Yeah. Yeah. Essentially, like, think what you're saying is that, like, basically, it's about the motion. Right? So, like, the way that people do it, like, [00:36:51] S0: you get a lot of [00:36:51] S2: that density in the motion of how we grab onto it. And so that's why it's working better with, like, the grabbing ability.
00:36:58 · Lawrence Zhu
[00:36:58] S1: Yeah. Also, in this case, I feel like when you actually really scale the human data, right, the perspective gap actually doesn't matter because your Right. Inference time becomes a strict subset of, like, your dense demonstration. Yeah. And for this push share task, we also try to evaluate EMMA with separate manipulation and navigation models so basically train two models instead of projecting them into the shared transformer trunk and what we observe here is ing, the first stage we actually called it bimanual engagement is like you have to push the chair with two arms instead of one arm, showing some of the failure cases before, so EMMA is basically a 50 out of 50 so, and for our baseline, RoboTalio baseline with thirty minutes of robot data I think we collected around 50 mobile robot demonstrations and of course like for the minimal comparison like the 17 demo ten minute robot data it just keeps failing and I think we're right about to, I think we're at the very end of our results. I think I want to mention one thing is, in this paper, we actually performed a lot of robot experiments and the rollouts. We actually written that we, so for every single data point here, we can report the arrow bar as 95% clobber Pearson because we actually perform 50 rollout for every single task. So it's like 15 experiments for all the experiments reported in this paper. I think that's... This design twice is basically, like... First thing is we wanna follow some, like, industrial level standards, like, from the Abby... LBM pink paper, large behavior models. Yeah. And we also try to kinda we're also kinda figuring out, like, how noisy real world robot evaluation can be even for a static robotic task. Right? Because there's no way you kinda reset the scene with the exact same setup as the previous evaluation. And for our case, the mobile, the mobile, robot evaluation is actually getting much more difficult because you can't really drive your robot back to the exact same slot. Mhmm. So I think we picked the number 50. It's a painful number for, like, every single data point takes, one hour and a half. But, like, I would say, we're actually able to report this arrow bar and also perform, like, like, two proportional z tests on, like, every single time we perform. Oh, like, we claim that it outperforms speed rather than we just have, like Yeah.