00:32:24 · Chris Paxton, Lawrence Zhu, Pranav
[00:32:24] S0: Okay. Okay. [00:32:24] S1: Yep. Cool. So for the handover wine task within the lab environment, so the original one, we performed the scaling, experiment where we basically try, to, collect data from fifteen minutes all the way to, like, sixteen minutes of, like, human and robot data. These are so all of our baseline mobile are co trained on the same set of like 50, static robot demos to ensure like fair comparison and in this case we can see like the scaling effect actually goes like with like a 30% success rate when we actually collide starts to scale human data with sixty minutes of data collection that will be I think maybe 200 of human devils and for robot tally ops that will be like around 60 to like 50 to 60 [00:33:07] S0: but this is purely just manipulation. Right? This is... Doesn't... Handover wine is purely just a... Is it... Or does it include the navigation part too? [00:33:15] S1: It's a mobile task. Yes. [00:33:17] S2: Yeah. So you you basically pick up the wine glass from the table, then you turn around and then drive to the person and then hand over the wine to the person. [00:33:25] S1: Yep. So this reported here is full task success rate. [00:33:28] S0: Mhmm. Okay.
00:33:29 · Lawrence Zhu
[00:33:29] S1: Cool. And we also performed the ablation study of the proposed action retargeting and phase switch. So, of course, without face switch, what we observe is, like, it will start to move at the precise manipulation phase. Like, for example, in grasping that wine glass, it will, like, start turning when it's, like, not even, the gripper's not even, like, cloped close to the wine glass. And for ammo without action retargeting, it usually misses the human because they're actually executing a human trajectory with differential drive base is actually going to lead the robot to the wrong direction, and that's going to miss the human recipient in the COW. So this is our added experiment. It also allows co training with mobile robot data to achieve coupled arm based movement. And we, let's say if we have the key assumption before, was like we split all the demonstration between, like a, from a human full body motion into navigation manipulation task, then that implicitly is like kind of limiting us from performing like coupled arm based movements. And so in that case, we will just collect a very small number of mobile robot data. And by co training this, this is... We can actually achieve coupled base arm movement. This is me collecting data again. We show that it's super painful to collect. And if you can see there, I have to kneel down every single time. It's because there's a kinematic collision. There's actually a collision between my arm and the leader arm over there. So I have to like go down to grab the back of the chair this is a push chair task what we try to show because for this task you have to actively adjust the backrest of the chair when pushing it yeah So what we show here is we achieve this arm based coordination by co training mobile robot data with mobile human data. And for EMMA, can see actually with the added human data, something that really surprises us is the grasp accuracy got improved a lot. So, yeah, it almost like it was a clip right at the back backrest of the chair, for, like, more than half of the time, and then it never fails something like this. Yeah. So this is like
00:35:39 · Michael Cho, Chris Paxton, Lawrence Zhu, Pranav
[00:35:39] S3: This is a failure that Yeah. That... Or... Okay. [00:35:41] S0: But is is this, you know, under your face, like, detector, is this considered manipulation? Like, the whole thing is manipulation? [00:35:48] S1: No. This... Sorry. For this model, we removed the face switch because we don't need it anymore. Right? We can just perform the navigation manipulation simultaneously because we have mobile robot data. [00:35:59] S3: Because it's super interesting that the human data would improve the grasp accuracy because that's actually kind of the exact opposite of what I would expect. [00:36:05] S1: Yeah. [00:36:05] S3: Why is that? [00:36:06] S1: It's... I would say it's more like we added a lot of demonstrations of Okay. When actually... When you actually see the end effector is cotrained. Right? So the end effector of a human hand, like, how how it approaches the backrest of the chair is actually densely demonstrated in our human data. For this task, we only have, like... We're starting from ten minutes of mobile robot data, so that will be around, like, 17 demos, if I remember correctly. [00:36:34] S3: Okay. [00:36:34] S1: And we start adding twenty minutes of human data. That will be around, like, 80 demonstrations per now. Is that, like, good estimation? [00:36:40] S2: Yeah. Yeah. Yeah. [00:36:41] S1: I think [00:36:41] S2: that's So, yeah, essentially all [00:36:43] S1: the data... You go ahead.
00:36:43 · Lawrence Zhu, Pranav, Chris Paxton
[00:36:43] S1: the data... You go ahead. [00:36:44] S2: Yeah. Yeah. Essentially, like, think what you're saying is that, like, basically, it's about the motion. Right? So, like, the way that people do it, like, [00:36:51] S0: you get a lot of [00:36:51] S2: that density in the motion of how we grab onto it. And so that's why it's working better with, like, the grabbing ability.
00:36:58 · Lawrence Zhu
[00:36:58] S1: Yeah. Also, in this case, I feel like when you actually really scale the human data, right, the perspective gap actually doesn't matter because your Right. Inference time becomes a strict subset of, like, your dense demonstration. Yeah. And for this push share task, we also try to evaluate EMMA with separate manipulation and navigation models so basically train two models instead of projecting them into the shared transformer trunk and what we observe here is ing, the first stage we actually called it bimanual engagement is like you have to push the chair with two arms instead of one arm, showing some of the failure cases before, so EMMA is basically a 50 out of 50 so, and for our baseline, RoboTalio baseline with thirty minutes of robot data I think we collected around 50 mobile robot demonstrations and of course like for the minimal comparison like the 17 demo ten minute robot data it just keeps failing and I think we're right about to, I think we're at the very end of our results. I think I want to mention one thing is, in this paper, we actually performed a lot of robot experiments and the rollouts. We actually written that we, so for every single data point here, we can report the arrow bar as 95% clobber Pearson because we actually perform 50 rollout for every single task. So it's like 15 experiments for all the experiments reported in this paper. I think that's... This design twice is basically, like... First thing is we wanna follow some, like, industrial level standards, like, from the Abby... LBM pink paper, large behavior models. Yeah. And we also try to kinda we're also kinda figuring out, like, how noisy real world robot evaluation can be even for a static robotic task. Right? Because there's no way you kinda reset the scene with the exact same setup as the previous evaluation. And for our case, the mobile, the mobile, robot evaluation is actually getting much more difficult because you can't really drive your robot back to the exact same slot. Mhmm. So I think we picked the number 50. It's a painful number for, like, every single data point takes, one hour and a half. But, like, I would say, we're actually able to report this arrow bar and also perform, like, like, two proportional z tests on, like, every single time we perform. Oh, like, we claim that it outperforms speed rather than we just have, like Yeah.
00:39:23 · Lawrence Zhu, Chris Paxton, Michael Cho
20 demos, and then it will be, like, relatively noisy. [00:39:27] S0: Yeah. It makes sense. I guess you you save a lot of time not doing teleoperation, and so you spend more [00:39:31] S1: time doing real time. Time spent. [00:39:34] S3: So, yeah, it makes sense. [00:39:37] S0: I'm just... I was just thinking, like, what if, like, right now, like, this particular push chair task, you are pushing to a chair that is, you know, to to to a to a to a place that is maybe only two, three meters away. I wonder if you... What if you keep pushing the chair and become, like, a longer horizon, like, more like a navigation task. [00:39:55] S1: Yeah. That's, like, a very interesting point. [00:39:58] S0: Yeah. Because you you you emphasize about how important it was that you have this almost 100% phase change. Right? So Mhmm. Like, where does... I mean, the most extreme case I I was thinking about it is, like, I might certainly be out of school of this study. I was just wondering, like, how do we use this for... Because sometimes you are manipulating things while you are navigating. You know? [00:40:17] S1: You know? Know? Exactly. [00:40:17] S0: The most extreme case, I could be playing my Rubik's cube when I'm walking around or, you know, down to school. Used to do that. I think Right.