00:27:05 · Lawrence Zhu
So the legend here of color here actually means which data is supervised certain has or stems. And on the right is our system, which we adopt pretty much the same one from EgoMimic, and then we mount it on the UGB differential drive local base. Yep. And, Shere Khan's, pretty much the complete pipeline of, EMMA, Egocentric Mobile Manipulation, is basically we... We're allowing the co training of mobile robot data with static robot... Sorry, mobile human data with, static robot data to perform a robot mobile manipulation. So we evaluate EMMA on, three challenging robot tasks in distribution, and we've recently added the fourth one. So, this is our first task as demonstrated before, the table service task, which you basically switch in between direct transferring from robot and co training from robot and human. And we also compare with one hour data collected. Mobile tally operation is much harder to scale, so we collected, I think, 31 or 33 that was a Mobile ALOHA we trained with the same hpd trunk and then we can perform that actually has a higher success rate our second task is a human robot interaction task So the robot is required to pick up a wine glass from a random... Randomly positioned on table map and then navigate towards a human randomly positioned in the corner of the room and then perform a handover. So this is basically a task that requires density to be super high, and that's when the scaling effect actually comes in.
00:28:37 · Chris Paxton
[00:28:37] S0: Sorry. So, Lawrence, I ask for the previous task? So for the human data part, do you literally have a human, like, picking up the glass and passing to another human? Is is that the human part?
00:28:47 · Lawrence Zhu, Michael Cho
[00:28:47] S1: Okay. That's how we collect data. We're gonna show that a bit later. Yep. Okay. Actually, kinda interesting. Yeah. So Yep. And our third task is a bimanual grocery shopping task. This task really beats the baseline hard. And what we have is like... Okay. So you have, like, two arm coordination and then bimanual pickup of the shopping bag, and now it's switched to the navigation mode transferred from human? Yep. Then place it for handover. Cool. And our in domain performance is shown here. And for the first two models, they actually trained with one checkpoint. So that's a combined model for two tasks. And then the only difference is whether the utensil is placed on that table. If we see the utensil there, then we perform the serve utensil task. If the utensil is gone, then we perform the qua song task, so basically pure obliteration conditioning. And for the other two tasks, we are significantly outperforming our baseline, Mobile ALOHA trained with a one hour data collected within an equivalent amount of time. So here comes the demo I mentioned before. So basically, when we try to deploy our robot to a new scene, we actually don't need to collect robot data, which means, like, you have to push your robot all the way to that new scene. So we can co train human full body data at the new scene with the static robot data in the original environment, and we actually achieve this scene generalization with a really good success rate. [00:30:17] S3: Yeah. And this is just one hour of data. So you can map
00:30:17 · Michael Cho, Lawrence Zhu, Pranav, Chris Paxton
[00:30:17] S3: Yeah. And this is just one hour of data. So you can map [00:30:19] S1: Oh, half an hour. Half an hour. Hour. Yep. Alright. Cool. And for scene generalization, we actually compared with the... Our baseline was not collected in the new scene. So we're basically trying to show in this case is the first grasping. Right? So we take the wine glass. That's a subtask. It's actually failing without human data, so that's something we wanna test in this case. [00:30:41] S2: Yeah. There's a huge observation difference in that situation. So, like, when you have robot only data and stuff, it doesn't see this new scene, and so it has no idea what to do. It constantly missed the wine glass. It constantly wasn't able to actually grab it, unlike EMMA, which was actually able to grab it and then actually navigate to the person and hand it over, which was pretty impressive. [00:31:02] S1: So for manipulation alone, it's, like, not solvable by, like, direct transferring to a new scene. Even the only thing that changed is the background. So... Yep. And... [00:31:12] S0: But I'm I'm trying to get an intuition why there's such a stark difference. Because what you're saying is that just by having Emma, which is the additional... This... Another embodiment Mhmm. Effectively, because the human body is in other embodiment. But just in the first, very first time, which is purely manipulation because you haven't navigate yet. Yeah. And yet it general... It's it's generalized so well to a different scene. I'm I'm trying to get... Yeah.
00:31:12 · Chris Paxton, Michael Cho, Lawrence Zhu
[00:31:12] S0: But I'm I'm trying to get an intuition why there's such a stark difference. Because what you're saying is that just by having Emma, which is the additional... This... Another embodiment Mhmm. Effectively, because the human body is in other embodiment. But just in the first, very first time, which is purely manipulation because you haven't navigate yet. Yeah. And yet it general... It's it's generalized so well to a different scene. I'm I'm trying to get... Yeah. [00:31:34] S3: There's thirty minutes of new human data in the new scene, but there's no new robot data in the new scene. [00:31:39] S1: Oh, I see. [00:31:40] S0: I see. See. Yeah. Oh, wow. [00:31:46] S1: Sorry. So in this case, the second and third subtask for the baseline actually doesn't matter that much because there's no data. But for our, we'll basically show for the first subtask without the human data collected, it's actually not able to, like, direct transfer even with some sort of perception overfit. So we kinda ablate this out. Yep. [00:32:04] S0: I see. So it's not like an unseen environment. Basically, it's it's seen. [00:32:08] S1: Yeah. It's transparent. [00:32:09] S0: But it's very easily seen. It's very easily seen because it's human doing it. Half an hour is done. Okay. Got it. Yep. Okay. So this [00:32:14] S3: can So, like, that's... So it's so it's a... It's an argument that, I guess, you can scale it easily. Not that Yeah. Yeah. Not that you don't need to scale it, I guess. Is that right? [00:32:24] S0: Okay. Okay.
00:32:24 · Chris Paxton, Lawrence Zhu, Pranav
[00:32:24] S0: Okay. Okay. [00:32:24] S1: Yep. Cool. So for the handover wine task within the lab environment, so the original one, we performed the scaling, experiment where we basically try, to, collect data from fifteen minutes all the way to, like, sixteen minutes of, like, human and robot data. These are so all of our baseline mobile are co trained on the same set of like 50, static robot demos to ensure like fair comparison and in this case we can see like the scaling effect actually goes like with like a 30% success rate when we actually collide starts to scale human data with sixty minutes of data collection that will be I think maybe 200 of human devils and for robot tally ops that will be like around 60 to like 50 to 60 [00:33:07] S0: but this is purely just manipulation. Right? This is... Doesn't... Handover wine is purely just a... Is it... Or does it include the navigation part too? [00:33:15] S1: It's a mobile task. Yes. [00:33:17] S2: Yeah. So you you basically pick up the wine glass from the table, then you turn around and then drive to the person and then hand over the wine to the person. [00:33:25] S1: Yep. So this reported here is full task success rate. [00:33:28] S0: Mhmm. Okay.