00:21:22 · Lawrence Zhu, Pranav
[00:21:22] S1: Yeah. So I think what we have here is, basically, we try to split our data. Right? Because we are co training the... We only have static robot data that's pretty much with some observation like this. It's, like, super close to the table. So what we want here is basically figuring out what part of the human data we're co training it with the robot data and what part we are actually direct transferring that we can actually leverage that part of data for navigation. So I would say it's more like an engineering choice. We need to keep it binary, but I would say with the similar approach, we'll be able to perform some sort of, like, fine grained subtask segmentation that... If we have the, like, complete map information. [00:22:06] S2: Yeah. Going off of that, I definitely do agree with I do agree with what you're saying, though. Like, when you're approaching a table, you do definitely, like, start to look at, like, where to put stuff on the table and what the setup is and everything. I think, yeah, the reason why we did this is because it's easiest to, like, completely direct transfer human to robot for navigation over, like, manipulation. And then, like, as you come to the table, once you're at the table and you phase switch into manipulation, you then get the entire thing. It's as if, like, the robot is starting new and it just, like, sees a scene and then does manipulation. And so it already has, like, the entire scene. So it's kinda just, like... If you think of, like, a manipulation task, it's, like, a fully, like, clean slate at that point.
00:22:06 · Pranav, Chris Paxton, Lawrence Zhu
[00:22:06] S2: Yeah. Going off of that, I definitely do agree with I do agree with what you're saying, though. Like, when you're approaching a table, you do definitely, like, start to look at, like, where to put stuff on the table and what the setup is and everything. I think, yeah, the reason why we did this is because it's easiest to, like, completely direct transfer human to robot for navigation over, like, manipulation. And then, like, as you come to the table, once you're at the table and you phase switch into manipulation, you then get the entire thing. It's as if, like, the robot is starting new and it just, like, sees a scene and then does manipulation. And so it already has, like, the entire scene. So it's kinda just, like... If you think of, like, a manipulation task, it's, like, a fully, like, clean slate at that point. [00:22:45] S0: And and I assume, like, even when you're navigating, you are, you know, purposely have... Leaving your hands, like, up in the frame. It's just so that you you can teach a robot what to do with his own hand. So Okay. Okay. Got it. [00:22:56] S1: In this case... Yeah. We'll just assume it's, like, dense demoed. Yep. [00:23:02] S0: Cool. I see. So this is more more because of the limitation of the hardware because you're using smart glasses that is only pointing kinda in front. You you can't see the whole body. Yep. Presumably, if there's, you know, if if somehow you you dangle another GoPro that's facing the whole... Your whole body, you could have leave your hands on the side. And
00:23:02 · Chris Paxton, Lawrence Zhu, Pranav, Michael Cho
[00:23:02] S0: Cool. I see. So this is more more because of the limitation of the hardware because you're using smart glasses that is only pointing kinda in front. You you can't see the whole body. Yep. Presumably, if there's, you know, if if somehow you you dangle another GoPro that's facing the whole... Your whole body, you could have leave your hands on the side. And [00:23:21] S1: Yeah. Yeah. Yeah. Okay. It's more like a FOV thing. Yep. [00:23:25] S0: Got it. I think one other [00:23:27] S2: thing though is that a lot of our tasks are relying on, like, carrying stuff. So, like, as yours... You see here, like, if you're holding a plate and stuff, usually, you don't, like, leave it dangling next to you. Like, hold it up in front of you. [00:23:37] S0: Got it. [00:23:37] S2: So some of the tasks do actually work out with that too. [00:23:40] S0: Alright. Cool. [00:23:41] S3: How how sensitive is it? How, like, how precise do you actually need to be when it comes to doing this classification?
00:23:41 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:23:41] S3: How how sensitive is it? How, like, how precise do you actually need to be when it comes to doing this classification? [00:23:47] S1: Exactly. It's actually super per... Precise with this, like, unsupervised clustering method. We report here is the main accuracy over frames. It's not in the current version of paper, but when this podcast is out, we'll update the version for this revision. So Okay. What we have here is, I think, at least the... So we have four tasks that requires the phase switch. The first one is the continuous table service with two variants. We... If you... If we notice that, like, we have the utensil, and then we have the kuisong. And we also have handover wine and handover wine under a new scene, and we have grocery shopping, which is a bimanual task. And all of them worked really well in this case. Yep. And we don't need to... We we don't actually need any sort of, like, hyperparameter fine tuning. We use the same set of parameter across all those tasks. Cool. So we basically saw first challenge. We kinda structurized human data into like navigation manipulation, [00:24:40] S0: and we have
00:24:41 · Lawrence Zhu
[00:24:41] S1: a near 100% accurate face identifier that allows us to throw out all the head movements in the manipulation phase, we can just zero them out. And to come to our second engineering challenge is like, there's a kinematic gap between the human and robots when they move. And so humans move omnidirectionally. Sometimes you have like fluticural movements and then like, sometimes you have sharp turns, but the robot must follow a differential drive model in our case. So for... To overcome this, we formulated this as like a optimization problem. So given a sequence of like my future wave points, which is still an action chunk of my future head pose projected into the ground plane, What what we actually did is basically we just assumed there's a virtual differential drive robot dragging in my optimization task. So in this case, we optimized a set of command velocities. And then until we have this differential drive dynamics that we have added weights on position and yaw and smoothness. And until this converges, we'll actually be able to see on the top, a little bit smaller figure. So on the left is the raw human trajectory. You can see a literal motion there. And for the retargeted trajectory, you actually perform a backward movements and then start go forward because we guarantee that by this optimization, it's 100% differential drives executable. So I have a video share with the same example. So you can see the Ximen trajectory actually is super noisy, and you can see here from the retargeting trajectory. Yep. It's like nice and smooth. So here comes our architecture design for the co training method. So it is built on top of a work called heterogeneous pre trained transformer and design choices borrowed from EgoMimic, our previous work. So given a, so what HPT does is it allows you to modular... Allows you to like add modules of perception and vision stems, which are those encoders. I kind of missed my mouse. So what you're able to do is you're able to kind of have several stamps that encode these visual features or proper perception features into a fixed number of tokens, in our case 16. And they're all projected into a shared latent space with the transformer trunk. And then we decode them from the same transformer trunk between our human robot and some shared supervision.
00:27:05 · Lawrence Zhu
So the legend here of color here actually means which data is supervised certain has or stems. And on the right is our system, which we adopt pretty much the same one from EgoMimic, and then we mount it on the UGB differential drive local base. Yep. And, Shere Khan's, pretty much the complete pipeline of, EMMA, Egocentric Mobile Manipulation, is basically we... We're allowing the co training of mobile robot data with static robot... Sorry, mobile human data with, static robot data to perform a robot mobile manipulation. So we evaluate EMMA on, three challenging robot tasks in distribution, and we've recently added the fourth one. So, this is our first task as demonstrated before, the table service task, which you basically switch in between direct transferring from robot and co training from robot and human. And we also compare with one hour data collected. Mobile tally operation is much harder to scale, so we collected, I think, 31 or 33 that was a Mobile ALOHA we trained with the same hpd trunk and then we can perform that actually has a higher success rate our second task is a human robot interaction task So the robot is required to pick up a wine glass from a random... Randomly positioned on table map and then navigate towards a human randomly positioned in the corner of the room and then perform a handover. So this is basically a task that requires density to be super high, and that's when the scaling effect actually comes in.