00:10:20 · Lawrence Zhu, Michael Cho
[00:10:20] S1: So we'll mention that in in a bit. Okay. So I'll... Let's... Cool. So what EgoMimic showed is basically by adding human data as whole training with robot data, they actually achieve much better performance than using robot data alone. This is because of the fact that the observation action pair got really increased. The density of trajectory, if we assume this as a three d faulting trajectories in the unified class space, and that's getting much denser by adding human data. And the key takeaway of EgoMimic here is basically they can show that human data can be treated as a separate embodiment. And when we say embodiment here, for example, the Mobile ALOHA robot could be one embodiment, Power robot could be one embodiment, and for example, an autonomous vehicle could be another embodiment. They all have different degrees of freedom. They perform tasks differently. And human can also be considered as one embodiment. And taking action labeled human data to co train a single policy and that can fit in the both human and robot environment. Any questions regarding KIT? [00:11:23] S3: Sorry. Post prediction here. So you're just masking one out. You'll need to basically back propagate through one of the two joint actions or post prediction. Is that right? [00:11:31] S1: Sorry. Could you repeat that? [00:11:32] S3: The... Sorry. The... So the point here is that you're you're masking out one of the two. Right? You're not... Like, for a particular example.
00:11:32 · Michael Cho, Lawrence Zhu
[00:11:32] S3: The... Sorry. The... So the point here is that you're you're masking out one of the two. Right? You're not... Like, for a particular example. [00:11:38] S1: Yes. So Yeah. For example, the vision encoder is cotrained, and for the action loss, they're, like, trained separately. [00:11:45] S3: Yeah. Yeah. Right. Okay. [00:11:46] S1: Yeah. Just... Cool. [00:11:47] S3: Makes sense.
00:11:48 · Lawrence Zhu
[00:11:48] S1: So, yeah, as... We're going back to the point is that collecting robot demonstration is expensive, and, of course, collecting mobile robot demonstration is more expensive as shown by Pranav and I for collecting data back there. So we came out with the key hypotheses or, like, the key key problem we wanna solve is how can we scale up mobile manipulation with human data, or how can we do ego mimic mobile? So this has been pretty much, this is our project name for a long, long time, Pranav, if you remembered. I think, we changed it to the alpha at the very end. Very basic solution to this would be, co training mobile human data with mobile robot data. So let's try to make similar move. And, the issue is mobile robot data is still super slow to collect. Right? We didn't solve the data bottleneck here. So, what's our hypothesis? Can we basically direct transfer navigation knowledge from human to robot? Because we know that hypothesis... We know that, like, transferring navigation will actually be super difficult as we we have in manipulation because grasping actually between human and robot has a huge gap. But for navigation, you're actually looking at the same scene with the same background. So we can achieve mobile manipulation by co training mobile human data plus static robot data, and the static robot data will be relatively easier to acquire. So the answer is yes. This is me testing, one of the very first results we got. Basically, I was recording me walking around, like, just back and forth between the two tables, and that's, like, zero shot transferred on the robot. And, although we might have different kinematics, we might look a little bit different, but by collecting enough data, we model the robots during inference time would be a strict subset. That is also, one of the key, assumption we make for this work is that it will able to, direct transfer navigation knowledge from human.
00:13:38 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:13:38] S3: Does the head height of the robot matter at all there? Like, does it need to be at exactly the same orientation or position as you reported the [00:13:45] S1: Actually, it needs to be a subset, but it doesn't need to [00:13:47] S3: be Subset. Okay. Yeah. Exactly. How much data did you use for this? [00:13:51] S1: So I think this task is relatively easy. Maybe, like, five, ten minutes, that'll be enough. But, like, for our tasks used in the result, we all control the human data to be one hour long data collection time. [00:14:02] S0: Yeah. So when you say FICO enemies, in in that case, you mean, like, you wear the Aria glasses, you literally walk between the table, the... With your cloth and then to the other table, like, for five minutes. You keep your hand and then... Okay. Got it. Okay. [00:14:15] S3: The robot is not crazy, like crouched walk or something to make sure you had a good distance your head, bothers. [00:14:21] S1: I don't have to torque my head like this. And then... But I I I will say if you Just natural walking. Yeah. Yeah. Just natural walking. If you intentionally, like, look up or look aside, then probably won't direct transfer.
00:14:21 · Lawrence Zhu, Pranav, Chris Paxton
[00:14:21] S1: I don't have to torque my head like this. And then... But I I I will say if you Just natural walking. Yeah. Yeah. Just natural walking. If you intentionally, like, look up or look aside, then probably won't direct transfer. [00:14:31] S2: Yeah. Yeah. That's what I was gonna say. Like, one thing that's important is that, like, you have to, like, walk, like, the path and be, like, very careful with that because, like, there is retargeting and stuff that Lawrence will talk about later, but, like, it is... Like, the path you go is the path that the robot will follow. One thing is that, like, the height of the robot, like, it is still very robust, but, like, we do notice that, like, we do have to, like, sometimes, like, try to match the visual viewpoint of the robot. It doesn't have to be exact, but it sometimes needs to be, like, close enough as as what [00:14:57] S1: I saying. Like a subset. [00:14:59] S2: Yes. A subset. [00:15:00] S0: Cool.
00:15:01 · Lawrence Zhu
[00:15:01] S1: Oh, yeah. I'll skip that slide. The key assumption here is, if we can't direct transfer navigation, then all we need to co train is mobile, robot data and static human data. So, I'm gonna make an example here is how do you put an elephant into the fridge. I'm making the assumption here, for example, when you have to, like, attach the hand on the fridge door, you, move your buddy backward to open the door, and then take the giraffe fox, for example, etcetera, etcetera. But this example is really just to show that I... We believe most of the human activities can be split into sequences of navigation and manipulation. And, that's basically our, key hypothesis here. And then we like our key assumption for our co training, this set of data. So we started data collection. This is one of our flagship tasks called table service. So we pretty much just serve croissant a baking oven, and then we collect static global data of picking up the croissant and serving it on the other table. So it's 15 demo for each variant. And then we start to actually try to train a model and then roll out this task. We realized there are challenges. So the first task is the human full body data is unstructured. So you don't know where exactly is the boundary between the manipulation and navigation phase. Right? So we need some sort of auxiliary prediction that splits the two. And there's always a kinematic gap between the human and robot because human move omnidirectionally you might have like sharp turns you might have lateral movements where our differential drive robot base is unable to kind of replay that so let's say we try first we try to resolve the first problem. So the first attempt we have is, like, we look back to what resources we got from Project Aria glasses. So we see, okay, we got amp pose and we got offline slam, we at least have some sort of localization or, like, we have some sort of map info during data processing. We don't have that during real time, but we have this in our data. And let's say if we can solve a two shot classification problem by just have some sort of a model to slur human data into navigation and manipulation phases. So because we are all CS majors, so let's put it in a neural network.