00:08:25 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:08:25] S3: Yeah. Mhmm. [00:08:26] S1: Cool. And I have a little demo here of me doing a dummy test. And the green trajectory on the left hand side will basically be the projection of temples in this frame. And on the right, the arrows means, like, the human orientation. So you can see here, it's basically, by projecting this. You can pretty much see this is like a video game of, like, my hand floating in the air, and there's, like, super accurate track over time of where the hand pose is going because they have, like, really good, close loop slam trajectories. [00:08:58] S3: This really is an incredibly good hand tracking Yeah. System through space. I think that's [00:09:02] S1: why it's offline as well. Yeah. So, like, they really have, like, great optimization for this. Okay. [00:09:08] S0: Yeah. So So Sorry. Sorry. I I... So so just just Yep. Yeah. So, basically, I mean, so I I see all this, like, you know, on I'm seeing on the right, right, you have this red color with a green... Yellow color arrow, and then that's... Can you comment what are these? Like, looks very fancy. Yeah. Sorry. How do you interpret this?
00:09:08 · Chris Paxton, Lawrence Zhu, Pranav
[00:09:08] S0: Yeah. So So Sorry. Sorry. I I... So so just just Yep. Yeah. So, basically, I mean, so I I see all this, like, you know, on I'm seeing on the right, right, you have this red color with a green... Yellow color arrow, and then that's... Can you comment what are these? Like, looks very fancy. Yeah. Sorry. How do you interpret this? [00:09:28] S1: More like a visualization during our debugging. So yellow is basically the in place rotation we have. So I'm gonna mention this later. We try to fit this into our differential drive robot model. And so when it's yellow, it means it's not having linear velocities. It's, like, turning in place. Yeah. And then the red means the current position, and then the green would be just regular pose of, like, a future hat in, for example, my demonstration. Yep. [00:09:56] S2: Mhmm. And another thing to notice is that, like, if you... It's a little bit hard to see in this picture, but, like, yeah, if you look on the right hand side, can actually see the entire, like, slam map too, like, some of the points and, like Yep. Exactly, like, the visualization of, like, the entire room that the ARIA was able to come up with. And so you can see, like, how was able to, like, visualize your position within that. And and then we can use that to then do a retargeting and stuff like that, as Lawrence will talk about.
00:10:20 · Lawrence Zhu, Michael Cho
[00:10:20] S1: So we'll mention that in in a bit. Okay. So I'll... Let's... Cool. So what EgoMimic showed is basically by adding human data as whole training with robot data, they actually achieve much better performance than using robot data alone. This is because of the fact that the observation action pair got really increased. The density of trajectory, if we assume this as a three d faulting trajectories in the unified class space, and that's getting much denser by adding human data. And the key takeaway of EgoMimic here is basically they can show that human data can be treated as a separate embodiment. And when we say embodiment here, for example, the Mobile ALOHA robot could be one embodiment, Power robot could be one embodiment, and for example, an autonomous vehicle could be another embodiment. They all have different degrees of freedom. They perform tasks differently. And human can also be considered as one embodiment. And taking action labeled human data to co train a single policy and that can fit in the both human and robot environment. Any questions regarding KIT? [00:11:23] S3: Sorry. Post prediction here. So you're just masking one out. You'll need to basically back propagate through one of the two joint actions or post prediction. Is that right? [00:11:31] S1: Sorry. Could you repeat that? [00:11:32] S3: The... Sorry. The... So the point here is that you're you're masking out one of the two. Right? You're not... Like, for a particular example.
00:11:32 · Michael Cho, Lawrence Zhu
[00:11:32] S3: The... Sorry. The... So the point here is that you're you're masking out one of the two. Right? You're not... Like, for a particular example. [00:11:38] S1: Yes. So Yeah. For example, the vision encoder is cotrained, and for the action loss, they're, like, trained separately. [00:11:45] S3: Yeah. Yeah. Right. Okay. [00:11:46] S1: Yeah. Just... Cool. [00:11:47] S3: Makes sense.
00:11:48 · Lawrence Zhu
[00:11:48] S1: So, yeah, as... We're going back to the point is that collecting robot demonstration is expensive, and, of course, collecting mobile robot demonstration is more expensive as shown by Pranav and I for collecting data back there. So we came out with the key hypotheses or, like, the key key problem we wanna solve is how can we scale up mobile manipulation with human data, or how can we do ego mimic mobile? So this has been pretty much, this is our project name for a long, long time, Pranav, if you remembered. I think, we changed it to the alpha at the very end. Very basic solution to this would be, co training mobile human data with mobile robot data. So let's try to make similar move. And, the issue is mobile robot data is still super slow to collect. Right? We didn't solve the data bottleneck here. So, what's our hypothesis? Can we basically direct transfer navigation knowledge from human to robot? Because we know that hypothesis... We know that, like, transferring navigation will actually be super difficult as we we have in manipulation because grasping actually between human and robot has a huge gap. But for navigation, you're actually looking at the same scene with the same background. So we can achieve mobile manipulation by co training mobile human data plus static robot data, and the static robot data will be relatively easier to acquire. So the answer is yes. This is me testing, one of the very first results we got. Basically, I was recording me walking around, like, just back and forth between the two tables, and that's, like, zero shot transferred on the robot. And, although we might have different kinematics, we might look a little bit different, but by collecting enough data, we model the robots during inference time would be a strict subset. That is also, one of the key, assumption we make for this work is that it will able to, direct transfer navigation knowledge from human.
00:13:38 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:13:38] S3: Does the head height of the robot matter at all there? Like, does it need to be at exactly the same orientation or position as you reported the [00:13:45] S1: Actually, it needs to be a subset, but it doesn't need to [00:13:47] S3: be Subset. Okay. Yeah. Exactly. How much data did you use for this? [00:13:51] S1: So I think this task is relatively easy. Maybe, like, five, ten minutes, that'll be enough. But, like, for our tasks used in the result, we all control the human data to be one hour long data collection time. [00:14:02] S0: Yeah. So when you say FICO enemies, in in that case, you mean, like, you wear the Aria glasses, you literally walk between the table, the... With your cloth and then to the other table, like, for five minutes. You keep your hand and then... Okay. Got it. Okay. [00:14:15] S3: The robot is not crazy, like crouched walk or something to make sure you had a good distance your head, bothers. [00:14:21] S1: I don't have to torque my head like this. And then... But I I I will say if you Just natural walking. Yeah. Yeah. Just natural walking. If you intentionally, like, look up or look aside, then probably won't direct transfer.