00:07:08 · Chris Paxton, Lawrence Zhu, Michael Cho
[00:07:08] S0: Yeah. Got it. [00:07:09] S1: So, yeah, I'll move on to how the Aria sensing modalities are used for a static co training system ego mimic as I mentioned before for ego mimics they mount the same aria glasses on the robot and with the human for collecting data we adopt this design as well So the issue here is when human collects data, you're actually moving your head. There's no way you'd fork your head like a robot there when when the robot actually needs a unified reference frame. Okay. So in this case, we need to stabilize the reference frame to use, like, human hand track as actions. Right? So the solution here is Aria provided SLAM. So what we basic... What they basically did was they project series of future hand pose, basically an action chunk, to the corresponding frame head pose so that those points got projected into the world frame. And in that case, if we project this whole chunk in world frame back to my current egocentric device frame, then I'll have a unified reference frame for all the future actions. Yeah. Any questions regarding this? [00:08:14] S0: Yeah. Got it. So so, basically, it's it's important to know... I mean, yeah, the the the camera pose itself is moving, so you you take in time with that. Yeah. Okay. [00:08:25] S3: Yeah. Mhmm.
00:08:25 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:08:25] S3: Yeah. Mhmm. [00:08:26] S1: Cool. And I have a little demo here of me doing a dummy test. And the green trajectory on the left hand side will basically be the projection of temples in this frame. And on the right, the arrows means, like, the human orientation. So you can see here, it's basically, by projecting this. You can pretty much see this is like a video game of, like, my hand floating in the air, and there's, like, super accurate track over time of where the hand pose is going because they have, like, really good, close loop slam trajectories. [00:08:58] S3: This really is an incredibly good hand tracking Yeah. System through space. I think that's [00:09:02] S1: why it's offline as well. Yeah. So, like, they really have, like, great optimization for this. Okay. [00:09:08] S0: Yeah. So So Sorry. Sorry. I I... So so just just Yep. Yeah. So, basically, I mean, so I I see all this, like, you know, on I'm seeing on the right, right, you have this red color with a green... Yellow color arrow, and then that's... Can you comment what are these? Like, looks very fancy. Yeah. Sorry. How do you interpret this?
00:09:08 · Chris Paxton, Lawrence Zhu, Pranav
[00:09:08] S0: Yeah. So So Sorry. Sorry. I I... So so just just Yep. Yeah. So, basically, I mean, so I I see all this, like, you know, on I'm seeing on the right, right, you have this red color with a green... Yellow color arrow, and then that's... Can you comment what are these? Like, looks very fancy. Yeah. Sorry. How do you interpret this? [00:09:28] S1: More like a visualization during our debugging. So yellow is basically the in place rotation we have. So I'm gonna mention this later. We try to fit this into our differential drive robot model. And so when it's yellow, it means it's not having linear velocities. It's, like, turning in place. Yeah. And then the red means the current position, and then the green would be just regular pose of, like, a future hat in, for example, my demonstration. Yep. [00:09:56] S2: Mhmm. And another thing to notice is that, like, if you... It's a little bit hard to see in this picture, but, like, yeah, if you look on the right hand side, can actually see the entire, like, slam map too, like, some of the points and, like Yep. Exactly, like, the visualization of, like, the entire room that the ARIA was able to come up with. And so you can see, like, how was able to, like, visualize your position within that. And and then we can use that to then do a retargeting and stuff like that, as Lawrence will talk about.
00:10:20 · Lawrence Zhu, Michael Cho
[00:10:20] S1: So we'll mention that in in a bit. Okay. So I'll... Let's... Cool. So what EgoMimic showed is basically by adding human data as whole training with robot data, they actually achieve much better performance than using robot data alone. This is because of the fact that the observation action pair got really increased. The density of trajectory, if we assume this as a three d faulting trajectories in the unified class space, and that's getting much denser by adding human data. And the key takeaway of EgoMimic here is basically they can show that human data can be treated as a separate embodiment. And when we say embodiment here, for example, the Mobile ALOHA robot could be one embodiment, Power robot could be one embodiment, and for example, an autonomous vehicle could be another embodiment. They all have different degrees of freedom. They perform tasks differently. And human can also be considered as one embodiment. And taking action labeled human data to co train a single policy and that can fit in the both human and robot environment. Any questions regarding KIT? [00:11:23] S3: Sorry. Post prediction here. So you're just masking one out. You'll need to basically back propagate through one of the two joint actions or post prediction. Is that right? [00:11:31] S1: Sorry. Could you repeat that? [00:11:32] S3: The... Sorry. The... So the point here is that you're you're masking out one of the two. Right? You're not... Like, for a particular example.
00:11:32 · Michael Cho, Lawrence Zhu
[00:11:32] S3: The... Sorry. The... So the point here is that you're you're masking out one of the two. Right? You're not... Like, for a particular example. [00:11:38] S1: Yes. So Yeah. For example, the vision encoder is cotrained, and for the action loss, they're, like, trained separately. [00:11:45] S3: Yeah. Yeah. Right. Okay. [00:11:46] S1: Yeah. Just... Cool. [00:11:47] S3: Makes sense.
00:11:48 · Lawrence Zhu
[00:11:48] S1: So, yeah, as... We're going back to the point is that collecting robot demonstration is expensive, and, of course, collecting mobile robot demonstration is more expensive as shown by Pranav and I for collecting data back there. So we came out with the key hypotheses or, like, the key key problem we wanna solve is how can we scale up mobile manipulation with human data, or how can we do ego mimic mobile? So this has been pretty much, this is our project name for a long, long time, Pranav, if you remembered. I think, we changed it to the alpha at the very end. Very basic solution to this would be, co training mobile human data with mobile robot data. So let's try to make similar move. And, the issue is mobile robot data is still super slow to collect. Right? We didn't solve the data bottleneck here. So, what's our hypothesis? Can we basically direct transfer navigation knowledge from human to robot? Because we know that hypothesis... We know that, like, transferring navigation will actually be super difficult as we we have in manipulation because grasping actually between human and robot has a huge gap. But for navigation, you're actually looking at the same scene with the same background. So we can achieve mobile manipulation by co training mobile human data plus static robot data, and the static robot data will be relatively easier to acquire. So the answer is yes. This is me testing, one of the very first results we got. Basically, I was recording me walking around, like, just back and forth between the two tables, and that's, like, zero shot transferred on the robot. And, although we might have different kinematics, we might look a little bit different, but by collecting enough data, we model the robots during inference time would be a strict subset. That is also, one of the key, assumption we make for this work is that it will able to, direct transfer navigation knowledge from human.