00:05:52 · Chris Paxton, Michael Cho, Lawrence Zhu
[00:05:52] S0: So You guys have a... I Yeah. [00:05:53] S3: I didn't notice that they actually sent it up to the meta server every time that you wanna do... Does it... Can you actually do it real time, or is this all, like Oh. Because it has to be post processed? Yeah. [00:06:02] S1: For the the first generation, they don't have anything, I think, real time. But for the second gen, they're gonna release, I think. They released it already. They're gonna have... It shipped to us, I think Okay. Like, later this year. That one has Yeah. [00:06:13] S0: Very [00:06:13] S1: cool. I think, real time depth estimation. In that case, we are actually... Yeah. Actually able to not slam. Yeah. Okay. [00:06:20] S3: Or not to close loop slam. Because, okay. Yeah. No. That makes sense.
00:06:20 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:06:20] S3: Or not to close loop slam. Because, okay. Yeah. No. That makes sense. [00:06:22] S1: Yeah. If we wanna extend a little bit here, I I watched your episode about EgoZero. That episode, they talk about MASt3R-SLAM. We tried exactly the same pipeline. So the reason we're not... Because, I think we... When we tried to design this project in the beginning, it's like we have some sort of co training of, like, robot data and human data, and we just have some random SLAM or, like, planning algorithm to help us to navigate from one point to another, and then we perform another co training manipulation task over there. But we realized, I think, for master SLAM, accuracy was okay, but the only issue is I think it runs, seven FPS on a desktop, forty ninety. That means we're using a laptop, it would be slower. And for a policy real time inference, I would say that's not enough for us. Okay. That makes sense. [00:07:08] S0: Yeah. Got it.
00:07:08 · Chris Paxton, Lawrence Zhu, Michael Cho
[00:07:08] S0: Yeah. Got it. [00:07:09] S1: So, yeah, I'll move on to how the Aria sensing modalities are used for a static co training system ego mimic as I mentioned before for ego mimics they mount the same aria glasses on the robot and with the human for collecting data we adopt this design as well So the issue here is when human collects data, you're actually moving your head. There's no way you'd fork your head like a robot there when when the robot actually needs a unified reference frame. Okay. So in this case, we need to stabilize the reference frame to use, like, human hand track as actions. Right? So the solution here is Aria provided SLAM. So what we basic... What they basically did was they project series of future hand pose, basically an action chunk, to the corresponding frame head pose so that those points got projected into the world frame. And in that case, if we project this whole chunk in world frame back to my current egocentric device frame, then I'll have a unified reference frame for all the future actions. Yeah. Any questions regarding this? [00:08:14] S0: Yeah. Got it. So so, basically, it's it's important to know... I mean, yeah, the the the camera pose itself is moving, so you you take in time with that. Yeah. Okay. [00:08:25] S3: Yeah. Mhmm.
00:08:25 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:08:25] S3: Yeah. Mhmm. [00:08:26] S1: Cool. And I have a little demo here of me doing a dummy test. And the green trajectory on the left hand side will basically be the projection of temples in this frame. And on the right, the arrows means, like, the human orientation. So you can see here, it's basically, by projecting this. You can pretty much see this is like a video game of, like, my hand floating in the air, and there's, like, super accurate track over time of where the hand pose is going because they have, like, really good, close loop slam trajectories. [00:08:58] S3: This really is an incredibly good hand tracking Yeah. System through space. I think that's [00:09:02] S1: why it's offline as well. Yeah. So, like, they really have, like, great optimization for this. Okay. [00:09:08] S0: Yeah. So So Sorry. Sorry. I I... So so just just Yep. Yeah. So, basically, I mean, so I I see all this, like, you know, on I'm seeing on the right, right, you have this red color with a green... Yellow color arrow, and then that's... Can you comment what are these? Like, looks very fancy. Yeah. Sorry. How do you interpret this?
00:09:08 · Chris Paxton, Lawrence Zhu, Pranav
[00:09:08] S0: Yeah. So So Sorry. Sorry. I I... So so just just Yep. Yeah. So, basically, I mean, so I I see all this, like, you know, on I'm seeing on the right, right, you have this red color with a green... Yellow color arrow, and then that's... Can you comment what are these? Like, looks very fancy. Yeah. Sorry. How do you interpret this? [00:09:28] S1: More like a visualization during our debugging. So yellow is basically the in place rotation we have. So I'm gonna mention this later. We try to fit this into our differential drive robot model. And so when it's yellow, it means it's not having linear velocities. It's, like, turning in place. Yeah. And then the red means the current position, and then the green would be just regular pose of, like, a future hat in, for example, my demonstration. Yep. [00:09:56] S2: Mhmm. And another thing to notice is that, like, if you... It's a little bit hard to see in this picture, but, like, yeah, if you look on the right hand side, can actually see the entire, like, slam map too, like, some of the points and, like Yep. Exactly, like, the visualization of, like, the entire room that the ARIA was able to come up with. And so you can see, like, how was able to, like, visualize your position within that. And and then we can use that to then do a retargeting and stuff like that, as Lawrence will talk about.
00:10:20 · Lawrence Zhu, Michael Cho
[00:10:20] S1: So we'll mention that in in a bit. Okay. So I'll... Let's... Cool. So what EgoMimic showed is basically by adding human data as whole training with robot data, they actually achieve much better performance than using robot data alone. This is because of the fact that the observation action pair got really increased. The density of trajectory, if we assume this as a three d faulting trajectories in the unified class space, and that's getting much denser by adding human data. And the key takeaway of EgoMimic here is basically they can show that human data can be treated as a separate embodiment. And when we say embodiment here, for example, the Mobile ALOHA robot could be one embodiment, Power robot could be one embodiment, and for example, an autonomous vehicle could be another embodiment. They all have different degrees of freedom. They perform tasks differently. And human can also be considered as one embodiment. And taking action labeled human data to co train a single policy and that can fit in the both human and robot environment. Any questions regarding KIT? [00:11:23] S3: Sorry. Post prediction here. So you're just masking one out. You'll need to basically back propagate through one of the two joint actions or post prediction. Is that right? [00:11:31] S1: Sorry. Could you repeat that? [00:11:32] S3: The... Sorry. The... So the point here is that you're you're masking out one of the two. Right? You're not... Like, for a particular example.