00:04:12 · Chris Paxton, Lawrence Zhu
[00:04:12] S0: So, like, you know, hand pose, offline slam, and even the eye glaze. Right? So in in your work, like, which of these are... Would you say is absolutely critical? And also to what extent are they critical? Like, for example, the slam, how how accurate do the slam have to be? Is it like, you know, Apple Vision Pro, you know, you walk away 100 meter, you come back. Is this... You know, like... Yeah. Yeah. What is useful enough so that you can do this mobile manipulation task? [00:04:39] S1: Yeah. So... Yeah. Thanks. I think we are pretty much the only works that, like, we actively leverage this SLAM for mobile tasks because I can't actually move on to the next slide. I'm gonna talk about how EgoMimic used it to solve a static problem. But in this case, in all the sensor modalities they provide, we... They they really have, like, super good closed loop trajectory because, I think they have this on a closed loop or... Sorry, closed closed source algorithms. So what we do is we collect a VRS package with human demonstration. We upload to the meta server, they process it. They come... They get it back to us. So the closed loop trajectory is really accurate, and I believe that is super crucial for... Not only for us, but also for, like, I think, static manipulation tasks because I'm gonna mention it in a bit, is they require a unified reference system that makes a hard hand... Sorry. Hand pose super critical in this case. I don't know if you wanna add.
00:04:39 · Lawrence Zhu, Pranav, Chris Paxton
[00:04:39] S1: Yeah. So... Yeah. Thanks. I think we are pretty much the only works that, like, we actively leverage this SLAM for mobile tasks because I can't actually move on to the next slide. I'm gonna talk about how EgoMimic used it to solve a static problem. But in this case, in all the sensor modalities they provide, we... They they really have, like, super good closed loop trajectory because, I think they have this on a closed loop or... Sorry, closed closed source algorithms. So what we do is we collect a VRS package with human demonstration. We upload to the meta server, they process it. They come... They get it back to us. So the closed loop trajectory is really accurate, and I believe that is super crucial for... Not only for us, but also for, like, I think, static manipulation tasks because I'm gonna mention it in a bit, is they require a unified reference system that makes a hard hand... Sorry. Hand pose super critical in this case. I don't know if you wanna add. [00:05:32] S2: Yeah. Going off of that, I think it depends a lot on the project. So, like, as Lawrence was saying, like, if you're focusing on static manipulation, hand pose can be very important. But, like, we're trying to extend that to mobile manipulation. Right? And so we want to have other things... Other tools that we need to use, like SLAM and Point Cloud. And so Lawrence is gonna talk about that in a little bit too. [00:05:51] S1: Yep. Sounds good. That's... [00:05:52] S0: So You guys have a... I Yeah.
00:05:52 · Chris Paxton, Michael Cho, Lawrence Zhu
[00:05:52] S0: So You guys have a... I Yeah. [00:05:53] S3: I didn't notice that they actually sent it up to the meta server every time that you wanna do... Does it... Can you actually do it real time, or is this all, like Oh. Because it has to be post processed? Yeah. [00:06:02] S1: For the the first generation, they don't have anything, I think, real time. But for the second gen, they're gonna release, I think. They released it already. They're gonna have... It shipped to us, I think Okay. Like, later this year. That one has Yeah. [00:06:13] S0: Very [00:06:13] S1: cool. I think, real time depth estimation. In that case, we are actually... Yeah. Actually able to not slam. Yeah. Okay. [00:06:20] S3: Or not to close loop slam. Because, okay. Yeah. No. That makes sense.
00:06:20 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:06:20] S3: Or not to close loop slam. Because, okay. Yeah. No. That makes sense. [00:06:22] S1: Yeah. If we wanna extend a little bit here, I I watched your episode about EgoZero. That episode, they talk about MASt3R-SLAM. We tried exactly the same pipeline. So the reason we're not... Because, I think we... When we tried to design this project in the beginning, it's like we have some sort of co training of, like, robot data and human data, and we just have some random SLAM or, like, planning algorithm to help us to navigate from one point to another, and then we perform another co training manipulation task over there. But we realized, I think, for master SLAM, accuracy was okay, but the only issue is I think it runs, seven FPS on a desktop, forty ninety. That means we're using a laptop, it would be slower. And for a policy real time inference, I would say that's not enough for us. Okay. That makes sense. [00:07:08] S0: Yeah. Got it.
00:07:08 · Chris Paxton, Lawrence Zhu, Michael Cho
[00:07:08] S0: Yeah. Got it. [00:07:09] S1: So, yeah, I'll move on to how the Aria sensing modalities are used for a static co training system ego mimic as I mentioned before for ego mimics they mount the same aria glasses on the robot and with the human for collecting data we adopt this design as well So the issue here is when human collects data, you're actually moving your head. There's no way you'd fork your head like a robot there when when the robot actually needs a unified reference frame. Okay. So in this case, we need to stabilize the reference frame to use, like, human hand track as actions. Right? So the solution here is Aria provided SLAM. So what we basic... What they basically did was they project series of future hand pose, basically an action chunk, to the corresponding frame head pose so that those points got projected into the world frame. And in that case, if we project this whole chunk in world frame back to my current egocentric device frame, then I'll have a unified reference frame for all the future actions. Yeah. Any questions regarding this? [00:08:14] S0: Yeah. Got it. So so, basically, it's it's important to know... I mean, yeah, the the the camera pose itself is moving, so you you take in time with that. Yeah. Okay. [00:08:25] S3: Yeah. Mhmm.
00:08:25 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:08:25] S3: Yeah. Mhmm. [00:08:26] S1: Cool. And I have a little demo here of me doing a dummy test. And the green trajectory on the left hand side will basically be the projection of temples in this frame. And on the right, the arrows means, like, the human orientation. So you can see here, it's basically, by projecting this. You can pretty much see this is like a video game of, like, my hand floating in the air, and there's, like, super accurate track over time of where the hand pose is going because they have, like, really good, close loop slam trajectories. [00:08:58] S3: This really is an incredibly good hand tracking Yeah. System through space. I think that's [00:09:02] S1: why it's offline as well. Yeah. So, like, they really have, like, great optimization for this. Okay. [00:09:08] S0: Yeah. So So Sorry. Sorry. I I... So so just just Yep. Yeah. So, basically, I mean, so I I see all this, like, you know, on I'm seeing on the right, right, you have this red color with a green... Yellow color arrow, and then that's... Can you comment what are these? Like, looks very fancy. Yeah. Sorry. How do you interpret this?