00:02:41 · Lawrence Zhu, Pranav
And especially scaling imitation learning, it's all about, like, we collect data about observation action pairs, and we try to fulfill this entire three d task space with as many trajectories as possible. For example, like one observation of this chip picking task, like if you have the chip bag at one position and then you move it slightly over to another random position, you should have a corresponding demonstration for it. And collecting robot data or what we call behavior cloning is extremely slow in this case so what we show, well sorry what they show in ego mimic is basically by adding human trajectories they're actually able to fulfill this three d space and provide quantitative scaling effect The shared sensor we use here is Aria glasses. So similar as when they use in EgoMimic and EgoZero, they've been in this full cast before. And Aria glasses, it has one RGB forward facing camera and two slam cameras, and they also have two inward facing eye track cameras. So they have a service, they're from Meta. So they have a service called machine perception system m p... Service MPS. So what we can get from these glasses is we can get a real accurate hand pose they recently added a 21 keypoint per hand and we also have an offline SLAM of six DOF glasses posing the world frame and they also have like a semi dense observation and point clouds also they offer eye track Shall we pause here for a little bit of discussion? Pranav, do wanna add anything? Or [00:04:02] S2: Actually, we should ask any... Like, do guys have any questions for us? Or, like Yeah.
00:04:02 · Pranav, Chris Paxton, Lawrence Zhu
[00:04:02] S2: Actually, we should ask any... Like, do guys have any questions for us? Or, like Yeah. [00:04:06] S0: Well... Yeah. Yeah. Well, thank you so much. Yeah. I I am actually very curious about these Aria glasses. [00:04:11] S1: Mhmm. [00:04:12] S0: So, like, you know, hand pose, offline slam, and even the eye glaze. Right? So in in your work, like, which of these are... Would you say is absolutely critical? And also to what extent are they critical? Like, for example, the slam, how how accurate do the slam have to be? Is it like, you know, Apple Vision Pro, you know, you walk away 100 meter, you come back. Is this... You know, like... Yeah. Yeah. What is useful enough so that you can do this mobile manipulation task?
00:04:12 · Chris Paxton, Lawrence Zhu
[00:04:12] S0: So, like, you know, hand pose, offline slam, and even the eye glaze. Right? So in in your work, like, which of these are... Would you say is absolutely critical? And also to what extent are they critical? Like, for example, the slam, how how accurate do the slam have to be? Is it like, you know, Apple Vision Pro, you know, you walk away 100 meter, you come back. Is this... You know, like... Yeah. Yeah. What is useful enough so that you can do this mobile manipulation task? [00:04:39] S1: Yeah. So... Yeah. Thanks. I think we are pretty much the only works that, like, we actively leverage this SLAM for mobile tasks because I can't actually move on to the next slide. I'm gonna talk about how EgoMimic used it to solve a static problem. But in this case, in all the sensor modalities they provide, we... They they really have, like, super good closed loop trajectory because, I think they have this on a closed loop or... Sorry, closed closed source algorithms. So what we do is we collect a VRS package with human demonstration. We upload to the meta server, they process it. They come... They get it back to us. So the closed loop trajectory is really accurate, and I believe that is super crucial for... Not only for us, but also for, like, I think, static manipulation tasks because I'm gonna mention it in a bit, is they require a unified reference system that makes a hard hand... Sorry. Hand pose super critical in this case. I don't know if you wanna add.
00:04:39 · Lawrence Zhu, Pranav, Chris Paxton
[00:04:39] S1: Yeah. So... Yeah. Thanks. I think we are pretty much the only works that, like, we actively leverage this SLAM for mobile tasks because I can't actually move on to the next slide. I'm gonna talk about how EgoMimic used it to solve a static problem. But in this case, in all the sensor modalities they provide, we... They they really have, like, super good closed loop trajectory because, I think they have this on a closed loop or... Sorry, closed closed source algorithms. So what we do is we collect a VRS package with human demonstration. We upload to the meta server, they process it. They come... They get it back to us. So the closed loop trajectory is really accurate, and I believe that is super crucial for... Not only for us, but also for, like, I think, static manipulation tasks because I'm gonna mention it in a bit, is they require a unified reference system that makes a hard hand... Sorry. Hand pose super critical in this case. I don't know if you wanna add. [00:05:32] S2: Yeah. Going off of that, I think it depends a lot on the project. So, like, as Lawrence was saying, like, if you're focusing on static manipulation, hand pose can be very important. But, like, we're trying to extend that to mobile manipulation. Right? And so we want to have other things... Other tools that we need to use, like SLAM and Point Cloud. And so Lawrence is gonna talk about that in a little bit too. [00:05:51] S1: Yep. Sounds good. That's... [00:05:52] S0: So You guys have a... I Yeah.
00:05:52 · Chris Paxton, Michael Cho, Lawrence Zhu
[00:05:52] S0: So You guys have a... I Yeah. [00:05:53] S3: I didn't notice that they actually sent it up to the meta server every time that you wanna do... Does it... Can you actually do it real time, or is this all, like Oh. Because it has to be post processed? Yeah. [00:06:02] S1: For the the first generation, they don't have anything, I think, real time. But for the second gen, they're gonna release, I think. They released it already. They're gonna have... It shipped to us, I think Okay. Like, later this year. That one has Yeah. [00:06:13] S0: Very [00:06:13] S1: cool. I think, real time depth estimation. In that case, we are actually... Yeah. Actually able to not slam. Yeah. Okay. [00:06:20] S3: Or not to close loop slam. Because, okay. Yeah. No. That makes sense.
00:06:20 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:06:20] S3: Or not to close loop slam. Because, okay. Yeah. No. That makes sense. [00:06:22] S1: Yeah. If we wanna extend a little bit here, I I watched your episode about EgoZero. That episode, they talk about MASt3R-SLAM. We tried exactly the same pipeline. So the reason we're not... Because, I think we... When we tried to design this project in the beginning, it's like we have some sort of co training of, like, robot data and human data, and we just have some random SLAM or, like, planning algorithm to help us to navigate from one point to another, and then we perform another co training manipulation task over there. But we realized, I think, for master SLAM, accuracy was okay, but the only issue is I think it runs, seven FPS on a desktop, forty ninety. That means we're using a laptop, it would be slower. And for a policy real time inference, I would say that's not enough for us. Okay. That makes sense. [00:07:08] S0: Yeah. Got it.