00:00:00 · Chris Paxton, Lawrence Zhu, Pranav
[00:00:00] S0: Alright, guys. Welcome to another episode of RoboPapers. Super happy today to have Lawrence and Pranav. They are two of the co authors of this paper called EMMA, scaling mobile manipulation via ecocentric human data. Thank you so much for making the time, guys. Maybe as a side, if you guys can share a little bit about yourself before we dive into the paper. [00:00:21] S1: Yep. Thanks for having us, Michael and Chris. My name's Lawrence. I'm a second year master's student at Georgia Tech with professor Danfei Xu And Pranav? [00:00:30] S2: Yeah. I'm Pranav. I am a first year master's student at Georgia Tech advised by doctor Danfei Xu as well.
00:00:36 · Lawrence Zhu
[00:00:36] S1: Yep. Cool. Thanks so much again. So today, I'll be presenting our paper, EMMA, scaling mobile manipulation with egos and trick human videos. So, we'll start with a little brief introduction about, like, what's our problem. Right? So when we talk about scaling, it means, like, we try to add more data to train a generalizable robot policy. And for mobile manipulation, this is, like, slightly different to, like, what we've been doing right now like, static table top manipulation because the data is much harder to acquire. Basically, teaching robots mobile manipulation is hard, and this complex task suppress, like, hundreds of hours of, like, mobile tally operation data. And we have a little demo here of me doing the same task, and Pranav and I tally operating the robot for exactly the same task with the exact same, like, eight times speed up. You can see, mobile tally up takes much longer, like, through, like, to complete the exact same task. And even for, Mobile ALOHA, I would con... I I would personally consider they are probably the most user friendly, mobile tally operation system because you can just push the robot around by, like, just pulling that piece of tissue... Sorry. Just pulling that piece of tissue out of the box, it take, like, half a minute. So this comes to, maybe our problem. So it's the knowledge transfer bottleneck for imitation learning. As when we try to transfer the human knowledge to robots, we basically try... What we're doing right now is tally operation. And for those of you who's familiar with tally operation, it's extremely difficult and slow. And so it requires robots in the loop, and the only benefit here is there's no embodiment gap. Whatever you tally up will be whatever the robot sees during the inference time. So we're trying to break our IntelliLab. We try to break this bottleneck with embodied human data, and we try to have alternative data sources humans instead of robots, and then contains embodied sensor motor experience and capture without continuous effort. So everything in, like, this direction starts with this paper called EgoMimic led by Simar. And this paper is done before Pranav and I joined the lab, so we're not taking any credits on it. But what they proposed here is basically when we talk about imitation learning, it's a pro... It's...
00:02:41 · Lawrence Zhu, Pranav
And especially scaling imitation learning, it's all about, like, we collect data about observation action pairs, and we try to fulfill this entire three d task space with as many trajectories as possible. For example, like one observation of this chip picking task, like if you have the chip bag at one position and then you move it slightly over to another random position, you should have a corresponding demonstration for it. And collecting robot data or what we call behavior cloning is extremely slow in this case so what we show, well sorry what they show in ego mimic is basically by adding human trajectories they're actually able to fulfill this three d space and provide quantitative scaling effect The shared sensor we use here is Aria glasses. So similar as when they use in EgoMimic and EgoZero, they've been in this full cast before. And Aria glasses, it has one RGB forward facing camera and two slam cameras, and they also have two inward facing eye track cameras. So they have a service, they're from Meta. So they have a service called machine perception system m p... Service MPS. So what we can get from these glasses is we can get a real accurate hand pose they recently added a 21 keypoint per hand and we also have an offline SLAM of six DOF glasses posing the world frame and they also have like a semi dense observation and point clouds also they offer eye track Shall we pause here for a little bit of discussion? Pranav, do wanna add anything? Or [00:04:02] S2: Actually, we should ask any... Like, do guys have any questions for us? Or, like Yeah.
00:04:02 · Pranav, Chris Paxton, Lawrence Zhu
[00:04:02] S2: Actually, we should ask any... Like, do guys have any questions for us? Or, like Yeah. [00:04:06] S0: Well... Yeah. Yeah. Well, thank you so much. Yeah. I I am actually very curious about these Aria glasses. [00:04:11] S1: Mhmm. [00:04:12] S0: So, like, you know, hand pose, offline slam, and even the eye glaze. Right? So in in your work, like, which of these are... Would you say is absolutely critical? And also to what extent are they critical? Like, for example, the slam, how how accurate do the slam have to be? Is it like, you know, Apple Vision Pro, you know, you walk away 100 meter, you come back. Is this... You know, like... Yeah. Yeah. What is useful enough so that you can do this mobile manipulation task?
00:04:12 · Chris Paxton, Lawrence Zhu
[00:04:12] S0: So, like, you know, hand pose, offline slam, and even the eye glaze. Right? So in in your work, like, which of these are... Would you say is absolutely critical? And also to what extent are they critical? Like, for example, the slam, how how accurate do the slam have to be? Is it like, you know, Apple Vision Pro, you know, you walk away 100 meter, you come back. Is this... You know, like... Yeah. Yeah. What is useful enough so that you can do this mobile manipulation task? [00:04:39] S1: Yeah. So... Yeah. Thanks. I think we are pretty much the only works that, like, we actively leverage this SLAM for mobile tasks because I can't actually move on to the next slide. I'm gonna talk about how EgoMimic used it to solve a static problem. But in this case, in all the sensor modalities they provide, we... They they really have, like, super good closed loop trajectory because, I think they have this on a closed loop or... Sorry, closed closed source algorithms. So what we do is we collect a VRS package with human demonstration. We upload to the meta server, they process it. They come... They get it back to us. So the closed loop trajectory is really accurate, and I believe that is super crucial for... Not only for us, but also for, like, I think, static manipulation tasks because I'm gonna mention it in a bit, is they require a unified reference system that makes a hard hand... Sorry. Hand pose super critical in this case. I don't know if you wanna add.
00:04:39 · Lawrence Zhu, Pranav, Chris Paxton
[00:04:39] S1: Yeah. So... Yeah. Thanks. I think we are pretty much the only works that, like, we actively leverage this SLAM for mobile tasks because I can't actually move on to the next slide. I'm gonna talk about how EgoMimic used it to solve a static problem. But in this case, in all the sensor modalities they provide, we... They they really have, like, super good closed loop trajectory because, I think they have this on a closed loop or... Sorry, closed closed source algorithms. So what we do is we collect a VRS package with human demonstration. We upload to the meta server, they process it. They come... They get it back to us. So the closed loop trajectory is really accurate, and I believe that is super crucial for... Not only for us, but also for, like, I think, static manipulation tasks because I'm gonna mention it in a bit, is they require a unified reference system that makes a hard hand... Sorry. Hand pose super critical in this case. I don't know if you wanna add. [00:05:32] S2: Yeah. Going off of that, I think it depends a lot on the project. So, like, as Lawrence was saying, like, if you're focusing on static manipulation, hand pose can be very important. But, like, we're trying to extend that to mobile manipulation. Right? And so we want to have other things... Other tools that we need to use, like SLAM and Point Cloud. And so Lawrence is gonna talk about that in a little bit too. [00:05:51] S1: Yep. Sounds good. That's... [00:05:52] S0: So You guys have a... I Yeah.