00:03:21 · Vincent Liu
Like, the previous smart glasses work, EgoMimic, they need to co train with Teleopt data. And so, you know, in all these previous works, you need to make assumptions about your data collection setup. And our goal was to sort of like, how can we actually design the system in a way where you can make basically no assumptions about your environment? And so really conceptually, EgoZero builds on point policy, which is a work that come that came out of the same group by Siddhant and Lerrel. And basically, in this paper, they rep... Sort of defined this unified representation space so that you can represent humans and robot morphologies in a single space. This way, if you define it well, in which case it's three d points, you're able to collect data in one, represent it in this space, and have it transferred to the other morphology. So this is exactly what they do. The process is they label some two d points on an expert image, map these points onto each demo, and track them throughout the demo. And then in point policy, he uses two cameras to triangulate these points into three d. And this triangulation is used also for detecting these hand key points. So he uses, like, some basic hand model, and you're able to triangulate these into three d. And so before going into, like, I guess, the egocentric setting, we should talk about why these three d points are good and, like, the benefits of these three d points. First is that they are morphology agnostic. So you're basically encoding, like, very sparse geometry and correspondence of the object and also, like, the location of your end effector. And, like, these concepts are applicable in both human data and, like, robot robot inference, I guess. And then also, because now your policy learning problem is a three d to three d mapping, it is very easy to learn and very easy to apply data augmentation in this very linear space. And because there's correspondence, the object pose is encoded explicitly where your points are ordered. And... Yeah. So so, like like, in the, like, the vision based, like, VLA approaches, you would kinda have to learn these this information through data. And, also, like, the three d to three d space avoids this difficult two d to three d learning problem, which is kind of inherent to all these vision based methods. Yeah.
00:05:33 · Vincent Liu, Chris Paxton
And so the main contribution of EgoZero is building on this unified three d representation space. We can show that it works in a zero shot setting from Ego centric in the wild data. And so these are some videos that you can find on our website as well. [00:05:48] S3: So question before I move on. So, like, I I love the point policy kind of, like, representation space. I think it's really cool. But, like, one issue with those kinds of things is always, like, what if the morphology of the robot is very different from the human? Now what do you what do you think about that? So, like, for example, like your Franka there. Right? Like, I have five fingers. The Franka only has two. How do you handle that? [00:06:10] S1: Right. I think, like, that issue sort of is also present in Teleop where you're... But, like, you're you're handling that in a different point in the stack. Right? And so... Yeah. So, basically, to to, like, retarget anything to, like, a gripper setting, you know, when you're teleoping by, like, a band manual gripper, you're also constraining... You're making assumptions about the tasks you can solve and how you're collecting the data. So in this work, it's pretty... You know, we just retarget, like, the the midpoint of the index and thumb to, the midpoint of the gripper, Just more so because we just wanted to demonstrate that, like, you know, this concept works in the egocentric setting. But you're right, like going forward, you know, maybe we can extract end effector positions of all five fingers then we target that to a dextrous hand.
00:06:10 · Vincent Liu, Ademi Adeniji, Michael Cho
[00:06:10] S1: Right. I think, like, that issue sort of is also present in Teleop where you're... But, like, you're you're handling that in a different point in the stack. Right? And so... Yeah. So, basically, to to, like, retarget anything to, like, a gripper setting, you know, when you're teleoping by, like, a band manual gripper, you're also constraining... You're making assumptions about the tasks you can solve and how you're collecting the data. So in this work, it's pretty... You know, we just retarget, like, the the midpoint of the index and thumb to, the midpoint of the gripper, Just more so because we just wanted to demonstrate that, like, you know, this concept works in the egocentric setting. But you're right, like going forward, you know, maybe we can extract end effector positions of all five fingers then we target that to a dextrous hand. [00:06:59] S2: Yeah. It makes sense. [00:07:00] S0: Yeah. I can see in your videos that... At least from those that... On your website that I can see, like, if I'm not wrong, you you ask the the the person to kind of, like, pretend his hand is kinda like a cripple. Right? So it's maybe a lot easier to just tell the guy what to do, like, you know, imitate a little bit like the the end. Is is that intentional? I I assume it's intentional.
00:07:00 · Michael Cho, Vincent Liu
[00:07:00] S0: Yeah. I can see in your videos that... At least from those that... On your website that I can see, like, if I'm not wrong, you you ask the the the person to kind of, like, pretend his hand is kinda like a cripple. Right? So it's maybe a lot easier to just tell the guy what to do, like, you know, imitate a little bit like the the end. Is is that intentional? I I assume it's intentional. [00:07:21] S1: Yeah. That was definitely slightly intentional for this just mainly because, like, again, we're sort of proving the concept of this. And in our work, we basic... I mean, there's many open directions of improvement for this. Right? Like, one is that, you know, we also don't, as a result, learn the rotation of the end effector of the Franka. And so therefore, you know, like, if if you go the extra step, you know, perhaps you can also extract the rotation of your hand based on the hand model. But we sort of like, there's already a lot going on to sort of get this proof of concept to work, then we just wanted to show that basically. Got it. Also, Adame, if you have any things you'd like to add, just feel free to jump in as well.
00:07:59 · Ademi Adeniji, Chris Paxton, Michael Cho
[00:07:59] S2: Yeah. I mean, I think like an interesting tangent here is that there's a lot of renewed interest in one to one morphology mapping between robots and humans in terms of hardware converging to the human form factor, which I think further motivates this idea that if you can collect ground truth for humans, and build a general purpose hardware system, a lot of the embodiment gaps that we spend a lot of time trying to solve with clever algorithms, a lot of those get solved just because your problem space is easier. Like there's tighter domain alignment between basically your training data and your inference. And so I think like on one hand, yes, there are some mild assumptions you make when you retarget to, for instance, an antipodal gripper. But I think it's also an advantage and the upshot is that if you have a sufficiently sort of high dimension... High dimensional end effector, you have like the ground truth information already baked into your data because you're you're operating with like basically your your full your full dexterity. So, yeah, it's like a trade off for sure. [00:09:13] S3: Yeah. So your hope would be to go more towards, like... So while while you're showing these things on the front, two two finger Franka gripper. Right? Like, you're... The hope is that you would be able to apply this to, like, a much more dexterous humanoid with a lower embodiment gap and that, therefore, these issues should be less pronounced. Right? [00:09:29] S2: Definitely. Okay. [00:09:30] S0: Yeah. Yeah.
00:09:30 · Michael Cho, Vincent Liu, Chris Paxton
[00:09:30] S0: Yeah. Yeah. [00:09:30] S1: And I think, like, some some of our collaborators at NYU are carrying this direction forward. You know, one, one goes like, okay. If you wanted to retarget to the Franka, you know, how how would you extract the rotation? How would you retarget thing? How would you know you'd, like, ask all these questions about how you would do the retargeting? But when you actually just move to the dextrous setting, you can just retarget the fingers, and it's relatively simple with, like, IK these days. Right? [00:09:54] S3: Yeah. That's fair. But I I mean, I guess, I don't wanna derail this. Like... But even when you're doing kinematic retargeting to, like, the unit three g one, like, that's that's, like, toddler size or something like this. People do this all the time for teleoperation, but, like, it's so much shorter than, me that I imagine there'd be a gap, but maybe not. That's a... It's an interesting question. Yeah.