00:07:00 · Michael Cho, Vincent Liu
[00:07:00] S0: Yeah. I can see in your videos that... At least from those that... On your website that I can see, like, if I'm not wrong, you you ask the the the person to kind of, like, pretend his hand is kinda like a cripple. Right? So it's maybe a lot easier to just tell the guy what to do, like, you know, imitate a little bit like the the end. Is is that intentional? I I assume it's intentional. [00:07:21] S1: Yeah. That was definitely slightly intentional for this just mainly because, like, again, we're sort of proving the concept of this. And in our work, we basic... I mean, there's many open directions of improvement for this. Right? Like, one is that, you know, we also don't, as a result, learn the rotation of the end effector of the Franka. And so therefore, you know, like, if if you go the extra step, you know, perhaps you can also extract the rotation of your hand based on the hand model. But we sort of like, there's already a lot going on to sort of get this proof of concept to work, then we just wanted to show that basically. Got it. Also, Adame, if you have any things you'd like to add, just feel free to jump in as well.
00:07:59 · Ademi Adeniji, Chris Paxton, Michael Cho
[00:07:59] S2: Yeah. I mean, I think like an interesting tangent here is that there's a lot of renewed interest in one to one morphology mapping between robots and humans in terms of hardware converging to the human form factor, which I think further motivates this idea that if you can collect ground truth for humans, and build a general purpose hardware system, a lot of the embodiment gaps that we spend a lot of time trying to solve with clever algorithms, a lot of those get solved just because your problem space is easier. Like there's tighter domain alignment between basically your training data and your inference. And so I think like on one hand, yes, there are some mild assumptions you make when you retarget to, for instance, an antipodal gripper. But I think it's also an advantage and the upshot is that if you have a sufficiently sort of high dimension... High dimensional end effector, you have like the ground truth information already baked into your data because you're you're operating with like basically your your full your full dexterity. So, yeah, it's like a trade off for sure. [00:09:13] S3: Yeah. So your hope would be to go more towards, like... So while while you're showing these things on the front, two two finger Franka gripper. Right? Like, you're... The hope is that you would be able to apply this to, like, a much more dexterous humanoid with a lower embodiment gap and that, therefore, these issues should be less pronounced. Right? [00:09:29] S2: Definitely. Okay. [00:09:30] S0: Yeah. Yeah.
00:09:30 · Michael Cho, Vincent Liu, Chris Paxton
[00:09:30] S0: Yeah. Yeah. [00:09:30] S1: And I think, like, some some of our collaborators at NYU are carrying this direction forward. You know, one, one goes like, okay. If you wanted to retarget to the Franka, you know, how how would you extract the rotation? How would you retarget thing? How would you know you'd, like, ask all these questions about how you would do the retargeting? But when you actually just move to the dextrous setting, you can just retarget the fingers, and it's relatively simple with, like, IK these days. Right? [00:09:54] S3: Yeah. That's fair. But I I mean, I guess, I don't wanna derail this. Like... But even when you're doing kinematic retargeting to, like, the unit three g one, like, that's that's, like, toddler size or something like this. People do this all the time for teleoperation, but, like, it's so much shorter than, me that I imagine there'd be a gap, but maybe not. That's a... It's an interesting question. Yeah.
00:09:54 · Chris Paxton, Ademi Adeniji, Michael Cho
[00:09:54] S3: Yeah. That's fair. But I I mean, I guess, I don't wanna derail this. Like... But even when you're doing kinematic retargeting to, like, the unit three g one, like, that's that's, like, toddler size or something like this. People do this all the time for teleoperation, but, like, it's so much shorter than, me that I imagine there'd be a gap, but maybe not. That's a... It's an interesting question. Yeah. [00:10:14] S2: Yeah. It's... I guess, so I'll just add one more point. Yeah. I think that's a great point. Like, scale does matter in terms of, like, literally the size. I think because so much of what we're looking at is dextrous manipulation, so long as your full body, whole body control works and your whole body IK is working, as long as the hands, I think, are the right size, you sort of are already almost there. We actually tried Okay. Getting this working with the Allegro hand. And the kinematic retargeting there is just super, super complicated because, you know Okay. You can't do it joint space wise because, like, the forward kinematics are completely different. And, yeah, like, sort of that's like a a a... I think an exercise in futility. So, we moved to the gripper and now hoping, like, hands are more one to one with human hands moving forward in terms of size. [00:11:00] S0: Yeah. I
00:11:00 · Michael Cho, Vincent Liu, Ademi Adeniji, Chris Paxton
[00:11:00] S0: Yeah. I [00:11:00] S1: I also think, like, there's a benefit here for representing things in three d in egocentric frame. Right? So you're basically... What you're what you're asking the policy to do is sort of predict movements of, like, of three d points in a three d space. You know? And then in inference time, you sort of, like, can arbitrarily, as we show, like, sort of put this camera somewhere. And then if the policy is well trained, like like, you know, the... It would just be sort of what a Demi is saying, like a full body retargeting there and hopefully, like, relatively morphology agnostic. [00:11:29] S2: Makes sense. [00:11:30] S1: K. Should I continue here? [00:11:32] S3: Yeah. So how do we collect data?
00:11:32 · Chris Paxton, Vincent Liu, Michael Cho
[00:11:32] S3: Yeah. So how do we collect data? [00:11:34] S1: Oh, yeah. Oh, yeah. So the... Exactly. So now we sort of get to the meat of the contribution of the paper, which is how do we collect three d data from smart glasses, specifically the Aria glasses. So the Aria glasses have very good SLAM. You know, it has two visual inertial cameras on the side and RGB fisheye on the left side. And it's software package. It's offline like a MPS service, basically. Comes with accurate six off hand poses. And, like, these are both really accurate, like, as we sort of played around with it. However, the downside to the Aria glasses, at least the Gen one, I'm... I've heard the Gen two is, you know, is slightly more amenable to some of these gen one drawbacks. But at least the gen one glasses have minimal FOV overlap between these SLAM cameras and also no, like, ground truth way of getting depth. And so, you know, the two way... The the two, like, most standard ways to get three d information is with a triangulation or, like, with a LIDAR or something, and the Aria glasses did not provide either. And so this is sort of, like, the main problem we had to overcome. [00:12:37] S0: I'm curious. I don't know whether that that they have ever publicized it. Like, roughly how much do these cost, Like, if you were to get more of these... Are these, like, regular, like, smart glasses pricing, like, couple $100? Or is it, like, I don't know whether you're able to share any of this or whether [00:12:54] S1: they I personally have no idea. [00:12:56] S0: Oh, okay.