00:06:10 · Vincent Liu, Ademi Adeniji, Michael Cho
[00:06:10] S1: Right. I think, like, that issue sort of is also present in Teleop where you're... But, like, you're you're handling that in a different point in the stack. Right? And so... Yeah. So, basically, to to, like, retarget anything to, like, a gripper setting, you know, when you're teleoping by, like, a band manual gripper, you're also constraining... You're making assumptions about the tasks you can solve and how you're collecting the data. So in this work, it's pretty... You know, we just retarget, like, the the midpoint of the index and thumb to, the midpoint of the gripper, Just more so because we just wanted to demonstrate that, like, you know, this concept works in the egocentric setting. But you're right, like going forward, you know, maybe we can extract end effector positions of all five fingers then we target that to a dextrous hand. [00:06:59] S2: Yeah. It makes sense. [00:07:00] S0: Yeah. I can see in your videos that... At least from those that... On your website that I can see, like, if I'm not wrong, you you ask the the the person to kind of, like, pretend his hand is kinda like a cripple. Right? So it's maybe a lot easier to just tell the guy what to do, like, you know, imitate a little bit like the the end. Is is that intentional? I I assume it's intentional.
00:07:00 · Michael Cho, Vincent Liu
[00:07:00] S0: Yeah. I can see in your videos that... At least from those that... On your website that I can see, like, if I'm not wrong, you you ask the the the person to kind of, like, pretend his hand is kinda like a cripple. Right? So it's maybe a lot easier to just tell the guy what to do, like, you know, imitate a little bit like the the end. Is is that intentional? I I assume it's intentional. [00:07:21] S1: Yeah. That was definitely slightly intentional for this just mainly because, like, again, we're sort of proving the concept of this. And in our work, we basic... I mean, there's many open directions of improvement for this. Right? Like, one is that, you know, we also don't, as a result, learn the rotation of the end effector of the Franka. And so therefore, you know, like, if if you go the extra step, you know, perhaps you can also extract the rotation of your hand based on the hand model. But we sort of like, there's already a lot going on to sort of get this proof of concept to work, then we just wanted to show that basically. Got it. Also, Adame, if you have any things you'd like to add, just feel free to jump in as well.
00:07:59 · Ademi Adeniji, Chris Paxton, Michael Cho
[00:07:59] S2: Yeah. I mean, I think like an interesting tangent here is that there's a lot of renewed interest in one to one morphology mapping between robots and humans in terms of hardware converging to the human form factor, which I think further motivates this idea that if you can collect ground truth for humans, and build a general purpose hardware system, a lot of the embodiment gaps that we spend a lot of time trying to solve with clever algorithms, a lot of those get solved just because your problem space is easier. Like there's tighter domain alignment between basically your training data and your inference. And so I think like on one hand, yes, there are some mild assumptions you make when you retarget to, for instance, an antipodal gripper. But I think it's also an advantage and the upshot is that if you have a sufficiently sort of high dimension... High dimensional end effector, you have like the ground truth information already baked into your data because you're you're operating with like basically your your full your full dexterity. So, yeah, it's like a trade off for sure. [00:09:13] S3: Yeah. So your hope would be to go more towards, like... So while while you're showing these things on the front, two two finger Franka gripper. Right? Like, you're... The hope is that you would be able to apply this to, like, a much more dexterous humanoid with a lower embodiment gap and that, therefore, these issues should be less pronounced. Right? [00:09:29] S2: Definitely. Okay. [00:09:30] S0: Yeah. Yeah.
00:09:30 · Michael Cho, Vincent Liu, Chris Paxton
[00:09:30] S0: Yeah. Yeah. [00:09:30] S1: And I think, like, some some of our collaborators at NYU are carrying this direction forward. You know, one, one goes like, okay. If you wanted to retarget to the Franka, you know, how how would you extract the rotation? How would you retarget thing? How would you know you'd, like, ask all these questions about how you would do the retargeting? But when you actually just move to the dextrous setting, you can just retarget the fingers, and it's relatively simple with, like, IK these days. Right? [00:09:54] S3: Yeah. That's fair. But I I mean, I guess, I don't wanna derail this. Like... But even when you're doing kinematic retargeting to, like, the unit three g one, like, that's that's, like, toddler size or something like this. People do this all the time for teleoperation, but, like, it's so much shorter than, me that I imagine there'd be a gap, but maybe not. That's a... It's an interesting question. Yeah.
00:09:54 · Chris Paxton, Ademi Adeniji, Michael Cho
[00:09:54] S3: Yeah. That's fair. But I I mean, I guess, I don't wanna derail this. Like... But even when you're doing kinematic retargeting to, like, the unit three g one, like, that's that's, like, toddler size or something like this. People do this all the time for teleoperation, but, like, it's so much shorter than, me that I imagine there'd be a gap, but maybe not. That's a... It's an interesting question. Yeah. [00:10:14] S2: Yeah. It's... I guess, so I'll just add one more point. Yeah. I think that's a great point. Like, scale does matter in terms of, like, literally the size. I think because so much of what we're looking at is dextrous manipulation, so long as your full body, whole body control works and your whole body IK is working, as long as the hands, I think, are the right size, you sort of are already almost there. We actually tried Okay. Getting this working with the Allegro hand. And the kinematic retargeting there is just super, super complicated because, you know Okay. You can't do it joint space wise because, like, the forward kinematics are completely different. And, yeah, like, sort of that's like a a a... I think an exercise in futility. So, we moved to the gripper and now hoping, like, hands are more one to one with human hands moving forward in terms of size. [00:11:00] S0: Yeah. I
00:11:00 · Michael Cho, Vincent Liu, Ademi Adeniji, Chris Paxton
[00:11:00] S0: Yeah. I [00:11:00] S1: I also think, like, there's a benefit here for representing things in three d in egocentric frame. Right? So you're basically... What you're what you're asking the policy to do is sort of predict movements of, like, of three d points in a three d space. You know? And then in inference time, you sort of, like, can arbitrarily, as we show, like, sort of put this camera somewhere. And then if the policy is well trained, like like, you know, the... It would just be sort of what a Demi is saying, like a full body retargeting there and hopefully, like, relatively morphology agnostic. [00:11:29] S2: Makes sense. [00:11:30] S1: K. Should I continue here? [00:11:32] S3: Yeah. So how do we collect data?