00:14:34 · Vincent Liu
[00:14:34] S1: Yeah. Okay. So sort of proceeding with the method then. Like I said, like, conceptually, this is inspired a lot by point policy. So, you know, we start out with the same sort of procedure where you label set of two d object points on the training image, like this is like the expert image, and then you map these in two d onto the first frame and throughout the demonstration for each example in your training set. But in the... With the Aria glasses in the egocentric setting, you do not have two stationary cameras that are calibrated to triangulate these points. And so we then sort of make a couple assumptions to get this to work. One is that the SLAM extrinsics are very good, which they are on these on on these glasses. And two, by assuming that the object is stationary, you're able to sort of triangulate over time, so, like, over the camera's trajectory. So it's like, you know, point policy is triangulating two cameras, fixed point in time. If you just imagine, like, as you move your head over, like, one second, you know, you you have, like, very accurate camera information, and you can localize the object. And so this is sort of how we get around can't triangulate stereo... With stereo per frame. You don't have, like, LiDAR or really ground truth depth measurements. But if you make this assumption and leverage, like, a good camera information, you're able to localize the object well. It's just that we can't actually track the object, right, because we're assuming that it's stationary and triangling over time. So the models we train are like, in a way, like open loop with respect to the object state, but we're able to, like, you know, track the fingertip movements in every frame. So it's like... That's why we sort of say closed loop. And also that... Yeah.
00:16:11 · Chris Paxton, Ademi Adeniji, Vincent Liu
[00:16:11] S3: That video earlier of it opening, like, a oven or something, though. Right? Or some some kinda... Yeah. So how does that work? Because that is not stationary. [00:16:20] S2: Yeah. [00:16:21] S1: Yes. Go ahead, Demi. If you go. [00:16:23] S2: So so, like, I think... So funnily enough, because you have proprioception closed loop and your model is ingesting proprioceptive input closed loop, you actually can solve a lot of these tasks without per frame object tracking. I will add also though that one, this approach is sort of, I think, necessarily limiting. So you can, for instance, the slam readings are very high, read out at very high FPS. So if there's sufficient camera movement, you can just triangulate over windows of time as opposed to over the entire trajectory. And we use RANSAC in order to actually do the PNP mapping solve for the actual pose of the object. But you know, we have very, very high precision there after triangulation. And so you could definitely shorten the window over which you do this triangulation and have something that approximates something more closely. Now, of course, the best solution would just be to have some ground truth depth estimation via LiDAR or just have two cameras. You could do stereo triangulation. Right.
00:16:23 · Ademi Adeniji, Vincent Liu
[00:16:23] S2: So so, like, I think... So funnily enough, because you have proprioception closed loop and your model is ingesting proprioceptive input closed loop, you actually can solve a lot of these tasks without per frame object tracking. I will add also though that one, this approach is sort of, I think, necessarily limiting. So you can, for instance, the slam readings are very high, read out at very high FPS. So if there's sufficient camera movement, you can just triangulate over windows of time as opposed to over the entire trajectory. And we use RANSAC in order to actually do the PNP mapping solve for the actual pose of the object. But you know, we have very, very high precision there after triangulation. And so you could definitely shorten the window over which you do this triangulation and have something that approximates something more closely. Now, of course, the best solution would just be to have some ground truth depth estimation via LiDAR or just have two cameras. You could do stereo triangulation. Right. [00:17:31] S1: I was gonna... Yeah. So back to like, how does it open the thing? In our our ablation... Or I guess baselines, we sort of test against a method where we just predict, like, a six point trajectory, like, and then just execute it. And so it's still able to do that. Right? So it's not so much like, oh, if you can't track things, how do you open the oven? It's just like, I guess, maybe your question is more... Well, yeah, maybe I'm not understanding your question.
00:17:31 · Vincent Liu, Chris Paxton, Ademi Adeniji
[00:17:31] S1: I was gonna... Yeah. So back to like, how does it open the thing? In our our ablation... Or I guess baselines, we sort of test against a method where we just predict, like, a six point trajectory, like, and then just execute it. And so it's still able to do that. Right? So it's not so much like, oh, if you can't track things, how do you open the oven? It's just like, I guess, maybe your question is more... Well, yeah, maybe I'm not understanding your question. [00:17:56] S3: Yes. Yeah. So so the... That's... In that case, opening the oven for... Is is open loop. You wouldn't, like, be able to respond if the oven got jammed, I guess. Is that... That's what I would [00:18:04] S2: imagine then? [00:18:05] S1: Okay. So basically, what the model sees is like a snapshot, like summarized with like three d points, and then it goes and knows where... So it's kind of like memorized where it is. Right? [00:18:15] S2: Yeah. Makes sense. [00:18:15] S1: So if the object is moving, you'd probably be in trouble. [00:18:17] S3: Sure. Yeah. So okay.
00:18:19 · Vincent Liu
[00:18:19] S1: Yeah. Okay. Yeah. So going forward then, so I just described how we get the object points. And so the other path of the equation is how do you get the end effector points or basically your index thumb finger from these recordings. So actually, shortly after we, I think, released EgoZero, this step of computing 21 hand key points in camera frame, like, it was released as an update to the Aria glasses. And we took a look at the outputs, and they look pretty similar. So maybe they're doing something similar here. But the idea is that because the Aria MPS service only outputs six DOF... Accurate six DOF transformations of, you know, your palms and wrist in camera frame, you don't have, you know, the fine grained information of how your... Where where your fingers are. Right? And so when we sort of look at monocular hand estimation models like HaMeR, we see that, you know, we we observed that, like, the hand shapes look pretty correct. It's just that if you read out the pose, the predicted pose of where the hand is in camera frame from HaMeR, it's very incorrect. Right? Because you're sort of estimating the three d from a single image, which is, like, kind of a difficult problem. And so what we thought is, okay, is there a way we can borrow the local hand information from HaMeR or these hand estimation models and, like, piece it together with accurate 6DOF, like, hand, like, pose in camera frame? And so this is sort of what we did. We, like, kind of took a look at where the palm is with respect to, like, the the morphology of the hand and the key points. We kind of compute a centroid and then align things, and then you add them together. So, like, conceptually, I like to think of this as you're, like, kind of projecting out this incorrect information predicted by HaMeR, and, like, you're sort of, like, pasting in this accurate information by Aria.
00:20:10 · Michael Cho, Vincent Liu
[00:20:10] S0: But you're saying that the... It sounds like Aria... I don't know. Is it like a MPS plus that basically have this... Maybe being able able to do the hands well as well is is what you're saying? [00:20:20] S1: Yeah. Yeah. So, like, I mean, literally, I think right before we open source the the... Our our code, they released the update to their SDK. And you literally can just read this out for free. You don't have to run HaMeR on your GPU and all this stuff. Yeah. But, like, the the the sort of, like, qualitative assessment of the two outputs look pretty similar. So, you know, I would imagine if they have, like, access to the whole hardware system, you know, the stereo cameras and SLAM and everything, it probably wouldn't be too hard to either like, do what we did or have a version of HaMeR that is more accurate because it's like, with stereo and camera information. Yeah, that's kind of like a general theme we've felt like post this project is, you know, a lot of these hoops that we had to jump through are pretty easily amenable. Right? Like, Aria now offers 21 hand key points for free. All you need to do for... Instead of triangulating over your SLAM trajectory, if you want a fully closed loop setting, you can just attach a LIDAR to it. Right? So it's like these solutions are sort of not that fundamental in a way. It's just sort of, like, to kinda prove this with the current hardware we had, we had to sort of be clever about. [00:21:26] S0: Makes sense. And