00:16:23 · Ademi Adeniji, Vincent Liu
[00:16:23] S2: So so, like, I think... So funnily enough, because you have proprioception closed loop and your model is ingesting proprioceptive input closed loop, you actually can solve a lot of these tasks without per frame object tracking. I will add also though that one, this approach is sort of, I think, necessarily limiting. So you can, for instance, the slam readings are very high, read out at very high FPS. So if there's sufficient camera movement, you can just triangulate over windows of time as opposed to over the entire trajectory. And we use RANSAC in order to actually do the PNP mapping solve for the actual pose of the object. But you know, we have very, very high precision there after triangulation. And so you could definitely shorten the window over which you do this triangulation and have something that approximates something more closely. Now, of course, the best solution would just be to have some ground truth depth estimation via LiDAR or just have two cameras. You could do stereo triangulation. Right. [00:17:31] S1: I was gonna... Yeah. So back to like, how does it open the thing? In our our ablation... Or I guess baselines, we sort of test against a method where we just predict, like, a six point trajectory, like, and then just execute it. And so it's still able to do that. Right? So it's not so much like, oh, if you can't track things, how do you open the oven? It's just like, I guess, maybe your question is more... Well, yeah, maybe I'm not understanding your question.
00:17:31 · Vincent Liu, Chris Paxton, Ademi Adeniji
[00:17:31] S1: I was gonna... Yeah. So back to like, how does it open the thing? In our our ablation... Or I guess baselines, we sort of test against a method where we just predict, like, a six point trajectory, like, and then just execute it. And so it's still able to do that. Right? So it's not so much like, oh, if you can't track things, how do you open the oven? It's just like, I guess, maybe your question is more... Well, yeah, maybe I'm not understanding your question. [00:17:56] S3: Yes. Yeah. So so the... That's... In that case, opening the oven for... Is is open loop. You wouldn't, like, be able to respond if the oven got jammed, I guess. Is that... That's what I would [00:18:04] S2: imagine then? [00:18:05] S1: Okay. So basically, what the model sees is like a snapshot, like summarized with like three d points, and then it goes and knows where... So it's kind of like memorized where it is. Right? [00:18:15] S2: Yeah. Makes sense. [00:18:15] S1: So if the object is moving, you'd probably be in trouble. [00:18:17] S3: Sure. Yeah. So okay.
00:18:19 · Vincent Liu
[00:18:19] S1: Yeah. Okay. Yeah. So going forward then, so I just described how we get the object points. And so the other path of the equation is how do you get the end effector points or basically your index thumb finger from these recordings. So actually, shortly after we, I think, released EgoZero, this step of computing 21 hand key points in camera frame, like, it was released as an update to the Aria glasses. And we took a look at the outputs, and they look pretty similar. So maybe they're doing something similar here. But the idea is that because the Aria MPS service only outputs six DOF... Accurate six DOF transformations of, you know, your palms and wrist in camera frame, you don't have, you know, the fine grained information of how your... Where where your fingers are. Right? And so when we sort of look at monocular hand estimation models like HaMeR, we see that, you know, we we observed that, like, the hand shapes look pretty correct. It's just that if you read out the pose, the predicted pose of where the hand is in camera frame from HaMeR, it's very incorrect. Right? Because you're sort of estimating the three d from a single image, which is, like, kind of a difficult problem. And so what we thought is, okay, is there a way we can borrow the local hand information from HaMeR or these hand estimation models and, like, piece it together with accurate 6DOF, like, hand, like, pose in camera frame? And so this is sort of what we did. We, like, kind of took a look at where the palm is with respect to, like, the the morphology of the hand and the key points. We kind of compute a centroid and then align things, and then you add them together. So, like, conceptually, I like to think of this as you're, like, kind of projecting out this incorrect information predicted by HaMeR, and, like, you're sort of, like, pasting in this accurate information by Aria.
00:20:10 · Michael Cho, Vincent Liu
[00:20:10] S0: But you're saying that the... It sounds like Aria... I don't know. Is it like a MPS plus that basically have this... Maybe being able able to do the hands well as well is is what you're saying? [00:20:20] S1: Yeah. Yeah. So, like, I mean, literally, I think right before we open source the the... Our our code, they released the update to their SDK. And you literally can just read this out for free. You don't have to run HaMeR on your GPU and all this stuff. Yeah. But, like, the the the sort of, like, qualitative assessment of the two outputs look pretty similar. So, you know, I would imagine if they have, like, access to the whole hardware system, you know, the stereo cameras and SLAM and everything, it probably wouldn't be too hard to either like, do what we did or have a version of HaMeR that is more accurate because it's like, with stereo and camera information. Yeah, that's kind of like a general theme we've felt like post this project is, you know, a lot of these hoops that we had to jump through are pretty easily amenable. Right? Like, Aria now offers 21 hand key points for free. All you need to do for... Instead of triangulating over your SLAM trajectory, if you want a fully closed loop setting, you can just attach a LIDAR to it. Right? So it's like these solutions are sort of not that fundamental in a way. It's just sort of, like, to kinda prove this with the current hardware we had, we had to sort of be clever about. [00:21:26] S0: Makes sense. And
00:21:26 · Michael Cho, Vincent Liu, Chris Paxton
[00:21:26] S0: Makes sense. And [00:21:28] S1: I guess the the sort of last, like, interesting, maybe not too surprising result is that our three d representations are very interpretable. You know, it's, again, it's like a very linear mapping space from states to actions. And in training, we sample random three d transformations and then apply them to entire demonstrations because everything's in three d. And so here on this plot, the blue area is the distribution of object points, egocentric object points in the training data. And then the gray area is sort of like this distribution that is augmented after sampling all these transformations. And you'll see that the model starts to... The model does very well as long as the inference time points are within this augmented training this... Training distribution. And as soon as you sort of get to the edge of this... Of its, like, support volume, it begins to fail, which sort of makes sense. And it also demonstrates that these augmentations are pretty important to be able to generalize from different distances, heights, you know, camera settings. So that's pretty much it. The longer version of our, I guess, our assist workshop talk. And we can go to the website too now if we wanna check out some more videos or details. Sure. [00:22:41] S0: Yeah. Yeah. That's what [00:22:42] S3: what kind of network architecture is this? Out of curiosity, is this one of those, like, I don't know, three d specialized models, or is this a transformer that you threw all this data into? How are you... How do you make it work?
00:22:42 · Chris Paxton, Vincent Liu, Michael Cho, Ademi Adeniji
[00:22:42] S3: what kind of network architecture is this? Out of curiosity, is this one of those, like, I don't know, three d specialized models, or is this a transformer that you threw all this data into? How are you... How do you make it work? [00:22:54] S1: Just a small, like, Baku transformer. Okay. Literally, like, the the number of parameters is, like, a low single digit millions. You don't really need much more to learn a very simple problem. [00:23:07] S0: Yep. Yeah. [00:23:08] S3: I mean, three d... I feel like three d models are usually pretty efficient... Data efficient. Right? [00:23:12] S1: So kind of Do you mean like like point nets and stuff? [00:23:16] S3: No. Well, I meant whatever. Anything. [00:23:18] S2: Okay. [00:23:18] S3: Using three d information. But, yes, those two [00:23:20] S0: are I see. [00:23:20] S1: I see. Demi, is there anything you wanna add? [00:23:25] S2: Yeah, I think that was pretty comprehensive. I think we should maybe look at some of the in the wild to inference time transfer results. Like I think there's some cool videos where you like fold a towel in your bedroom and it transfers.