00:18:19 · Vincent Liu
[00:18:19] S1: Yeah. Okay. Yeah. So going forward then, so I just described how we get the object points. And so the other path of the equation is how do you get the end effector points or basically your index thumb finger from these recordings. So actually, shortly after we, I think, released EgoZero, this step of computing 21 hand key points in camera frame, like, it was released as an update to the Aria glasses. And we took a look at the outputs, and they look pretty similar. So maybe they're doing something similar here. But the idea is that because the Aria MPS service only outputs six DOF... Accurate six DOF transformations of, you know, your palms and wrist in camera frame, you don't have, you know, the fine grained information of how your... Where where your fingers are. Right? And so when we sort of look at monocular hand estimation models like HaMeR, we see that, you know, we we observed that, like, the hand shapes look pretty correct. It's just that if you read out the pose, the predicted pose of where the hand is in camera frame from HaMeR, it's very incorrect. Right? Because you're sort of estimating the three d from a single image, which is, like, kind of a difficult problem. And so what we thought is, okay, is there a way we can borrow the local hand information from HaMeR or these hand estimation models and, like, piece it together with accurate 6DOF, like, hand, like, pose in camera frame? And so this is sort of what we did. We, like, kind of took a look at where the palm is with respect to, like, the the morphology of the hand and the key points. We kind of compute a centroid and then align things, and then you add them together. So, like, conceptually, I like to think of this as you're, like, kind of projecting out this incorrect information predicted by HaMeR, and, like, you're sort of, like, pasting in this accurate information by Aria.
00:20:10 · Michael Cho, Vincent Liu
[00:20:10] S0: But you're saying that the... It sounds like Aria... I don't know. Is it like a MPS plus that basically have this... Maybe being able able to do the hands well as well is is what you're saying? [00:20:20] S1: Yeah. Yeah. So, like, I mean, literally, I think right before we open source the the... Our our code, they released the update to their SDK. And you literally can just read this out for free. You don't have to run HaMeR on your GPU and all this stuff. Yeah. But, like, the the the sort of, like, qualitative assessment of the two outputs look pretty similar. So, you know, I would imagine if they have, like, access to the whole hardware system, you know, the stereo cameras and SLAM and everything, it probably wouldn't be too hard to either like, do what we did or have a version of HaMeR that is more accurate because it's like, with stereo and camera information. Yeah, that's kind of like a general theme we've felt like post this project is, you know, a lot of these hoops that we had to jump through are pretty easily amenable. Right? Like, Aria now offers 21 hand key points for free. All you need to do for... Instead of triangulating over your SLAM trajectory, if you want a fully closed loop setting, you can just attach a LIDAR to it. Right? So it's like these solutions are sort of not that fundamental in a way. It's just sort of, like, to kinda prove this with the current hardware we had, we had to sort of be clever about. [00:21:26] S0: Makes sense. And
00:21:26 · Michael Cho, Vincent Liu, Chris Paxton
[00:21:26] S0: Makes sense. And [00:21:28] S1: I guess the the sort of last, like, interesting, maybe not too surprising result is that our three d representations are very interpretable. You know, it's, again, it's like a very linear mapping space from states to actions. And in training, we sample random three d transformations and then apply them to entire demonstrations because everything's in three d. And so here on this plot, the blue area is the distribution of object points, egocentric object points in the training data. And then the gray area is sort of like this distribution that is augmented after sampling all these transformations. And you'll see that the model starts to... The model does very well as long as the inference time points are within this augmented training this... Training distribution. And as soon as you sort of get to the edge of this... Of its, like, support volume, it begins to fail, which sort of makes sense. And it also demonstrates that these augmentations are pretty important to be able to generalize from different distances, heights, you know, camera settings. So that's pretty much it. The longer version of our, I guess, our assist workshop talk. And we can go to the website too now if we wanna check out some more videos or details. Sure. [00:22:41] S0: Yeah. Yeah. That's what [00:22:42] S3: what kind of network architecture is this? Out of curiosity, is this one of those, like, I don't know, three d specialized models, or is this a transformer that you threw all this data into? How are you... How do you make it work?
00:22:42 · Chris Paxton, Vincent Liu, Michael Cho, Ademi Adeniji
[00:22:42] S3: what kind of network architecture is this? Out of curiosity, is this one of those, like, I don't know, three d specialized models, or is this a transformer that you threw all this data into? How are you... How do you make it work? [00:22:54] S1: Just a small, like, Baku transformer. Okay. Literally, like, the the number of parameters is, like, a low single digit millions. You don't really need much more to learn a very simple problem. [00:23:07] S0: Yep. Yeah. [00:23:08] S3: I mean, three d... I feel like three d models are usually pretty efficient... Data efficient. Right? [00:23:12] S1: So kind of Do you mean like like point nets and stuff? [00:23:16] S3: No. Well, I meant whatever. Anything. [00:23:18] S2: Okay. [00:23:18] S3: Using three d information. But, yes, those two [00:23:20] S0: are I see. [00:23:20] S1: I see. Demi, is there anything you wanna add? [00:23:25] S2: Yeah, I think that was pretty comprehensive. I think we should maybe look at some of the in the wild to inference time transfer results. Like I think there's some cool videos where you like fold a towel in your bedroom and it transfers.
00:23:25 · Ademi Adeniji, Vincent Liu, Chris Paxton
[00:23:25] S2: Yeah, I think that was pretty comprehensive. I think we should maybe look at some of the in the wild to inference time transfer results. Like I think there's some cool videos where you like fold a towel in your bedroom and it transfers. [00:23:37] S1: Yes. Let's see. We can probably find... You guys can still see my screen, right? I guess these aren't in my... Well, maybe this... Yeah, this one is. Yeah. So the nice part about this is like with just twenty minutes of data basically, because everything is super summarized, like like the... If if you have, like, this general, like, learning stack pipeline, however you wanna call it, where you go from dense video to just actions, in a way, we're, like, sort of move... We're shrinking the portion of policy, like, that policy learning is responsible for. Right? We're sort of, like, extract... We're expanding a lot of compute and, like, knowledge contained in, like, grounded vision models to simplify the learning problem. So, like, that kind of comes with some trade offs. But one of the benefits is that, you know, with just twenty minutes of data across four four different... Very different visual settings here, none of these are even in a lab. These these these three here. These sort of point information is robust across all these settings, which is why, you know, the robot's able to fold the towel as well. [00:24:38] S3: Yeah. What's the success rate on this, for example?
00:24:38 · Chris Paxton, Vincent Liu
[00:24:38] S3: Yeah. What's the success rate on this, for example? [00:24:41] S1: Everything was around... Let's see. What is it? 60 or 70%? I think 70 on average. And it definitely we noticed that, like, the performance of this of this sort of framework was more of a function of how good the three d measurements are, as opposed to how much data you have. And, like, it was relatively easy to get general behavior because you're able to do these augmentations, you know, to expand your domain input, basically. And also, like, these demonstrations aren't too complicated. It's not like there's many different ways to fold the towel that we've demonstrated. So we really saw that, you know, if you have, for example, HaMeR plus ARIA hand detection, about, like, centimeter of error, whatever. And then so you're therefore limited to sort of that level of precision. [00:25:33] S3: Yeah. Makes sense. [00:25:34] S1: Yeah. And then I guess another nice property, and this is what point policy shows, is that because, again, you're using points, which is just more like spatial geometric information, you can swap out the literal, like, training object, and you're able to, you know, fold a different color towel or, like, you know, just change the color, like, kind of variations of of your training object. [00:25:57] S3: Would it work with distractors? Like, you put another towel down, what would happen?