00:09:54 · Chris Paxton, Ademi Adeniji, Michael Cho
[00:09:54] S3: Yeah. That's fair. But I I mean, I guess, I don't wanna derail this. Like... But even when you're doing kinematic retargeting to, like, the unit three g one, like, that's that's, like, toddler size or something like this. People do this all the time for teleoperation, but, like, it's so much shorter than, me that I imagine there'd be a gap, but maybe not. That's a... It's an interesting question. Yeah. [00:10:14] S2: Yeah. It's... I guess, so I'll just add one more point. Yeah. I think that's a great point. Like, scale does matter in terms of, like, literally the size. I think because so much of what we're looking at is dextrous manipulation, so long as your full body, whole body control works and your whole body IK is working, as long as the hands, I think, are the right size, you sort of are already almost there. We actually tried Okay. Getting this working with the Allegro hand. And the kinematic retargeting there is just super, super complicated because, you know Okay. You can't do it joint space wise because, like, the forward kinematics are completely different. And, yeah, like, sort of that's like a a a... I think an exercise in futility. So, we moved to the gripper and now hoping, like, hands are more one to one with human hands moving forward in terms of size. [00:11:00] S0: Yeah. I
00:11:00 · Michael Cho, Vincent Liu, Ademi Adeniji, Chris Paxton
[00:11:00] S0: Yeah. I [00:11:00] S1: I also think, like, there's a benefit here for representing things in three d in egocentric frame. Right? So you're basically... What you're what you're asking the policy to do is sort of predict movements of, like, of three d points in a three d space. You know? And then in inference time, you sort of, like, can arbitrarily, as we show, like, sort of put this camera somewhere. And then if the policy is well trained, like like, you know, the... It would just be sort of what a Demi is saying, like a full body retargeting there and hopefully, like, relatively morphology agnostic. [00:11:29] S2: Makes sense. [00:11:30] S1: K. Should I continue here? [00:11:32] S3: Yeah. So how do we collect data?
00:11:32 · Chris Paxton, Vincent Liu, Michael Cho
[00:11:32] S3: Yeah. So how do we collect data? [00:11:34] S1: Oh, yeah. Oh, yeah. So the... Exactly. So now we sort of get to the meat of the contribution of the paper, which is how do we collect three d data from smart glasses, specifically the Aria glasses. So the Aria glasses have very good SLAM. You know, it has two visual inertial cameras on the side and RGB fisheye on the left side. And it's software package. It's offline like a MPS service, basically. Comes with accurate six off hand poses. And, like, these are both really accurate, like, as we sort of played around with it. However, the downside to the Aria glasses, at least the Gen one, I'm... I've heard the Gen two is, you know, is slightly more amenable to some of these gen one drawbacks. But at least the gen one glasses have minimal FOV overlap between these SLAM cameras and also no, like, ground truth way of getting depth. And so, you know, the two way... The the two, like, most standard ways to get three d information is with a triangulation or, like, with a LIDAR or something, and the Aria glasses did not provide either. And so this is sort of, like, the main problem we had to overcome. [00:12:37] S0: I'm curious. I don't know whether that that they have ever publicized it. Like, roughly how much do these cost, Like, if you were to get more of these... Are these, like, regular, like, smart glasses pricing, like, couple $100? Or is it, like, I don't know whether you're able to share any of this or whether [00:12:54] S1: they I personally have no idea. [00:12:56] S0: Oh, okay.
00:12:56 · Michael Cho, Ademi Adeniji, Vincent Liu, Chris Paxton
[00:12:56] S0: Oh, okay. [00:12:58] S2: Yeah. I mean, the Aria the Aria glasses are sort of like the research, like the r and d version of like the consumer product, like the meta, like Ray Bans and now, like, I think they have Oakleys. Yep. They're they're much more sensor complete than the consumer grade version. So I imagine much more expensive, but I don't have a specific number. [00:13:17] S1: Yeah. I would imagine they they feel like they'd be expensive though, to be honest. [00:13:22] S0: Yeah. I'm surprised, that the Ray Bans are are, like, like, a regular pair of Ray Bans, or maybe Ray Bans just overpriced in in the first place. [00:13:31] S2: I think [00:13:35] S1: Yeah. They might just be slightly expensive in the... I don't know. Like, $1.50, I think. Just the normal ones. [00:13:40] S0: Anyway I mean, sorry. Sorry. [00:13:41] S3: Quest three is $300. Right? So that's like an upper bound, I would think. But I don't know. I feel like [00:13:47] S0: Oh, you think it's upper bound? I don't know. My my guess is that this is [00:13:51] S3: Depends. [00:13:53] S0: Because if if you are trying to squeeze into... Yeah. I don't know. It's it's... Yeah. Think the [00:13:58] S1: mini set [00:13:58] S2: might might drive the price up. Yeah. [00:14:02] S3: Yeah. But the question is the compute in the screen. So this is also a tangent we don't need to go with. [00:14:06] S1: Yeah. Should get the Aria guys on the podcast next time.
00:14:06 · Vincent Liu, Michael Cho
[00:14:06] S1: Yeah. Should get the Aria guys on the podcast next time. [00:14:10] S0: Yeah. Yeah. Yeah. Because I'm I'm just thinking, I mean, ultimately, all this thing, you know, to your point, your your earliest slice is to scale this. Right? And if, you know, Aria glasses or form factor like Aria glasses, which is, like, slight... Slightly more fancy, less smart glasses that can do SLAM well, then, you know, the cost become a a important bottleneck. But, anyway, we don't have to go there for today. Yeah. I'm just curious if you know. Alright. Cool. Cool. Sorry.
00:14:34 · Vincent Liu
[00:14:34] S1: Yeah. Okay. So sort of proceeding with the method then. Like I said, like, conceptually, this is inspired a lot by point policy. So, you know, we start out with the same sort of procedure where you label set of two d object points on the training image, like this is like the expert image, and then you map these in two d onto the first frame and throughout the demonstration for each example in your training set. But in the... With the Aria glasses in the egocentric setting, you do not have two stationary cameras that are calibrated to triangulate these points. And so we then sort of make a couple assumptions to get this to work. One is that the SLAM extrinsics are very good, which they are on these on on these glasses. And two, by assuming that the object is stationary, you're able to sort of triangulate over time, so, like, over the camera's trajectory. So it's like, you know, point policy is triangulating two cameras, fixed point in time. If you just imagine, like, as you move your head over, like, one second, you know, you you have, like, very accurate camera information, and you can localize the object. And so this is sort of how we get around can't triangulate stereo... With stereo per frame. You don't have, like, LiDAR or really ground truth depth measurements. But if you make this assumption and leverage, like, a good camera information, you're able to localize the object well. It's just that we can't actually track the object, right, because we're assuming that it's stationary and triangling over time. So the models we train are like, in a way, like open loop with respect to the object state, but we're able to, like, you know, track the fingertip movements in every frame. So it's like... That's why we sort of say closed loop. And also that... Yeah.