00:11:32 · Chris Paxton, Vincent Liu, Michael Cho
[00:11:32] S3: Yeah. So how do we collect data? [00:11:34] S1: Oh, yeah. Oh, yeah. So the... Exactly. So now we sort of get to the meat of the contribution of the paper, which is how do we collect three d data from smart glasses, specifically the Aria glasses. So the Aria glasses have very good SLAM. You know, it has two visual inertial cameras on the side and RGB fisheye on the left side. And it's software package. It's offline like a MPS service, basically. Comes with accurate six off hand poses. And, like, these are both really accurate, like, as we sort of played around with it. However, the downside to the Aria glasses, at least the Gen one, I'm... I've heard the Gen two is, you know, is slightly more amenable to some of these gen one drawbacks. But at least the gen one glasses have minimal FOV overlap between these SLAM cameras and also no, like, ground truth way of getting depth. And so, you know, the two way... The the two, like, most standard ways to get three d information is with a triangulation or, like, with a LIDAR or something, and the Aria glasses did not provide either. And so this is sort of, like, the main problem we had to overcome. [00:12:37] S0: I'm curious. I don't know whether that that they have ever publicized it. Like, roughly how much do these cost, Like, if you were to get more of these... Are these, like, regular, like, smart glasses pricing, like, couple $100? Or is it, like, I don't know whether you're able to share any of this or whether [00:12:54] S1: they I personally have no idea. [00:12:56] S0: Oh, okay.
00:12:56 · Michael Cho, Ademi Adeniji, Vincent Liu, Chris Paxton
[00:12:56] S0: Oh, okay. [00:12:58] S2: Yeah. I mean, the Aria the Aria glasses are sort of like the research, like the r and d version of like the consumer product, like the meta, like Ray Bans and now, like, I think they have Oakleys. Yep. They're they're much more sensor complete than the consumer grade version. So I imagine much more expensive, but I don't have a specific number. [00:13:17] S1: Yeah. I would imagine they they feel like they'd be expensive though, to be honest. [00:13:22] S0: Yeah. I'm surprised, that the Ray Bans are are, like, like, a regular pair of Ray Bans, or maybe Ray Bans just overpriced in in the first place. [00:13:31] S2: I think [00:13:35] S1: Yeah. They might just be slightly expensive in the... I don't know. Like, $1.50, I think. Just the normal ones. [00:13:40] S0: Anyway I mean, sorry. Sorry. [00:13:41] S3: Quest three is $300. Right? So that's like an upper bound, I would think. But I don't know. I feel like [00:13:47] S0: Oh, you think it's upper bound? I don't know. My my guess is that this is [00:13:51] S3: Depends. [00:13:53] S0: Because if if you are trying to squeeze into... Yeah. I don't know. It's it's... Yeah. Think the [00:13:58] S1: mini set [00:13:58] S2: might might drive the price up. Yeah. [00:14:02] S3: Yeah. But the question is the compute in the screen. So this is also a tangent we don't need to go with. [00:14:06] S1: Yeah. Should get the Aria guys on the podcast next time.
00:14:06 · Vincent Liu, Michael Cho
[00:14:06] S1: Yeah. Should get the Aria guys on the podcast next time. [00:14:10] S0: Yeah. Yeah. Yeah. Because I'm I'm just thinking, I mean, ultimately, all this thing, you know, to your point, your your earliest slice is to scale this. Right? And if, you know, Aria glasses or form factor like Aria glasses, which is, like, slight... Slightly more fancy, less smart glasses that can do SLAM well, then, you know, the cost become a a important bottleneck. But, anyway, we don't have to go there for today. Yeah. I'm just curious if you know. Alright. Cool. Cool. Sorry.
00:14:34 · Vincent Liu
[00:14:34] S1: Yeah. Okay. So sort of proceeding with the method then. Like I said, like, conceptually, this is inspired a lot by point policy. So, you know, we start out with the same sort of procedure where you label set of two d object points on the training image, like this is like the expert image, and then you map these in two d onto the first frame and throughout the demonstration for each example in your training set. But in the... With the Aria glasses in the egocentric setting, you do not have two stationary cameras that are calibrated to triangulate these points. And so we then sort of make a couple assumptions to get this to work. One is that the SLAM extrinsics are very good, which they are on these on on these glasses. And two, by assuming that the object is stationary, you're able to sort of triangulate over time, so, like, over the camera's trajectory. So it's like, you know, point policy is triangulating two cameras, fixed point in time. If you just imagine, like, as you move your head over, like, one second, you know, you you have, like, very accurate camera information, and you can localize the object. And so this is sort of how we get around can't triangulate stereo... With stereo per frame. You don't have, like, LiDAR or really ground truth depth measurements. But if you make this assumption and leverage, like, a good camera information, you're able to localize the object well. It's just that we can't actually track the object, right, because we're assuming that it's stationary and triangling over time. So the models we train are like, in a way, like open loop with respect to the object state, but we're able to, like, you know, track the fingertip movements in every frame. So it's like... That's why we sort of say closed loop. And also that... Yeah.
00:16:11 · Chris Paxton, Ademi Adeniji, Vincent Liu
[00:16:11] S3: That video earlier of it opening, like, a oven or something, though. Right? Or some some kinda... Yeah. So how does that work? Because that is not stationary. [00:16:20] S2: Yeah. [00:16:21] S1: Yes. Go ahead, Demi. If you go. [00:16:23] S2: So so, like, I think... So funnily enough, because you have proprioception closed loop and your model is ingesting proprioceptive input closed loop, you actually can solve a lot of these tasks without per frame object tracking. I will add also though that one, this approach is sort of, I think, necessarily limiting. So you can, for instance, the slam readings are very high, read out at very high FPS. So if there's sufficient camera movement, you can just triangulate over windows of time as opposed to over the entire trajectory. And we use RANSAC in order to actually do the PNP mapping solve for the actual pose of the object. But you know, we have very, very high precision there after triangulation. And so you could definitely shorten the window over which you do this triangulation and have something that approximates something more closely. Now, of course, the best solution would just be to have some ground truth depth estimation via LiDAR or just have two cameras. You could do stereo triangulation. Right.
00:16:23 · Ademi Adeniji, Vincent Liu
[00:16:23] S2: So so, like, I think... So funnily enough, because you have proprioception closed loop and your model is ingesting proprioceptive input closed loop, you actually can solve a lot of these tasks without per frame object tracking. I will add also though that one, this approach is sort of, I think, necessarily limiting. So you can, for instance, the slam readings are very high, read out at very high FPS. So if there's sufficient camera movement, you can just triangulate over windows of time as opposed to over the entire trajectory. And we use RANSAC in order to actually do the PNP mapping solve for the actual pose of the object. But you know, we have very, very high precision there after triangulation. And so you could definitely shorten the window over which you do this triangulation and have something that approximates something more closely. Now, of course, the best solution would just be to have some ground truth depth estimation via LiDAR or just have two cameras. You could do stereo triangulation. Right. [00:17:31] S1: I was gonna... Yeah. So back to like, how does it open the thing? In our our ablation... Or I guess baselines, we sort of test against a method where we just predict, like, a six point trajectory, like, and then just execute it. And so it's still able to do that. Right? So it's not so much like, oh, if you can't track things, how do you open the oven? It's just like, I guess, maybe your question is more... Well, yeah, maybe I'm not understanding your question.