00:12:56 · Michael Cho, Ademi Adeniji, Vincent Liu, Chris Paxton
[00:12:56] S0: Oh, okay. [00:12:58] S2: Yeah. I mean, the Aria the Aria glasses are sort of like the research, like the r and d version of like the consumer product, like the meta, like Ray Bans and now, like, I think they have Oakleys. Yep. They're they're much more sensor complete than the consumer grade version. So I imagine much more expensive, but I don't have a specific number. [00:13:17] S1: Yeah. I would imagine they they feel like they'd be expensive though, to be honest. [00:13:22] S0: Yeah. I'm surprised, that the Ray Bans are are, like, like, a regular pair of Ray Bans, or maybe Ray Bans just overpriced in in the first place. [00:13:31] S2: I think [00:13:35] S1: Yeah. They might just be slightly expensive in the... I don't know. Like, $1.50, I think. Just the normal ones. [00:13:40] S0: Anyway I mean, sorry. Sorry. [00:13:41] S3: Quest three is $300. Right? So that's like an upper bound, I would think. But I don't know. I feel like [00:13:47] S0: Oh, you think it's upper bound? I don't know. My my guess is that this is [00:13:51] S3: Depends. [00:13:53] S0: Because if if you are trying to squeeze into... Yeah. I don't know. It's it's... Yeah. Think the [00:13:58] S1: mini set [00:13:58] S2: might might drive the price up. Yeah. [00:14:02] S3: Yeah. But the question is the compute in the screen. So this is also a tangent we don't need to go with. [00:14:06] S1: Yeah. Should get the Aria guys on the podcast next time.
00:14:06 · Vincent Liu, Michael Cho
[00:14:06] S1: Yeah. Should get the Aria guys on the podcast next time. [00:14:10] S0: Yeah. Yeah. Yeah. Because I'm I'm just thinking, I mean, ultimately, all this thing, you know, to your point, your your earliest slice is to scale this. Right? And if, you know, Aria glasses or form factor like Aria glasses, which is, like, slight... Slightly more fancy, less smart glasses that can do SLAM well, then, you know, the cost become a a important bottleneck. But, anyway, we don't have to go there for today. Yeah. I'm just curious if you know. Alright. Cool. Cool. Sorry.
00:14:34 · Vincent Liu
[00:14:34] S1: Yeah. Okay. So sort of proceeding with the method then. Like I said, like, conceptually, this is inspired a lot by point policy. So, you know, we start out with the same sort of procedure where you label set of two d object points on the training image, like this is like the expert image, and then you map these in two d onto the first frame and throughout the demonstration for each example in your training set. But in the... With the Aria glasses in the egocentric setting, you do not have two stationary cameras that are calibrated to triangulate these points. And so we then sort of make a couple assumptions to get this to work. One is that the SLAM extrinsics are very good, which they are on these on on these glasses. And two, by assuming that the object is stationary, you're able to sort of triangulate over time, so, like, over the camera's trajectory. So it's like, you know, point policy is triangulating two cameras, fixed point in time. If you just imagine, like, as you move your head over, like, one second, you know, you you have, like, very accurate camera information, and you can localize the object. And so this is sort of how we get around can't triangulate stereo... With stereo per frame. You don't have, like, LiDAR or really ground truth depth measurements. But if you make this assumption and leverage, like, a good camera information, you're able to localize the object well. It's just that we can't actually track the object, right, because we're assuming that it's stationary and triangling over time. So the models we train are like, in a way, like open loop with respect to the object state, but we're able to, like, you know, track the fingertip movements in every frame. So it's like... That's why we sort of say closed loop. And also that... Yeah.
00:16:11 · Chris Paxton, Ademi Adeniji, Vincent Liu
[00:16:11] S3: That video earlier of it opening, like, a oven or something, though. Right? Or some some kinda... Yeah. So how does that work? Because that is not stationary. [00:16:20] S2: Yeah. [00:16:21] S1: Yes. Go ahead, Demi. If you go. [00:16:23] S2: So so, like, I think... So funnily enough, because you have proprioception closed loop and your model is ingesting proprioceptive input closed loop, you actually can solve a lot of these tasks without per frame object tracking. I will add also though that one, this approach is sort of, I think, necessarily limiting. So you can, for instance, the slam readings are very high, read out at very high FPS. So if there's sufficient camera movement, you can just triangulate over windows of time as opposed to over the entire trajectory. And we use RANSAC in order to actually do the PNP mapping solve for the actual pose of the object. But you know, we have very, very high precision there after triangulation. And so you could definitely shorten the window over which you do this triangulation and have something that approximates something more closely. Now, of course, the best solution would just be to have some ground truth depth estimation via LiDAR or just have two cameras. You could do stereo triangulation. Right.
00:16:23 · Ademi Adeniji, Vincent Liu
[00:16:23] S2: So so, like, I think... So funnily enough, because you have proprioception closed loop and your model is ingesting proprioceptive input closed loop, you actually can solve a lot of these tasks without per frame object tracking. I will add also though that one, this approach is sort of, I think, necessarily limiting. So you can, for instance, the slam readings are very high, read out at very high FPS. So if there's sufficient camera movement, you can just triangulate over windows of time as opposed to over the entire trajectory. And we use RANSAC in order to actually do the PNP mapping solve for the actual pose of the object. But you know, we have very, very high precision there after triangulation. And so you could definitely shorten the window over which you do this triangulation and have something that approximates something more closely. Now, of course, the best solution would just be to have some ground truth depth estimation via LiDAR or just have two cameras. You could do stereo triangulation. Right. [00:17:31] S1: I was gonna... Yeah. So back to like, how does it open the thing? In our our ablation... Or I guess baselines, we sort of test against a method where we just predict, like, a six point trajectory, like, and then just execute it. And so it's still able to do that. Right? So it's not so much like, oh, if you can't track things, how do you open the oven? It's just like, I guess, maybe your question is more... Well, yeah, maybe I'm not understanding your question.
00:17:31 · Vincent Liu, Chris Paxton, Ademi Adeniji
[00:17:31] S1: I was gonna... Yeah. So back to like, how does it open the thing? In our our ablation... Or I guess baselines, we sort of test against a method where we just predict, like, a six point trajectory, like, and then just execute it. And so it's still able to do that. Right? So it's not so much like, oh, if you can't track things, how do you open the oven? It's just like, I guess, maybe your question is more... Well, yeah, maybe I'm not understanding your question. [00:17:56] S3: Yes. Yeah. So so the... That's... In that case, opening the oven for... Is is open loop. You wouldn't, like, be able to respond if the oven got jammed, I guess. Is that... That's what I would [00:18:04] S2: imagine then? [00:18:05] S1: Okay. So basically, what the model sees is like a snapshot, like summarized with like three d points, and then it goes and knows where... So it's kind of like memorized where it is. Right? [00:18:15] S2: Yeah. Makes sense. [00:18:15] S1: So if the object is moving, you'd probably be in trouble. [00:18:17] S3: Sure. Yeah. So okay.