00:35:49 · Chris Paxton, Vincent Liu
[00:35:49] S3: So just because Adam mentioned it, do you... So do you think that this is a... You're gonna be able to leverage, like, YouTube video data at some point in the future, or do you think this this kind of... That is that is, like, an obvious strength of this approach. Right? Like, this is much closer to, like, what you would get from just third party data. Like, how much do you really need the the Aria SLAM kinda toolkit here? [00:36:11] S1: I'd say with the the second part, at least, with the Aria SLAM, this is... Like, the SLAM is also used for... In addition to how we use it to get all these point informations, it's also used for just projecting things in a fixed frame. And I guess... And so, like, that is probably pretty important for inference, right, where your robot's not really moving, or at least it starts out, like, in a certain spot. And I guess that question is more like... There's, like, a broader question maybe there, which is, like, how important is this camera information? And I feel like that's the downside of YouTube data is you kinda just have unknown camera information. So if you really are trying to extract accurate BD, like, proprioception, that will be very hard to do. [00:36:55] S3: Right. Well, certainly, if there's, like, cuts or whatever, you can't pull from that from that video. But, like, I was wondering if, like, structure for motion, you can get accurate enough camera pose, you'd be able to do this. Or if that's just, like, you don't think that's there, that technology is too far off. Like Yes,
00:36:55 · Chris Paxton, Ademi Adeniji
[00:36:55] S3: Right. Well, certainly, if there's, like, cuts or whatever, you can't pull from that from that video. But, like, I was wondering if, like, structure for motion, you can get accurate enough camera pose, you'd be able to do this. Or if that's just, like, you don't think that's there, that technology is too far off. Like Yes, [00:37:09] S2: we Yeah. I mean, my my view is we've tried to get structure for motion working for a really long time, and people have spent a lot of effort trying to get really precise in six. But I think there's been cool work actually. That's leverage structure for mentioned, like, ideas. Like, I think there's some stuff out of CMU and a few other places that try and do something actually quite similar to what we're doing, but using this, again, I'll use it dirtier data and trying to impute camera information. Yeah, I just think the signal isn't there generally. And really, we should just invest in scaling the right signal. And I think that underscores the importance of having something very frictionless such that you actually can get to that scale, but with the signal in the data and in in your data annotations, importantly. So that... That's my view anyway.
00:38:04 · Vincent Liu, Michael Cho, Ademi Adeniji
[00:38:04] S1: Yeah. I'll add something concrete to that. So in the process of trying to get depth, let's just say depth estimation solution working, we've tried a lot of things. And one of them was actually, like, MASt3R-SLAM. I'm not sure if you've heard about it or if there's been a lot of work here. But, like, those... The the videos they would post would be pretty compelling, like, where you can just take, like, a walkthrough in a video game and would map out the whole thing. Right? And so we actually tried this. We ran MASt3R-SLAM and tried to get depth from, like, the point clouds that it will produce, and we just weren't able to get enough, like, precision there. Like, we we we spend, like, a few weeks trying this. And so, you know, conceptually, I'm like, you know, also, like, I... My background before time at NYU was at a three d startup for a couple years. And so I'm familiar with, like, these all these three d approaches. And so, you know, I would think that, you know, you can just build, like, a, like, a three d map from from random videos, but, like, in practice, we've had a hard time doing that. [00:39:02] S0: Yep. I was wondering if... Do do you think it's possible that because in this case, like, the the view of you coming from the the two cameras on on Aria, very wide view, do you think it matters? Because if... [00:39:14] S2: I I
00:39:14 · Ademi Adeniji, Michael Cho, Vincent Liu
[00:39:14] S2: I I [00:39:15] S0: don't know specifically how how is master... I mean, a bunch of these, like, camera pose. Because oftentimes, their their viewer view is... The training pipeline, they use narrower, let's say, you know, 90 degrees. I'm just eyeballing this. This look pretty wide. I... I'm guessing it's, like, close to... I don't I don't know. Yeah. A 160 degrees, maybe? [00:39:32] S1: Yeah. They are pretty wide. MASt3R-SLAM specifically had an option for you to input the camera intrinsics, which we did. And like for Aria, they provide these. [00:39:41] S0: I see.
00:39:41 · Michael Cho, Vincent Liu
[00:39:41] S0: I see. [00:39:41] S1: And I guess, you know, you have like your KD, whatever distortion coefficients, and those default to zero, which in the fisheye, you have high distortion. So we actually undistort the image, but even if you undistort these fisheye images, they don't look very, like, pinhole y. You know? They still look pretty warped. And so I feel like this might have been an issue. This might have been, like, a reason why, you know, when we try, you know, using, like, a binocular depth estimation model and we we ground it in, like, a ArUco tags in the scene or even master slam here, maybe these are reasons why, like, those methods don't work so well because the camera is, like, very not... It's, like, very distorted. Right? But but also, like, I guess, another aspect of the fisheye lens is that it's actually very nice because it minimizes occlusion. So Mhmm. You know, like, as I'm grabbing the fruit here, you're pretty... It's pretty much not occluded until you actually cover it with your hand. And, again, I mentioned, like, Ademina had tried setup close... Similar to DexWild where we used the iPhone, a chest mounted iPhone, and also because it's lower, but also because it's a linear kind of pinhole. One of the reasons we think it didn't work well is because as soon as you reached out to it, you would occlude the entire scene. And so, like, the fisheye is kinda nice in this way, but also maybe it lends itself to, like, a difficult time extracting depth.
00:40:56 · Michael Cho, Vincent Liu
[00:40:56] S0: Yeah. I'm I'm trying to think what what what do you think will be the end state of scaling something like this? Like, practically. Like, give a bunch of these kind of glasses to a bunch of people and just ask them to do, like, just do stuff in the wild. Basically, do stuff in the They don't even, like, because if it's it's meant to be so natural anyway. So even if the the... Like, there's dirtier data that come back up because you you probably get so much data, you should be able to get, like, you know, good good signal data from. Because I do agree. I mean, having text while and all that. So we actually printed text while... I I haven't seen it, but I haven't tried myself, but my team may have tried. It's... Yeah. Just fitting it on is not very natural. I I think the the the s hand version, like, you feel like a robot. Right? And and definitely, there's stuff that you can't do. Way worse than a menace glove, I would say. [00:41:57] S1: So the question was, like [00:42:00] S0: Yeah. I mean, how how do you think that this? Yeah. Yeah. How do you think this will scale? Right? I mean, do you think it's it's just as simple as giving, you know, a bunch of these glasses out to, you know, literally random folks and just record their daily life? Right. And you you take the slam info that you can get from the glasses, and that could be a much better way to to scale this compared to TeleOp and compared to, like, DexWild, you know, DexUMI kind of thing.