00:32:49 · Ademi Adeniji
[00:32:49] S2: Yeah. I think two things. So for one, I actually would beg to differ in terms of ergonomics of just using your hands, which you're sort of kind of like maybe evolutionarily prescient to just do, like operate within your own embodiment. And also people are just used to doing like by operating their everyday lives. I think there is a gap there in terms of latency. And I think it's also pretty difficult to convince people to use instrumentation that occludes their natural function. So I think, yeah, there's a gap there. I think there's this point about you want really precise and high upper bound ground truth in your data. You want your data to be pure in some sense. And then you want your algorithms and your models push towards that ground truth and approximate it as closely as you can. And I think once you start integrating more instrumentation, you sort of reduce the fidelity of your ground truth. And I think that this can be constraining. And yeah, I think the other piece there is just in terms of scalability, right? I think we've got lots of data online on the internet of people operating within their natural morphology that while we don't actually explicitly use in this project, although we do use hand tracking, which is trained on those large datasets, I think there's so many interesting directions where you've got a small, maybe high fidelity dataset with super accurate three d information, and you couple that with large scale internet, let's say dirtier data, but that still has that ground truth in terms of it being humans actually doing things as they naturally would. And so, yeah, I think gloves and these appendages that people use are interesting directions. But yeah, I think we should really focus on trying to preserve the fidelity of our ground truth as much as possible.
00:34:51 · Vincent Liu, Chris Paxton
[00:34:51] S1: Even from, like, personal experience... So actually, what we tried before... EgoZero and Field Force, I guess, came out of one big project that Ademi and I had worked on for a while before either of these two. And so we sort of started last summer around August, I'd say. We probably started working on a big thing. And, basically, what we tried from August to February was very similar to DexWild, where we actually used the manus gloves and and a rucotag and, like, a chest mounted iPhone. And it's like all these things are... You know, it's like, oh, you know, it's might not be that big of a deal to have someone do a chest mounted iPhone or all these things. But even wearing something like manus gloves, you just kinda felt that you couldn't do so many things very naturally. Mhmm. It was cool to see, like, the authors of DexWild had a setting very similar to ours, and they end up getting it to work. We sort of, like, we sort of after, like, January took a step back and realized that this sort of isn't in the direction of ergonomics that we felt would really scale. [00:35:49] S3: So just because Adam mentioned it, do you... So do you think that this is a... You're gonna be able to leverage, like, YouTube video data at some point in the future, or do you think this this kind of... That is that is, like, an obvious strength of this approach. Right? Like, this is much closer to, like, what you would get from just third party data. Like, how much do you really need the the Aria SLAM kinda toolkit here?
00:35:49 · Chris Paxton, Vincent Liu
[00:35:49] S3: So just because Adam mentioned it, do you... So do you think that this is a... You're gonna be able to leverage, like, YouTube video data at some point in the future, or do you think this this kind of... That is that is, like, an obvious strength of this approach. Right? Like, this is much closer to, like, what you would get from just third party data. Like, how much do you really need the the Aria SLAM kinda toolkit here? [00:36:11] S1: I'd say with the the second part, at least, with the Aria SLAM, this is... Like, the SLAM is also used for... In addition to how we use it to get all these point informations, it's also used for just projecting things in a fixed frame. And I guess... And so, like, that is probably pretty important for inference, right, where your robot's not really moving, or at least it starts out, like, in a certain spot. And I guess that question is more like... There's, like, a broader question maybe there, which is, like, how important is this camera information? And I feel like that's the downside of YouTube data is you kinda just have unknown camera information. So if you really are trying to extract accurate BD, like, proprioception, that will be very hard to do. [00:36:55] S3: Right. Well, certainly, if there's, like, cuts or whatever, you can't pull from that from that video. But, like, I was wondering if, like, structure for motion, you can get accurate enough camera pose, you'd be able to do this. Or if that's just, like, you don't think that's there, that technology is too far off. Like Yes,
00:36:55 · Chris Paxton, Ademi Adeniji
[00:36:55] S3: Right. Well, certainly, if there's, like, cuts or whatever, you can't pull from that from that video. But, like, I was wondering if, like, structure for motion, you can get accurate enough camera pose, you'd be able to do this. Or if that's just, like, you don't think that's there, that technology is too far off. Like Yes, [00:37:09] S2: we Yeah. I mean, my my view is we've tried to get structure for motion working for a really long time, and people have spent a lot of effort trying to get really precise in six. But I think there's been cool work actually. That's leverage structure for mentioned, like, ideas. Like, I think there's some stuff out of CMU and a few other places that try and do something actually quite similar to what we're doing, but using this, again, I'll use it dirtier data and trying to impute camera information. Yeah, I just think the signal isn't there generally. And really, we should just invest in scaling the right signal. And I think that underscores the importance of having something very frictionless such that you actually can get to that scale, but with the signal in the data and in in your data annotations, importantly. So that... That's my view anyway.
00:38:04 · Vincent Liu, Michael Cho, Ademi Adeniji
[00:38:04] S1: Yeah. I'll add something concrete to that. So in the process of trying to get depth, let's just say depth estimation solution working, we've tried a lot of things. And one of them was actually, like, MASt3R-SLAM. I'm not sure if you've heard about it or if there's been a lot of work here. But, like, those... The the videos they would post would be pretty compelling, like, where you can just take, like, a walkthrough in a video game and would map out the whole thing. Right? And so we actually tried this. We ran MASt3R-SLAM and tried to get depth from, like, the point clouds that it will produce, and we just weren't able to get enough, like, precision there. Like, we we we spend, like, a few weeks trying this. And so, you know, conceptually, I'm like, you know, also, like, I... My background before time at NYU was at a three d startup for a couple years. And so I'm familiar with, like, these all these three d approaches. And so, you know, I would think that, you know, you can just build, like, a, like, a three d map from from random videos, but, like, in practice, we've had a hard time doing that. [00:39:02] S0: Yep. I was wondering if... Do do you think it's possible that because in this case, like, the the view of you coming from the the two cameras on on Aria, very wide view, do you think it matters? Because if... [00:39:14] S2: I I
00:39:14 · Ademi Adeniji, Michael Cho, Vincent Liu
[00:39:14] S2: I I [00:39:15] S0: don't know specifically how how is master... I mean, a bunch of these, like, camera pose. Because oftentimes, their their viewer view is... The training pipeline, they use narrower, let's say, you know, 90 degrees. I'm just eyeballing this. This look pretty wide. I... I'm guessing it's, like, close to... I don't I don't know. Yeah. A 160 degrees, maybe? [00:39:32] S1: Yeah. They are pretty wide. MASt3R-SLAM specifically had an option for you to input the camera intrinsics, which we did. And like for Aria, they provide these. [00:39:41] S0: I see.