00:27:44 · Chris Paxton, Vincent Liu
[00:27:44] S3: Go ahead. Yeah. [00:27:46] S1: Yeah. And so like another, like, side note is, you know, we collect data with these Aria glasses, which is fisheye, and they're kind of yellowy or, you know, a little look grainy looking. And in inference, we have the iPhone because we just wanna unproject with depth. And so diff is really not a strong model at all or like a algorithm at all. And we basically help it out by using grounding dyno to segment out the object. So this is what ADEMI means when it's language conditioned. Like, the system is sort of language conditioned. So we segment out the object with DINO, which works pretty well. And even when... Even, like, when you run diff on the segmentation, sometimes it gets the points all wrong. Right? So it's like jumbling up the correspondence information or basically the orientation of the object. Yeah. And then sometimes with the iPhone depth map, which is very low resolution, if you... If your points are near the edges of of the object, maybe the depth map flickers. And so you... All of a sudden, you're reading a point that's, like, 30 centimeters further. And so it's just like this kind of, like, system level thing, which is maybe in a way a drawback of having, like, a not fully end to end setting. [00:28:53] S3: Yeah. That makes sense. Well, you could always, like, train your own point correspondence model or something like that. I guess that obviously out of scope for this paper, but is that the... Do you think that's, like, the way forward to improve... To best improve performance then?
00:28:53 · Chris Paxton, Vincent Liu, Ademi Adeniji
[00:28:53] S3: Yeah. That makes sense. Well, you could always, like, train your own point correspondence model or something like that. I guess that obviously out of scope for this paper, but is that the... Do you think that's, like, the way forward to improve... To best improve performance then? [00:29:05] S1: I... Again, it's, like, one of those things where that might be optimizing or, like, trying to push a local optima. [00:29:11] S2: Sure. [00:29:11] S1: Yeah. Me too, if you wanna speak to... Like, we were thinking of, like, some extensions of this in a more foundation model setting, and we realized that correspondence is probably one of the things that had to go first. [00:29:19] S3: Like, to remove it, not to improve it. [00:29:21] S2: Yeah. Yeah. I totally agree. Yeah. [00:29:23] S3: I think [00:29:24] S2: ideally you break the correspondence completely by basically learning invariance in your training. So yeah, I think the next step here would be basically, can you basically not have the system rely on correspondence at all? And, still change points, but like break the strict requirement that like, you know, the first point corresponds to like, you know, something semantic. So instead your transformer is learning something more geometric and spatial as opposed to having to keep this like specific ordered set of points consistent.
00:30:06 · Vincent Liu
[00:30:06] S1: Right. So there's a very practical reason for that too. And it's that when you have this correspondence, you're basically for each task, defining like the finite set of points that is fixed for that task. And so, you know, even if you had five points for your bread, five points for your oven, the points... The point IDs, like, wouldn't actually mean anything. Like, it wouldn't transfer across these tasks. So you'd sort of have to get rid of this in a way to train a multitask policy that's more than just like a like a expert, whatever, network ensemble. Yeah. And so the issue kind of with... I I agree with the Demi too. Like, you kinda want to absorb this pose information and, like, more implicitly, ideally. And so I think maybe the reason that might be hard would be that you don't... Like, the world doesn't necessarily have that much dense, diverse three d data to learn these things on, right? And so that's why VLMs and VLAs have done so well. It's because the image modality is much more rich and there's much more data and it's much easier to get. But you know, you can't really have like a swap out the vision, swap out the images for point clouds and train something at maybe that large of a scale that will actually learn correspondence well. Or at least it's... I I feel like it's unclear how much how much scale you would need to get there. And so there's been a lot of work lately, actually, shortly after, again, we released EgoZero, where there's a lot of, like, now variations of VLAs. I think, Chris, I've seen you retweet a couple of them where it's like infusing VLAs with three d information, so you're trying to get the best of both worlds. Right?
00:31:39 · Chris Paxton, Michael Cho
[00:31:39] S3: Yeah. Yeah. That makes a lot of sense. [00:31:41] S0: Yeah. I I have a question about the the trade off, I guess, between this kind of smart glasses way of collecting versus something... What? Definitely, I think it's it's... Everyone knows how how painful it is to do to teleop with real real robot data. But I'm just thinking something like DexUMI or DexWild. Right? This kind of project. It it seems to me they... It's it's it's like a it's like a small hardware thing that they... You know, it could be a glove or maybe like a QR code thingy that they put on their wrist. I wonder if by doing that, I mean, it seems to me it's not that much more inconvenient in terms of a UX for for the human. And I wonder if those those... I'm trying to think what is the trade off. Right? Yeah. Why don't... I mean, why don't we do smart glasses with the DexUMI stuff if the DexUMI are are quite easy also? Or do you think actually there is benefit to using actual human hands because the human hands can can do more things? Like, what do you think of that?
00:32:49 · Ademi Adeniji
[00:32:49] S2: Yeah. I think two things. So for one, I actually would beg to differ in terms of ergonomics of just using your hands, which you're sort of kind of like maybe evolutionarily prescient to just do, like operate within your own embodiment. And also people are just used to doing like by operating their everyday lives. I think there is a gap there in terms of latency. And I think it's also pretty difficult to convince people to use instrumentation that occludes their natural function. So I think, yeah, there's a gap there. I think there's this point about you want really precise and high upper bound ground truth in your data. You want your data to be pure in some sense. And then you want your algorithms and your models push towards that ground truth and approximate it as closely as you can. And I think once you start integrating more instrumentation, you sort of reduce the fidelity of your ground truth. And I think that this can be constraining. And yeah, I think the other piece there is just in terms of scalability, right? I think we've got lots of data online on the internet of people operating within their natural morphology that while we don't actually explicitly use in this project, although we do use hand tracking, which is trained on those large datasets, I think there's so many interesting directions where you've got a small, maybe high fidelity dataset with super accurate three d information, and you couple that with large scale internet, let's say dirtier data, but that still has that ground truth in terms of it being humans actually doing things as they naturally would. And so, yeah, I think gloves and these appendages that people use are interesting directions. But yeah, I think we should really focus on trying to preserve the fidelity of our ground truth as much as possible.
00:34:51 · Vincent Liu, Chris Paxton
[00:34:51] S1: Even from, like, personal experience... So actually, what we tried before... EgoZero and Field Force, I guess, came out of one big project that Ademi and I had worked on for a while before either of these two. And so we sort of started last summer around August, I'd say. We probably started working on a big thing. And, basically, what we tried from August to February was very similar to DexWild, where we actually used the manus gloves and and a rucotag and, like, a chest mounted iPhone. And it's like all these things are... You know, it's like, oh, you know, it's might not be that big of a deal to have someone do a chest mounted iPhone or all these things. But even wearing something like manus gloves, you just kinda felt that you couldn't do so many things very naturally. Mhmm. It was cool to see, like, the authors of DexWild had a setting very similar to ours, and they end up getting it to work. We sort of, like, we sort of after, like, January took a step back and realized that this sort of isn't in the direction of ergonomics that we felt would really scale. [00:35:49] S3: So just because Adam mentioned it, do you... So do you think that this is a... You're gonna be able to leverage, like, YouTube video data at some point in the future, or do you think this this kind of... That is that is, like, an obvious strength of this approach. Right? Like, this is much closer to, like, what you would get from just third party data. Like, how much do you really need the the Aria SLAM kinda toolkit here?