00:24:38 · Chris Paxton, Vincent Liu
[00:24:38] S3: Yeah. What's the success rate on this, for example? [00:24:41] S1: Everything was around... Let's see. What is it? 60 or 70%? I think 70 on average. And it definitely we noticed that, like, the performance of this of this sort of framework was more of a function of how good the three d measurements are, as opposed to how much data you have. And, like, it was relatively easy to get general behavior because you're able to do these augmentations, you know, to expand your domain input, basically. And also, like, these demonstrations aren't too complicated. It's not like there's many different ways to fold the towel that we've demonstrated. So we really saw that, you know, if you have, for example, HaMeR plus ARIA hand detection, about, like, centimeter of error, whatever. And then so you're therefore limited to sort of that level of precision. [00:25:33] S3: Yeah. Makes sense. [00:25:34] S1: Yeah. And then I guess another nice property, and this is what point policy shows, is that because, again, you're using points, which is just more like spatial geometric information, you can swap out the literal, like, training object, and you're able to, you know, fold a different color towel or, like, you know, just change the color, like, kind of variations of of your training object. [00:25:57] S3: Would it work with distractors? Like, you put another towel down, what would happen?
00:25:57 · Chris Paxton, Vincent Liu, Ademi Adeniji
[00:25:57] S3: Would it work with distractors? Like, you put another towel down, what would happen? [00:26:01] S1: That's a good point. I don't think we've really tried that. I think that my my solution might... If I had to work around that, I'd probably just engineer, like, some higher level thing that sort of sorts this out. [00:26:11] S3: Yeah. Just use like Sam or something. Yeah. That makes [00:26:13] S1: sense. Yeah. Exactly. Yeah. Exactly. [00:26:14] S2: I think the... We have something kind of similar with the fruit bowl experiment. [00:26:17] S1: Yes. That's right. So that's kind of what we do for the fruit bowl or the... Yeah. The... Put the fruit in the bowl. You sort of exploit the fact that you can sort of... Like these fruits are about the same size and so if you label like points on them, they sort of mean the same thing to the model. And therefore, like depending on how you prompt this SAM model or like the correspondence model, you're able to have it go target different fruits, even though sort of like we just have one model trained on pick and place or like pick and place of like different fruits. [00:26:48] S2: Yeah. So basically it's language conditioned, right? Like the segmentation and the actual inference time point selection is kind of guided by how you specify the task. But the actual model is sort of more modular in some sense because, like, the geometric reasoning is exactly the same.
00:26:48 · Ademi Adeniji, Vincent Liu, Chris Paxton
[00:26:48] S2: Yeah. So basically it's language conditioned, right? Like the segmentation and the actual inference time point selection is kind of guided by how you specify the task. But the actual model is sort of more modular in some sense because, like, the geometric reasoning is exactly the same. [00:27:09] S1: Yeah. So this is, like, benefits and... Or there's trade offs to this. Right? So another instance of why we have, like, error rates at all is that we noticed that the highest source of error comes from... Comes in inference from the correspondence model. So it's basically this, like, you just use some old stable diffusion model, you query it for a step, you take out, like, a feature map from its activations. And then you do this for your training image, and you do this for your inference image. And then you just sort of like try to match, like, the cosine similarity for each point that you want to map onto this. And so... [00:27:44] S3: Go ahead. Yeah.
00:27:44 · Chris Paxton, Vincent Liu
[00:27:44] S3: Go ahead. Yeah. [00:27:46] S1: Yeah. And so like another, like, side note is, you know, we collect data with these Aria glasses, which is fisheye, and they're kind of yellowy or, you know, a little look grainy looking. And in inference, we have the iPhone because we just wanna unproject with depth. And so diff is really not a strong model at all or like a algorithm at all. And we basically help it out by using grounding dyno to segment out the object. So this is what ADEMI means when it's language conditioned. Like, the system is sort of language conditioned. So we segment out the object with DINO, which works pretty well. And even when... Even, like, when you run diff on the segmentation, sometimes it gets the points all wrong. Right? So it's like jumbling up the correspondence information or basically the orientation of the object. Yeah. And then sometimes with the iPhone depth map, which is very low resolution, if you... If your points are near the edges of of the object, maybe the depth map flickers. And so you... All of a sudden, you're reading a point that's, like, 30 centimeters further. And so it's just like this kind of, like, system level thing, which is maybe in a way a drawback of having, like, a not fully end to end setting. [00:28:53] S3: Yeah. That makes sense. Well, you could always, like, train your own point correspondence model or something like that. I guess that obviously out of scope for this paper, but is that the... Do you think that's, like, the way forward to improve... To best improve performance then?
00:28:53 · Chris Paxton, Vincent Liu, Ademi Adeniji
[00:28:53] S3: Yeah. That makes sense. Well, you could always, like, train your own point correspondence model or something like that. I guess that obviously out of scope for this paper, but is that the... Do you think that's, like, the way forward to improve... To best improve performance then? [00:29:05] S1: I... Again, it's, like, one of those things where that might be optimizing or, like, trying to push a local optima. [00:29:11] S2: Sure. [00:29:11] S1: Yeah. Me too, if you wanna speak to... Like, we were thinking of, like, some extensions of this in a more foundation model setting, and we realized that correspondence is probably one of the things that had to go first. [00:29:19] S3: Like, to remove it, not to improve it. [00:29:21] S2: Yeah. Yeah. I totally agree. Yeah. [00:29:23] S3: I think [00:29:24] S2: ideally you break the correspondence completely by basically learning invariance in your training. So yeah, I think the next step here would be basically, can you basically not have the system rely on correspondence at all? And, still change points, but like break the strict requirement that like, you know, the first point corresponds to like, you know, something semantic. So instead your transformer is learning something more geometric and spatial as opposed to having to keep this like specific ordered set of points consistent.
00:30:06 · Vincent Liu
[00:30:06] S1: Right. So there's a very practical reason for that too. And it's that when you have this correspondence, you're basically for each task, defining like the finite set of points that is fixed for that task. And so, you know, even if you had five points for your bread, five points for your oven, the points... The point IDs, like, wouldn't actually mean anything. Like, it wouldn't transfer across these tasks. So you'd sort of have to get rid of this in a way to train a multitask policy that's more than just like a like a expert, whatever, network ensemble. Yeah. And so the issue kind of with... I I agree with the Demi too. Like, you kinda want to absorb this pose information and, like, more implicitly, ideally. And so I think maybe the reason that might be hard would be that you don't... Like, the world doesn't necessarily have that much dense, diverse three d data to learn these things on, right? And so that's why VLMs and VLAs have done so well. It's because the image modality is much more rich and there's much more data and it's much easier to get. But you know, you can't really have like a swap out the vision, swap out the images for point clouds and train something at maybe that large of a scale that will actually learn correspondence well. Or at least it's... I I feel like it's unclear how much how much scale you would need to get there. And so there's been a lot of work lately, actually, shortly after, again, we released EgoZero, where there's a lot of, like, now variations of VLAs. I think, Chris, I've seen you retweet a couple of them where it's like infusing VLAs with three d information, so you're trying to get the best of both worlds. Right?