00:21:26 · Michael Cho, Vincent Liu, Chris Paxton
[00:21:26] S0: Makes sense. And [00:21:28] S1: I guess the the sort of last, like, interesting, maybe not too surprising result is that our three d representations are very interpretable. You know, it's, again, it's like a very linear mapping space from states to actions. And in training, we sample random three d transformations and then apply them to entire demonstrations because everything's in three d. And so here on this plot, the blue area is the distribution of object points, egocentric object points in the training data. And then the gray area is sort of like this distribution that is augmented after sampling all these transformations. And you'll see that the model starts to... The model does very well as long as the inference time points are within this augmented training this... Training distribution. And as soon as you sort of get to the edge of this... Of its, like, support volume, it begins to fail, which sort of makes sense. And it also demonstrates that these augmentations are pretty important to be able to generalize from different distances, heights, you know, camera settings. So that's pretty much it. The longer version of our, I guess, our assist workshop talk. And we can go to the website too now if we wanna check out some more videos or details. Sure. [00:22:41] S0: Yeah. Yeah. That's what [00:22:42] S3: what kind of network architecture is this? Out of curiosity, is this one of those, like, I don't know, three d specialized models, or is this a transformer that you threw all this data into? How are you... How do you make it work?
00:22:42 · Chris Paxton, Vincent Liu, Michael Cho, Ademi Adeniji
[00:22:42] S3: what kind of network architecture is this? Out of curiosity, is this one of those, like, I don't know, three d specialized models, or is this a transformer that you threw all this data into? How are you... How do you make it work? [00:22:54] S1: Just a small, like, Baku transformer. Okay. Literally, like, the the number of parameters is, like, a low single digit millions. You don't really need much more to learn a very simple problem. [00:23:07] S0: Yep. Yeah. [00:23:08] S3: I mean, three d... I feel like three d models are usually pretty efficient... Data efficient. Right? [00:23:12] S1: So kind of Do you mean like like point nets and stuff? [00:23:16] S3: No. Well, I meant whatever. Anything. [00:23:18] S2: Okay. [00:23:18] S3: Using three d information. But, yes, those two [00:23:20] S0: are I see. [00:23:20] S1: I see. Demi, is there anything you wanna add? [00:23:25] S2: Yeah, I think that was pretty comprehensive. I think we should maybe look at some of the in the wild to inference time transfer results. Like I think there's some cool videos where you like fold a towel in your bedroom and it transfers.
00:23:25 · Ademi Adeniji, Vincent Liu, Chris Paxton
[00:23:25] S2: Yeah, I think that was pretty comprehensive. I think we should maybe look at some of the in the wild to inference time transfer results. Like I think there's some cool videos where you like fold a towel in your bedroom and it transfers. [00:23:37] S1: Yes. Let's see. We can probably find... You guys can still see my screen, right? I guess these aren't in my... Well, maybe this... Yeah, this one is. Yeah. So the nice part about this is like with just twenty minutes of data basically, because everything is super summarized, like like the... If if you have, like, this general, like, learning stack pipeline, however you wanna call it, where you go from dense video to just actions, in a way, we're, like, sort of move... We're shrinking the portion of policy, like, that policy learning is responsible for. Right? We're sort of, like, extract... We're expanding a lot of compute and, like, knowledge contained in, like, grounded vision models to simplify the learning problem. So, like, that kind of comes with some trade offs. But one of the benefits is that, you know, with just twenty minutes of data across four four different... Very different visual settings here, none of these are even in a lab. These these these three here. These sort of point information is robust across all these settings, which is why, you know, the robot's able to fold the towel as well. [00:24:38] S3: Yeah. What's the success rate on this, for example?
00:24:38 · Chris Paxton, Vincent Liu
[00:24:38] S3: Yeah. What's the success rate on this, for example? [00:24:41] S1: Everything was around... Let's see. What is it? 60 or 70%? I think 70 on average. And it definitely we noticed that, like, the performance of this of this sort of framework was more of a function of how good the three d measurements are, as opposed to how much data you have. And, like, it was relatively easy to get general behavior because you're able to do these augmentations, you know, to expand your domain input, basically. And also, like, these demonstrations aren't too complicated. It's not like there's many different ways to fold the towel that we've demonstrated. So we really saw that, you know, if you have, for example, HaMeR plus ARIA hand detection, about, like, centimeter of error, whatever. And then so you're therefore limited to sort of that level of precision. [00:25:33] S3: Yeah. Makes sense. [00:25:34] S1: Yeah. And then I guess another nice property, and this is what point policy shows, is that because, again, you're using points, which is just more like spatial geometric information, you can swap out the literal, like, training object, and you're able to, you know, fold a different color towel or, like, you know, just change the color, like, kind of variations of of your training object. [00:25:57] S3: Would it work with distractors? Like, you put another towel down, what would happen?
00:25:57 · Chris Paxton, Vincent Liu, Ademi Adeniji
[00:25:57] S3: Would it work with distractors? Like, you put another towel down, what would happen? [00:26:01] S1: That's a good point. I don't think we've really tried that. I think that my my solution might... If I had to work around that, I'd probably just engineer, like, some higher level thing that sort of sorts this out. [00:26:11] S3: Yeah. Just use like Sam or something. Yeah. That makes [00:26:13] S1: sense. Yeah. Exactly. Yeah. Exactly. [00:26:14] S2: I think the... We have something kind of similar with the fruit bowl experiment. [00:26:17] S1: Yes. That's right. So that's kind of what we do for the fruit bowl or the... Yeah. The... Put the fruit in the bowl. You sort of exploit the fact that you can sort of... Like these fruits are about the same size and so if you label like points on them, they sort of mean the same thing to the model. And therefore, like depending on how you prompt this SAM model or like the correspondence model, you're able to have it go target different fruits, even though sort of like we just have one model trained on pick and place or like pick and place of like different fruits. [00:26:48] S2: Yeah. So basically it's language conditioned, right? Like the segmentation and the actual inference time point selection is kind of guided by how you specify the task. But the actual model is sort of more modular in some sense because, like, the geometric reasoning is exactly the same.
00:26:48 · Ademi Adeniji, Vincent Liu, Chris Paxton
[00:26:48] S2: Yeah. So basically it's language conditioned, right? Like the segmentation and the actual inference time point selection is kind of guided by how you specify the task. But the actual model is sort of more modular in some sense because, like, the geometric reasoning is exactly the same. [00:27:09] S1: Yeah. So this is, like, benefits and... Or there's trade offs to this. Right? So another instance of why we have, like, error rates at all is that we noticed that the highest source of error comes from... Comes in inference from the correspondence model. So it's basically this, like, you just use some old stable diffusion model, you query it for a step, you take out, like, a feature map from its activations. And then you do this for your training image, and you do this for your inference image. And then you just sort of like try to match, like, the cosine similarity for each point that you want to map onto this. And so... [00:27:44] S3: Go ahead. Yeah.