00:17:11 · Lawrence Zhu
So what we try to feed into this super simple three layer MLP ReLU and softmax network is like, I collected five minute of manipulation data of me performing random hand movements on the table and I collected another five minute navigation data of me walking around the room without doing anything with my hand and we actually tried to train this it actually worked Even with such a simple, like, simple input, it's basically just my hand and hand pose. Actually, if you see the annotation on the left, if it's green, it means I'm navigating. If it's purple, it's manipulation phase. Just you can see our entire team in the lab. They're not confused. What am I doing here? Yeah. It's actually some pretty good split, but the issue is sometimes you can see when my hand moves, but, like, it's got a little bit delayed, not 100% accurate. It works for all tasks with similar motion and is provided in the training set which is like my five minutes manipulation and walking but the disadvantage here that it requires demonstration to have similar speed or trajectory for example we actually tried one data episode collected by me trained, slid it with this model works perfectly, but it probably won't work, for, the data Pranav collected. So, here comes the problem is why do we need a near 100% accurate face slidder? For evil mimics, they don't actually have this splitting problem because, first of all, they didn't move. The problem... Another, problem is, like, we are actually using this ad movement as a action we wanna predict during navigation phase. So this noise during manipulation phase actually needs to be Otherwise, cleaned when you're actually approaching a table, you're actually not predicting stop there. Right? You're actually going forward if you have, like, noisy hat movements performed during your manipulation phase. So we actually really wanna clean this up. That means we need a near 100% accurate face splitter.
00:19:05 · Michael Cho, Lawrence Zhu
[00:19:05] S3: This kinda sounds like you guys need the humanoid robot as the Exactly. Yeah. Right? Yep. So... [00:19:12] S1: Yeah. Let's say how how about we use some sort of a a world frame information. So in this case, this is another demo of the complete table service task. We can see in the right, we do have a super accurate trajectory and the world and the unified world frame that can actually give us some information about this task. So what we have, step two, let's just use clustering. Right? Because if trained model are super kinda overfitting to my five minute demo each, then let's try just have a unsupervised clustering method. So we just compute the hand to hand velocity ratio because that's like a super simple heuristic to determine whether we're performing manipulation or navigation. And for step two, we just detect ten day manipulation segments with a simple threshold it will just be a cut and then that will still be noisy so what we try to model this is we have x y coordinates and why don't we just model the continuous three d sorry continuous two d space as a distribution. Right? So we can just pretty much use a Gaussian mixture model. It's more like a pure engineering choice here, but we can pretty much have the three d space as a distribution that if you can see in the right... Yeah. So the jet color maps there actually shows the likelihood of being manipulation. Though you can see, it highlights the two manipulation zooms at the table. And on the left, the annotation is actually working much better.
00:20:38 · Chris Paxton, Lawrence Zhu
[00:20:38] S0: Yeah. I I wonder whether this... So I guess you you guys are trying to take this really binary. Right? Is you... You're either manipulating or you are navigating. I wonder... I'm just trying to think as a human. I suppose when I about... I'm about to approach a table. Mhmm. Maybe when I'm one meter away, I'm starting... The last one, two steps, I'm starting to... Slow down. Is... The navigation is like a active vision kind of like... I'm... I'm exploring. I'm looking at, you know, getting a sense of the object, like, doing Mhmm. Things like that. [00:21:08] S1: Mhmm. [00:21:09] S0: How do you think about that? Like like, do you think it's a trade off? Maybe this this problem is simpler by doing, like, two... Just like like, yeah, two kind of classes versus this more transitory, especially in the last, let's say, one meter. [00:21:22] S1: Yeah. So I think what we have here is, basically, we try to split our data. Right? Because we are co training the... We only have static robot data that's pretty much with some observation like this. It's, like, super close to the table. So what we want here is basically figuring out what part of the human data we're co training it with the robot data and what part we are actually direct transferring that we can actually leverage that part of data for navigation. So I would say it's more like an engineering choice. We need to keep it binary, but I would say with the similar approach, we'll be able to perform some sort of, like, fine grained subtask segmentation that... If we have the, like, complete map information.
00:21:22 · Lawrence Zhu, Pranav
[00:21:22] S1: Yeah. So I think what we have here is, basically, we try to split our data. Right? Because we are co training the... We only have static robot data that's pretty much with some observation like this. It's, like, super close to the table. So what we want here is basically figuring out what part of the human data we're co training it with the robot data and what part we are actually direct transferring that we can actually leverage that part of data for navigation. So I would say it's more like an engineering choice. We need to keep it binary, but I would say with the similar approach, we'll be able to perform some sort of, like, fine grained subtask segmentation that... If we have the, like, complete map information. [00:22:06] S2: Yeah. Going off of that, I definitely do agree with I do agree with what you're saying, though. Like, when you're approaching a table, you do definitely, like, start to look at, like, where to put stuff on the table and what the setup is and everything. I think, yeah, the reason why we did this is because it's easiest to, like, completely direct transfer human to robot for navigation over, like, manipulation. And then, like, as you come to the table, once you're at the table and you phase switch into manipulation, you then get the entire thing. It's as if, like, the robot is starting new and it just, like, sees a scene and then does manipulation. And so it already has, like, the entire scene. So it's kinda just, like... If you think of, like, a manipulation task, it's, like, a fully, like, clean slate at that point.
00:22:06 · Pranav, Chris Paxton, Lawrence Zhu
[00:22:06] S2: Yeah. Going off of that, I definitely do agree with I do agree with what you're saying, though. Like, when you're approaching a table, you do definitely, like, start to look at, like, where to put stuff on the table and what the setup is and everything. I think, yeah, the reason why we did this is because it's easiest to, like, completely direct transfer human to robot for navigation over, like, manipulation. And then, like, as you come to the table, once you're at the table and you phase switch into manipulation, you then get the entire thing. It's as if, like, the robot is starting new and it just, like, sees a scene and then does manipulation. And so it already has, like, the entire scene. So it's kinda just, like... If you think of, like, a manipulation task, it's, like, a fully, like, clean slate at that point. [00:22:45] S0: And and I assume, like, even when you're navigating, you are, you know, purposely have... Leaving your hands, like, up in the frame. It's just so that you you can teach a robot what to do with his own hand. So Okay. Okay. Got it. [00:22:56] S1: In this case... Yeah. We'll just assume it's, like, dense demoed. Yep. [00:23:02] S0: Cool. I see. So this is more more because of the limitation of the hardware because you're using smart glasses that is only pointing kinda in front. You you can't see the whole body. Yep. Presumably, if there's, you know, if if somehow you you dangle another GoPro that's facing the whole... Your whole body, you could have leave your hands on the side. And
00:23:02 · Chris Paxton, Lawrence Zhu, Pranav, Michael Cho
[00:23:02] S0: Cool. I see. So this is more more because of the limitation of the hardware because you're using smart glasses that is only pointing kinda in front. You you can't see the whole body. Yep. Presumably, if there's, you know, if if somehow you you dangle another GoPro that's facing the whole... Your whole body, you could have leave your hands on the side. And [00:23:21] S1: Yeah. Yeah. Yeah. Okay. It's more like a FOV thing. Yep. [00:23:25] S0: Got it. I think one other [00:23:27] S2: thing though is that a lot of our tasks are relying on, like, carrying stuff. So, like, as yours... You see here, like, if you're holding a plate and stuff, usually, you don't, like, leave it dangling next to you. Like, hold it up in front of you. [00:23:37] S0: Got it. [00:23:37] S2: So some of the tasks do actually work out with that too. [00:23:40] S0: Alright. Cool. [00:23:41] S3: How how sensitive is it? How, like, how precise do you actually need to be when it comes to doing this classification?