00:14:21 · Lawrence Zhu, Pranav, Chris Paxton
[00:14:21] S1: I don't have to torque my head like this. And then... But I I I will say if you Just natural walking. Yeah. Yeah. Just natural walking. If you intentionally, like, look up or look aside, then probably won't direct transfer. [00:14:31] S2: Yeah. Yeah. That's what I was gonna say. Like, one thing that's important is that, like, you have to, like, walk, like, the path and be, like, very careful with that because, like, there is retargeting and stuff that Lawrence will talk about later, but, like, it is... Like, the path you go is the path that the robot will follow. One thing is that, like, the height of the robot, like, it is still very robust, but, like, we do notice that, like, we do have to, like, sometimes, like, try to match the visual viewpoint of the robot. It doesn't have to be exact, but it sometimes needs to be, like, close enough as as what [00:14:57] S1: I saying. Like a subset. [00:14:59] S2: Yes. A subset. [00:15:00] S0: Cool.
00:15:01 · Lawrence Zhu
[00:15:01] S1: Oh, yeah. I'll skip that slide. The key assumption here is, if we can't direct transfer navigation, then all we need to co train is mobile, robot data and static human data. So, I'm gonna make an example here is how do you put an elephant into the fridge. I'm making the assumption here, for example, when you have to, like, attach the hand on the fridge door, you, move your buddy backward to open the door, and then take the giraffe fox, for example, etcetera, etcetera. But this example is really just to show that I... We believe most of the human activities can be split into sequences of navigation and manipulation. And, that's basically our, key hypothesis here. And then we like our key assumption for our co training, this set of data. So we started data collection. This is one of our flagship tasks called table service. So we pretty much just serve croissant a baking oven, and then we collect static global data of picking up the croissant and serving it on the other table. So it's 15 demo for each variant. And then we start to actually try to train a model and then roll out this task. We realized there are challenges. So the first task is the human full body data is unstructured. So you don't know where exactly is the boundary between the manipulation and navigation phase. Right? So we need some sort of auxiliary prediction that splits the two. And there's always a kinematic gap between the human and robot because human move omnidirectionally you might have like sharp turns you might have lateral movements where our differential drive robot base is unable to kind of replay that so let's say we try first we try to resolve the first problem. So the first attempt we have is, like, we look back to what resources we got from Project Aria glasses. So we see, okay, we got amp pose and we got offline slam, we at least have some sort of localization or, like, we have some sort of map info during data processing. We don't have that during real time, but we have this in our data. And let's say if we can solve a two shot classification problem by just have some sort of a model to slur human data into navigation and manipulation phases. So because we are all CS majors, so let's put it in a neural network.
00:17:11 · Lawrence Zhu
So what we try to feed into this super simple three layer MLP ReLU and softmax network is like, I collected five minute of manipulation data of me performing random hand movements on the table and I collected another five minute navigation data of me walking around the room without doing anything with my hand and we actually tried to train this it actually worked Even with such a simple, like, simple input, it's basically just my hand and hand pose. Actually, if you see the annotation on the left, if it's green, it means I'm navigating. If it's purple, it's manipulation phase. Just you can see our entire team in the lab. They're not confused. What am I doing here? Yeah. It's actually some pretty good split, but the issue is sometimes you can see when my hand moves, but, like, it's got a little bit delayed, not 100% accurate. It works for all tasks with similar motion and is provided in the training set which is like my five minutes manipulation and walking but the disadvantage here that it requires demonstration to have similar speed or trajectory for example we actually tried one data episode collected by me trained, slid it with this model works perfectly, but it probably won't work, for, the data Pranav collected. So, here comes the problem is why do we need a near 100% accurate face slidder? For evil mimics, they don't actually have this splitting problem because, first of all, they didn't move. The problem... Another, problem is, like, we are actually using this ad movement as a action we wanna predict during navigation phase. So this noise during manipulation phase actually needs to be Otherwise, cleaned when you're actually approaching a table, you're actually not predicting stop there. Right? You're actually going forward if you have, like, noisy hat movements performed during your manipulation phase. So we actually really wanna clean this up. That means we need a near 100% accurate face splitter.
00:19:05 · Michael Cho, Lawrence Zhu
[00:19:05] S3: This kinda sounds like you guys need the humanoid robot as the Exactly. Yeah. Right? Yep. So... [00:19:12] S1: Yeah. Let's say how how about we use some sort of a a world frame information. So in this case, this is another demo of the complete table service task. We can see in the right, we do have a super accurate trajectory and the world and the unified world frame that can actually give us some information about this task. So what we have, step two, let's just use clustering. Right? Because if trained model are super kinda overfitting to my five minute demo each, then let's try just have a unsupervised clustering method. So we just compute the hand to hand velocity ratio because that's like a super simple heuristic to determine whether we're performing manipulation or navigation. And for step two, we just detect ten day manipulation segments with a simple threshold it will just be a cut and then that will still be noisy so what we try to model this is we have x y coordinates and why don't we just model the continuous three d sorry continuous two d space as a distribution. Right? So we can just pretty much use a Gaussian mixture model. It's more like a pure engineering choice here, but we can pretty much have the three d space as a distribution that if you can see in the right... Yeah. So the jet color maps there actually shows the likelihood of being manipulation. Though you can see, it highlights the two manipulation zooms at the table. And on the left, the annotation is actually working much better.
00:20:38 · Chris Paxton, Lawrence Zhu
[00:20:38] S0: Yeah. I I wonder whether this... So I guess you you guys are trying to take this really binary. Right? Is you... You're either manipulating or you are navigating. I wonder... I'm just trying to think as a human. I suppose when I about... I'm about to approach a table. Mhmm. Maybe when I'm one meter away, I'm starting... The last one, two steps, I'm starting to... Slow down. Is... The navigation is like a active vision kind of like... I'm... I'm exploring. I'm looking at, you know, getting a sense of the object, like, doing Mhmm. Things like that. [00:21:08] S1: Mhmm. [00:21:09] S0: How do you think about that? Like like, do you think it's a trade off? Maybe this this problem is simpler by doing, like, two... Just like like, yeah, two kind of classes versus this more transitory, especially in the last, let's say, one meter. [00:21:22] S1: Yeah. So I think what we have here is, basically, we try to split our data. Right? Because we are co training the... We only have static robot data that's pretty much with some observation like this. It's, like, super close to the table. So what we want here is basically figuring out what part of the human data we're co training it with the robot data and what part we are actually direct transferring that we can actually leverage that part of data for navigation. So I would say it's more like an engineering choice. We need to keep it binary, but I would say with the similar approach, we'll be able to perform some sort of, like, fine grained subtask segmentation that... If we have the, like, complete map information.
00:21:22 · Lawrence Zhu, Pranav
[00:21:22] S1: Yeah. So I think what we have here is, basically, we try to split our data. Right? Because we are co training the... We only have static robot data that's pretty much with some observation like this. It's, like, super close to the table. So what we want here is basically figuring out what part of the human data we're co training it with the robot data and what part we are actually direct transferring that we can actually leverage that part of data for navigation. So I would say it's more like an engineering choice. We need to keep it binary, but I would say with the similar approach, we'll be able to perform some sort of, like, fine grained subtask segmentation that... If we have the, like, complete map information. [00:22:06] S2: Yeah. Going off of that, I definitely do agree with I do agree with what you're saying, though. Like, when you're approaching a table, you do definitely, like, start to look at, like, where to put stuff on the table and what the setup is and everything. I think, yeah, the reason why we did this is because it's easiest to, like, completely direct transfer human to robot for navigation over, like, manipulation. And then, like, as you come to the table, once you're at the table and you phase switch into manipulation, you then get the entire thing. It's as if, like, the robot is starting new and it just, like, sees a scene and then does manipulation. And so it already has, like, the entire scene. So it's kinda just, like... If you think of, like, a manipulation task, it's, like, a fully, like, clean slate at that point.