00:11:48 · Lawrence Zhu
[00:11:48] S1: So, yeah, as... We're going back to the point is that collecting robot demonstration is expensive, and, of course, collecting mobile robot demonstration is more expensive as shown by Pranav and I for collecting data back there. So we came out with the key hypotheses or, like, the key key problem we wanna solve is how can we scale up mobile manipulation with human data, or how can we do ego mimic mobile? So this has been pretty much, this is our project name for a long, long time, Pranav, if you remembered. I think, we changed it to the alpha at the very end. Very basic solution to this would be, co training mobile human data with mobile robot data. So let's try to make similar move. And, the issue is mobile robot data is still super slow to collect. Right? We didn't solve the data bottleneck here. So, what's our hypothesis? Can we basically direct transfer navigation knowledge from human to robot? Because we know that hypothesis... We know that, like, transferring navigation will actually be super difficult as we we have in manipulation because grasping actually between human and robot has a huge gap. But for navigation, you're actually looking at the same scene with the same background. So we can achieve mobile manipulation by co training mobile human data plus static robot data, and the static robot data will be relatively easier to acquire. So the answer is yes. This is me testing, one of the very first results we got. Basically, I was recording me walking around, like, just back and forth between the two tables, and that's, like, zero shot transferred on the robot. And, although we might have different kinematics, we might look a little bit different, but by collecting enough data, we model the robots during inference time would be a strict subset. That is also, one of the key, assumption we make for this work is that it will able to, direct transfer navigation knowledge from human.
00:13:38 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:13:38] S3: Does the head height of the robot matter at all there? Like, does it need to be at exactly the same orientation or position as you reported the [00:13:45] S1: Actually, it needs to be a subset, but it doesn't need to [00:13:47] S3: be Subset. Okay. Yeah. Exactly. How much data did you use for this? [00:13:51] S1: So I think this task is relatively easy. Maybe, like, five, ten minutes, that'll be enough. But, like, for our tasks used in the result, we all control the human data to be one hour long data collection time. [00:14:02] S0: Yeah. So when you say FICO enemies, in in that case, you mean, like, you wear the Aria glasses, you literally walk between the table, the... With your cloth and then to the other table, like, for five minutes. You keep your hand and then... Okay. Got it. Okay. [00:14:15] S3: The robot is not crazy, like crouched walk or something to make sure you had a good distance your head, bothers. [00:14:21] S1: I don't have to torque my head like this. And then... But I I I will say if you Just natural walking. Yeah. Yeah. Just natural walking. If you intentionally, like, look up or look aside, then probably won't direct transfer.
00:14:21 · Lawrence Zhu, Pranav, Chris Paxton
[00:14:21] S1: I don't have to torque my head like this. And then... But I I I will say if you Just natural walking. Yeah. Yeah. Just natural walking. If you intentionally, like, look up or look aside, then probably won't direct transfer. [00:14:31] S2: Yeah. Yeah. That's what I was gonna say. Like, one thing that's important is that, like, you have to, like, walk, like, the path and be, like, very careful with that because, like, there is retargeting and stuff that Lawrence will talk about later, but, like, it is... Like, the path you go is the path that the robot will follow. One thing is that, like, the height of the robot, like, it is still very robust, but, like, we do notice that, like, we do have to, like, sometimes, like, try to match the visual viewpoint of the robot. It doesn't have to be exact, but it sometimes needs to be, like, close enough as as what [00:14:57] S1: I saying. Like a subset. [00:14:59] S2: Yes. A subset. [00:15:00] S0: Cool.
00:15:01 · Lawrence Zhu
[00:15:01] S1: Oh, yeah. I'll skip that slide. The key assumption here is, if we can't direct transfer navigation, then all we need to co train is mobile, robot data and static human data. So, I'm gonna make an example here is how do you put an elephant into the fridge. I'm making the assumption here, for example, when you have to, like, attach the hand on the fridge door, you, move your buddy backward to open the door, and then take the giraffe fox, for example, etcetera, etcetera. But this example is really just to show that I... We believe most of the human activities can be split into sequences of navigation and manipulation. And, that's basically our, key hypothesis here. And then we like our key assumption for our co training, this set of data. So we started data collection. This is one of our flagship tasks called table service. So we pretty much just serve croissant a baking oven, and then we collect static global data of picking up the croissant and serving it on the other table. So it's 15 demo for each variant. And then we start to actually try to train a model and then roll out this task. We realized there are challenges. So the first task is the human full body data is unstructured. So you don't know where exactly is the boundary between the manipulation and navigation phase. Right? So we need some sort of auxiliary prediction that splits the two. And there's always a kinematic gap between the human and robot because human move omnidirectionally you might have like sharp turns you might have lateral movements where our differential drive robot base is unable to kind of replay that so let's say we try first we try to resolve the first problem. So the first attempt we have is, like, we look back to what resources we got from Project Aria glasses. So we see, okay, we got amp pose and we got offline slam, we at least have some sort of localization or, like, we have some sort of map info during data processing. We don't have that during real time, but we have this in our data. And let's say if we can solve a two shot classification problem by just have some sort of a model to slur human data into navigation and manipulation phases. So because we are all CS majors, so let's put it in a neural network.
00:17:11 · Lawrence Zhu
So what we try to feed into this super simple three layer MLP ReLU and softmax network is like, I collected five minute of manipulation data of me performing random hand movements on the table and I collected another five minute navigation data of me walking around the room without doing anything with my hand and we actually tried to train this it actually worked Even with such a simple, like, simple input, it's basically just my hand and hand pose. Actually, if you see the annotation on the left, if it's green, it means I'm navigating. If it's purple, it's manipulation phase. Just you can see our entire team in the lab. They're not confused. What am I doing here? Yeah. It's actually some pretty good split, but the issue is sometimes you can see when my hand moves, but, like, it's got a little bit delayed, not 100% accurate. It works for all tasks with similar motion and is provided in the training set which is like my five minutes manipulation and walking but the disadvantage here that it requires demonstration to have similar speed or trajectory for example we actually tried one data episode collected by me trained, slid it with this model works perfectly, but it probably won't work, for, the data Pranav collected. So, here comes the problem is why do we need a near 100% accurate face slidder? For evil mimics, they don't actually have this splitting problem because, first of all, they didn't move. The problem... Another, problem is, like, we are actually using this ad movement as a action we wanna predict during navigation phase. So this noise during manipulation phase actually needs to be Otherwise, cleaned when you're actually approaching a table, you're actually not predicting stop there. Right? You're actually going forward if you have, like, noisy hat movements performed during your manipulation phase. So we actually really wanna clean this up. That means we need a near 100% accurate face splitter.
00:19:05 · Michael Cho, Lawrence Zhu
[00:19:05] S3: This kinda sounds like you guys need the humanoid robot as the Exactly. Yeah. Right? Yep. So... [00:19:12] S1: Yeah. Let's say how how about we use some sort of a a world frame information. So in this case, this is another demo of the complete table service task. We can see in the right, we do have a super accurate trajectory and the world and the unified world frame that can actually give us some information about this task. So what we have, step two, let's just use clustering. Right? Because if trained model are super kinda overfitting to my five minute demo each, then let's try just have a unsupervised clustering method. So we just compute the hand to hand velocity ratio because that's like a super simple heuristic to determine whether we're performing manipulation or navigation. And for step two, we just detect ten day manipulation segments with a simple threshold it will just be a cut and then that will still be noisy so what we try to model this is we have x y coordinates and why don't we just model the continuous three d sorry continuous two d space as a distribution. Right? So we can just pretty much use a Gaussian mixture model. It's more like a pure engineering choice here, but we can pretty much have the three d space as a distribution that if you can see in the right... Yeah. So the jet color maps there actually shows the likelihood of being manipulation. Though you can see, it highlights the two manipulation zooms at the table. And on the left, the annotation is actually working much better.