00:35:39 · Michael Cho, Chris Paxton, Lawrence Zhu, Pranav
[00:35:39] S3: This is a failure that Yeah. That... Or... Okay. [00:35:41] S0: But is is this, you know, under your face, like, detector, is this considered manipulation? Like, the whole thing is manipulation? [00:35:48] S1: No. This... Sorry. For this model, we removed the face switch because we don't need it anymore. Right? We can just perform the navigation manipulation simultaneously because we have mobile robot data. [00:35:59] S3: Because it's super interesting that the human data would improve the grasp accuracy because that's actually kind of the exact opposite of what I would expect. [00:36:05] S1: Yeah. [00:36:05] S3: Why is that? [00:36:06] S1: It's... I would say it's more like we added a lot of demonstrations of Okay. When actually... When you actually see the end effector is cotrained. Right? So the end effector of a human hand, like, how how it approaches the backrest of the chair is actually densely demonstrated in our human data. For this task, we only have, like... We're starting from ten minutes of mobile robot data, so that will be around, like, 17 demos, if I remember correctly. [00:36:34] S3: Okay. [00:36:34] S1: And we start adding twenty minutes of human data. That will be around, like, 80 demonstrations per now. Is that, like, good estimation? [00:36:40] S2: Yeah. Yeah. Yeah. [00:36:41] S1: I think [00:36:41] S2: that's So, yeah, essentially all [00:36:43] S1: the data... You go ahead.
00:36:43 · Lawrence Zhu, Pranav, Chris Paxton
[00:36:43] S1: the data... You go ahead. [00:36:44] S2: Yeah. Yeah. Essentially, like, think what you're saying is that, like, basically, it's about the motion. Right? So, like, the way that people do it, like, [00:36:51] S0: you get a lot of [00:36:51] S2: that density in the motion of how we grab onto it. And so that's why it's working better with, like, the grabbing ability.
00:36:58 · Lawrence Zhu
[00:36:58] S1: Yeah. Also, in this case, I feel like when you actually really scale the human data, right, the perspective gap actually doesn't matter because your Right. Inference time becomes a strict subset of, like, your dense demonstration. Yeah. And for this push share task, we also try to evaluate EMMA with separate manipulation and navigation models so basically train two models instead of projecting them into the shared transformer trunk and what we observe here is ing, the first stage we actually called it bimanual engagement is like you have to push the chair with two arms instead of one arm, showing some of the failure cases before, so EMMA is basically a 50 out of 50 so, and for our baseline, RoboTalio baseline with thirty minutes of robot data I think we collected around 50 mobile robot demonstrations and of course like for the minimal comparison like the 17 demo ten minute robot data it just keeps failing and I think we're right about to, I think we're at the very end of our results. I think I want to mention one thing is, in this paper, we actually performed a lot of robot experiments and the rollouts. We actually written that we, so for every single data point here, we can report the arrow bar as 95% clobber Pearson because we actually perform 50 rollout for every single task. So it's like 15 experiments for all the experiments reported in this paper. I think that's... This design twice is basically, like... First thing is we wanna follow some, like, industrial level standards, like, from the Abby... LBM pink paper, large behavior models. Yeah. And we also try to kinda we're also kinda figuring out, like, how noisy real world robot evaluation can be even for a static robotic task. Right? Because there's no way you kinda reset the scene with the exact same setup as the previous evaluation. And for our case, the mobile, the mobile, robot evaluation is actually getting much more difficult because you can't really drive your robot back to the exact same slot. Mhmm. So I think we picked the number 50. It's a painful number for, like, every single data point takes, one hour and a half. But, like, I would say, we're actually able to report this arrow bar and also perform, like, like, two proportional z tests on, like, every single time we perform. Oh, like, we claim that it outperforms speed rather than we just have, like Yeah.
00:39:23 · Lawrence Zhu, Chris Paxton, Michael Cho
20 demos, and then it will be, like, relatively noisy. [00:39:27] S0: Yeah. It makes sense. I guess you you save a lot of time not doing teleoperation, and so you spend more [00:39:31] S1: time doing real time. Time spent. [00:39:34] S3: So, yeah, it makes sense. [00:39:37] S0: I'm just... I was just thinking, like, what if, like, right now, like, this particular push chair task, you are pushing to a chair that is, you know, to to to a to a to a place that is maybe only two, three meters away. I wonder if you... What if you keep pushing the chair and become, like, a longer horizon, like, more like a navigation task. [00:39:55] S1: Yeah. That's, like, a very interesting point. [00:39:58] S0: Yeah. Because you you you emphasize about how important it was that you have this almost 100% phase change. Right? So Mhmm. Like, where does... I mean, the most extreme case I I was thinking about it is, like, I might certainly be out of school of this study. I was just wondering, like, how do we use this for... Because sometimes you are manipulating things while you are navigating. You know? [00:40:17] S1: You know? Know? Exactly. [00:40:17] S0: The most extreme case, I could be playing my Rubik's cube when I'm walking around or, you know, down to school. Used to do that. I think Right.
00:40:17 · Chris Paxton, Pranav, Lawrence Zhu
[00:40:17] S0: The most extreme case, I could be playing my Rubik's cube when I'm walking around or, you know, down to school. Used to do that. I think Right. [00:40:23] S2: So Yeah. Yeah. I think that's what we're trying to show with this push share task. Like, we can show that, like, with some minor modifications, we can do those tasks where you have both manipulation and navigation at the same time, where we don't need that phase switch to, like... So that you can... To get new Rubik's cubes while you're, like, walking around, as you're saying. [00:40:42] S1: Yeah. Actually, one hypothesis I'm drawing, I just... So, like, we haven't tested this, but I'm assuming if we collect static robot data with some sort of positional variation, and we'll actually be able to co train the mobile human data and the static robot data to actually perform these sort of simultaneous rollouts but the limitation here I would assume would be the inference frequency because as you can see, if we co train this with static robot data, so you're basically having a future action chunk of grasping the chair like that. But at the same time, you're also moving your robot base. So that means you have to inference really frequently to adapt to the new observation that actually helps you to kind of correct your hands back to the correct conditions so that'll be actually some interesting topic to kind of work on future. [00:41:32] S0: Yeah. Yeah. That's interesting. Because, like, you you do have to move the base in order to even get that manipulation done. It's it's still [00:41:39] S1: Exactly. Yeah.
00:41:39 · Lawrence Zhu, Chris Paxton
[00:41:39] S1: Exactly. Yeah. [00:41:40] S0: Yeah. There's a little bit of navigation, however small that is. I actually do do want to... If you can cycle back a few slides, you you have all these, like Yeah. Charts about the different success rate. Yep. I noticed they they do have maybe different... Yeah. Before this. Mhmm. I think all the different tasks that you have, you were comparing... Yeah. It's it's an earlier slide. [00:42:09] S1: The big one. Right? [00:42:10] S0: Yeah. This one. This one. Yeah. Yep. Could you comment a little bit more, like, what... Like like, why is grocery shopping, you know, like, you know, seems pretty good and then, you know, stuff utensils. Like, I... I'm curious, like, what's your intuition about, like, the relative, like, success rate between these?