00:45:45 · Lawrence Zhu, Chris Paxton
[00:45:45] S1: Yeah. For the the one... Yep. Sorry. For, you mean for the navigation part, for the, this one? [00:45:53] S0: Yeah. It's it's... No. Yeah. So if you see the charts, right? So what was curious to me was that, for example, handover wine, grocery shopping, like when you go to navigation, literally the success rate didn't drop. That means navigation was basically a 100%. Right? Yeah. So... Perfect. But then, you know, in the case of surf utensils... So that's why I was curious, like, why when you... Like, that navigation suck and then the other one. But but I think what your explanation, which is surf utensils, it's actually a pretty long horizon navigation task. So Yeah. And that's why it's kind of failing still. [00:46:21] S1: And I think for both handover wine and grocery shopping, we basically... At the start of navigation, I will say the... There are... Of course, as Pranav said, we have dense visual feature that helps navigation. But I think another another, like, justification is that the goal was relatively thin at the early stage of navigation. Unlike, like, servo utensil and servo quasong, you have, like, it's pretty much a 135 degree turn followed by a 90 degree turn. So you don't really see your goal points. The... During your policy inference, you have to, like, figuring out what's the correct future viewpoints from, like, very limited visual features. [00:47:02] S0: Right. And and I suppose if this environment becomes a lot more dynamic, do you think the navigation will start to suffer?
00:47:02 · Chris Paxton, Lawrence Zhu, Pranav
[00:47:02] S0: Right. And and I suppose if this environment becomes a lot more dynamic, do you think the navigation will start to suffer? [00:47:13] S1: I think it depends on whether it is included in the demonstration. Right? Because, yeah, we're basically a behavior cloning pipeline. And we actually tried in the handoverwind task when the human recipient actually moves around. It... It's just gonna keep following until, like, it reaches out to the distance that's close enough to human that, like, demo... That that is demoed in the human data that you should perform handover one. It actually generalized really well for the handover one task. I think it just extracts the feature of, like, a human shape and then some sort of a lag in the screen, and then it just learns the general pattern to just go to the human, I think. [00:47:56] S2: Yeah. It was fun because we once had, like, a couple of people who were, like, visiting our lab, and they were interested in, like, seeing a demonstration. So we're like, oh, we'll do, like, handover wine. And so there's, like, a seven year old kid who was just standing there, and then, like, we're like, oh, is it gonna generalize to him? And I was actually able to hand over the wine to him. Like [00:48:12] S1: Yeah. Because, yeah, the fish sandwich on the mug. [00:48:17] S0: Yeah. Too too too too young to drink. Yeah. Yeah. [00:48:21] S1: We were a little bit scared because, apparently, like, the visual looked different.
00:48:21 · Lawrence Zhu, Chris Paxton
[00:48:21] S1: We were a little bit scared because, apparently, like, the visual looked different. [00:48:27] S0: Yeah. That's fascinating. I mean, I'm I'm curiously interested in in... Because navigation is is a very different thing from manipulation. And especially real world, like, in the wild navigation, like, I think the tough part about it is, like, you have, you know, autonomous agents, humans, you know, usually also walking around. So [00:48:46] S1: Exactly. How [00:48:47] S0: they move around you. It's it's a... So yeah but I would say certainly though like what is is also very convincing here is just in the manipulation part of this pipeline like it definitely having the human data just just just just augment by default a lot more. I I... It seems across all the tasks, if it's just manipulation before it starts navigating, it's it's all have a have a significant jump. Yeah. Yeah. Yeah. So how do you guys think, like, long term having, like, this, like, hopefully, cheapest smart glasses or whatever, eco centric video data collection things... Devices will will help improve the performance of of this? Yeah. [00:49:30] S1: Yeah. Pranav, do you wanna Yeah. Talk about
00:49:30 · Lawrence Zhu, Pranav
[00:49:30] S1: Yeah. Pranav, do you wanna Yeah. Talk about [00:49:33] S2: I think as we have, like, smart glasses, like, they become more popular, everyone starts wearing them and everything. They're starting to, like, walk around with them. I think it'll be very... It'll be much more easy to scale up data. Right? Because right now, if you wanna have a robot policy doing something, you go to the robot, collect tons and tons of data, and train a model, and then deploy on the robot. But then if you have all of this data readily available, then it's much more easy to then train a policy on a robot and then transfer it from human to robot and then have your robot deploy in a setting that maybe humans are already in. [00:50:13] S1: Yeah. I would say the real... Like, the true value of these sort of, like, wearable data collection devices is, like, in our three d space, right, it's basically shared in between the human and robot. So basically the scene representation of whatever you're looking at is actually there's no difference so what this work is showing is we put everything into like a two d image so there will be some sort of appearance gap some sort of perspective gap. But I believe in the future, when the second generation of these glasses came out, when we have like depth destination and when we have like three d vision, then that is pretty much... There's no domain gap between the human observation and the robot observation, ideally. So I would say that will pretty much help us to scale robot learning a lot.
00:50:13 · Lawrence Zhu, Michael Cho
[00:50:13] S1: Yeah. I would say the real... Like, the true value of these sort of, like, wearable data collection devices is, like, in our three d space, right, it's basically shared in between the human and robot. So basically the scene representation of whatever you're looking at is actually there's no difference so what this work is showing is we put everything into like a two d image so there will be some sort of appearance gap some sort of perspective gap. But I believe in the future, when the second generation of these glasses came out, when we have like depth destination and when we have like three d vision, then that is pretty much... There's no domain gap between the human observation and the robot observation, ideally. So I would say that will pretty much help us to scale robot learning a lot. [00:51:03] S3: How much... So what do you think would happen if you spent, like, I don't know, eight hours collecting data instead of half an hour? [00:51:08] S1: Like, what kinds [00:51:09] S3: of things do you think you could do? Yeah. [00:51:11] S1: Yeah. That's a very good question, and, we're actively exploring that. So I think there will be eventually a separating point of, like, how much data you add. But we want that separating point to be as... Let's say, it's success, we want that to be, like, as high as possible. So, hopefully, like, you will learn the unified representation rather than learning, like, my human arm singer. Right? So Yeah.
00:51:11 · Lawrence Zhu, Michael Cho, Pranav
[00:51:11] S1: Yeah. That's a very good question, and, we're actively exploring that. So I think there will be eventually a separating point of, like, how much data you add. But we want that separating point to be as... Let's say, it's success, we want that to be, like, as high as possible. So, hopefully, like, you will learn the unified representation rather than learning, like, my human arm singer. Right? So Yeah. [00:51:36] S3: I guess, yeah, you'd like... You have the... So there's two dimensions here. So, like, the first one being, like, like, you could do long horizon tasks. Like, you had a video at the beginning where you walk out of a office and then go to a desk. Like, do you think you could do that with an hour's worth of data or two hours worth of data or something like that? Like, with a robot, I mean. [00:51:53] S1: Fair enough. [00:51:53] S2: Yeah. I would definitely do something like walking out of the office and stuff like that with one to two hours. In fact, you can probably do it with less than that. [00:52:01] S1: Yeah.