00:48:21 · Lawrence Zhu, Chris Paxton
[00:48:21] S1: We were a little bit scared because, apparently, like, the visual looked different. [00:48:27] S0: Yeah. That's fascinating. I mean, I'm I'm curiously interested in in... Because navigation is is a very different thing from manipulation. And especially real world, like, in the wild navigation, like, I think the tough part about it is, like, you have, you know, autonomous agents, humans, you know, usually also walking around. So [00:48:46] S1: Exactly. How [00:48:47] S0: they move around you. It's it's a... So yeah but I would say certainly though like what is is also very convincing here is just in the manipulation part of this pipeline like it definitely having the human data just just just just augment by default a lot more. I I... It seems across all the tasks, if it's just manipulation before it starts navigating, it's it's all have a have a significant jump. Yeah. Yeah. Yeah. So how do you guys think, like, long term having, like, this, like, hopefully, cheapest smart glasses or whatever, eco centric video data collection things... Devices will will help improve the performance of of this? Yeah. [00:49:30] S1: Yeah. Pranav, do you wanna Yeah. Talk about
00:49:30 · Lawrence Zhu, Pranav
[00:49:30] S1: Yeah. Pranav, do you wanna Yeah. Talk about [00:49:33] S2: I think as we have, like, smart glasses, like, they become more popular, everyone starts wearing them and everything. They're starting to, like, walk around with them. I think it'll be very... It'll be much more easy to scale up data. Right? Because right now, if you wanna have a robot policy doing something, you go to the robot, collect tons and tons of data, and train a model, and then deploy on the robot. But then if you have all of this data readily available, then it's much more easy to then train a policy on a robot and then transfer it from human to robot and then have your robot deploy in a setting that maybe humans are already in. [00:50:13] S1: Yeah. I would say the real... Like, the true value of these sort of, like, wearable data collection devices is, like, in our three d space, right, it's basically shared in between the human and robot. So basically the scene representation of whatever you're looking at is actually there's no difference so what this work is showing is we put everything into like a two d image so there will be some sort of appearance gap some sort of perspective gap. But I believe in the future, when the second generation of these glasses came out, when we have like depth destination and when we have like three d vision, then that is pretty much... There's no domain gap between the human observation and the robot observation, ideally. So I would say that will pretty much help us to scale robot learning a lot.
00:50:13 · Lawrence Zhu, Michael Cho
[00:50:13] S1: Yeah. I would say the real... Like, the true value of these sort of, like, wearable data collection devices is, like, in our three d space, right, it's basically shared in between the human and robot. So basically the scene representation of whatever you're looking at is actually there's no difference so what this work is showing is we put everything into like a two d image so there will be some sort of appearance gap some sort of perspective gap. But I believe in the future, when the second generation of these glasses came out, when we have like depth destination and when we have like three d vision, then that is pretty much... There's no domain gap between the human observation and the robot observation, ideally. So I would say that will pretty much help us to scale robot learning a lot. [00:51:03] S3: How much... So what do you think would happen if you spent, like, I don't know, eight hours collecting data instead of half an hour? [00:51:08] S1: Like, what kinds [00:51:09] S3: of things do you think you could do? Yeah. [00:51:11] S1: Yeah. That's a very good question, and, we're actively exploring that. So I think there will be eventually a separating point of, like, how much data you add. But we want that separating point to be as... Let's say, it's success, we want that to be, like, as high as possible. So, hopefully, like, you will learn the unified representation rather than learning, like, my human arm singer. Right? So Yeah.
00:51:11 · Lawrence Zhu, Michael Cho, Pranav
[00:51:11] S1: Yeah. That's a very good question, and, we're actively exploring that. So I think there will be eventually a separating point of, like, how much data you add. But we want that separating point to be as... Let's say, it's success, we want that to be, like, as high as possible. So, hopefully, like, you will learn the unified representation rather than learning, like, my human arm singer. Right? So Yeah. [00:51:36] S3: I guess, yeah, you'd like... You have the... So there's two dimensions here. So, like, the first one being, like, like, you could do long horizon tasks. Like, you had a video at the beginning where you walk out of a office and then go to a desk. Like, do you think you could do that with an hour's worth of data or two hours worth of data or something like that? Like, with a robot, I mean. [00:51:53] S1: Fair enough. [00:51:53] S2: Yeah. I would definitely do something like walking out of the office and stuff like that with one to two hours. In fact, you can probably do it with less than that. [00:52:01] S1: Yeah.
00:52:01 · Lawrence Zhu, Pranav, Michael Cho, Chris Paxton
[00:52:01] S1: Yeah. [00:52:01] S2: But I think as we get more and more data, like, the quality and, like, the robustness of the models will start to increase. So, like, as we've seen, like, with behavior cloning, if you, like, look at some... Like like, if you look at companies that have been, like, scaling up tons and tons of data and start training, like, behavior cloning models, they have a lot more robustness than something that, like, we train in our lab with, like, much less data. And so I believe that, like, as you have much more of this data, your policies will end up getting much more robust as well and much more deployable in the real world. [00:52:31] S3: Yep. Any... That that that makes a lot of sense. Do you have any thoughts about how... What the ideal ratio is of, like, robot data to real data to whatever? Or can it stay fixed? Can you just need that 10? That's all you... That ten minutes. That's all you ever need. [00:52:44] S0: Yeah. Three one one. My exact question too. What's a code training mix? Yeah. What's your suspicion? [00:52:50] S3: It's it's [00:52:51] S0: Would it be 100 to one?
00:52:51 · Chris Paxton, Lawrence Zhu, Michael Cho
[00:52:51] S0: Would it be 100 to one? [00:52:54] S1: For like, code training, maybe I would say... Because there are actually different designs. For example, you can code train, you can pre train on a robot and fine train on a human, or pre train on human and fine train on a robot. But I think I think for co training, what we've been working on is around a one two three ratio because that's, like, the exact ratio of, like, how much faster human data collection compared to, like, robot data collection. But if we're talking about, like, yeah, how... If we're training a foundation model level of data, right, then it's really... I I don't really have any thoughts on this. [00:53:35] S3: But So sorry. Is that one to three, like, the per batch training ratio or the dataset rate coaching ratio or both? [00:53:42] S1: Dataset, the dataset contribution. So the per batch ratio, we strictly use one to one. Yeah.