00:50:13 · Lawrence Zhu, Michael Cho
[00:50:13] S1: Yeah. I would say the real... Like, the true value of these sort of, like, wearable data collection devices is, like, in our three d space, right, it's basically shared in between the human and robot. So basically the scene representation of whatever you're looking at is actually there's no difference so what this work is showing is we put everything into like a two d image so there will be some sort of appearance gap some sort of perspective gap. But I believe in the future, when the second generation of these glasses came out, when we have like depth destination and when we have like three d vision, then that is pretty much... There's no domain gap between the human observation and the robot observation, ideally. So I would say that will pretty much help us to scale robot learning a lot. [00:51:03] S3: How much... So what do you think would happen if you spent, like, I don't know, eight hours collecting data instead of half an hour? [00:51:08] S1: Like, what kinds [00:51:09] S3: of things do you think you could do? Yeah. [00:51:11] S1: Yeah. That's a very good question, and, we're actively exploring that. So I think there will be eventually a separating point of, like, how much data you add. But we want that separating point to be as... Let's say, it's success, we want that to be, like, as high as possible. So, hopefully, like, you will learn the unified representation rather than learning, like, my human arm singer. Right? So Yeah.
00:51:11 · Lawrence Zhu, Michael Cho, Pranav
[00:51:11] S1: Yeah. That's a very good question, and, we're actively exploring that. So I think there will be eventually a separating point of, like, how much data you add. But we want that separating point to be as... Let's say, it's success, we want that to be, like, as high as possible. So, hopefully, like, you will learn the unified representation rather than learning, like, my human arm singer. Right? So Yeah. [00:51:36] S3: I guess, yeah, you'd like... You have the... So there's two dimensions here. So, like, the first one being, like, like, you could do long horizon tasks. Like, you had a video at the beginning where you walk out of a office and then go to a desk. Like, do you think you could do that with an hour's worth of data or two hours worth of data or something like that? Like, with a robot, I mean. [00:51:53] S1: Fair enough. [00:51:53] S2: Yeah. I would definitely do something like walking out of the office and stuff like that with one to two hours. In fact, you can probably do it with less than that. [00:52:01] S1: Yeah.
00:52:01 · Lawrence Zhu, Pranav, Michael Cho, Chris Paxton
[00:52:01] S1: Yeah. [00:52:01] S2: But I think as we get more and more data, like, the quality and, like, the robustness of the models will start to increase. So, like, as we've seen, like, with behavior cloning, if you, like, look at some... Like like, if you look at companies that have been, like, scaling up tons and tons of data and start training, like, behavior cloning models, they have a lot more robustness than something that, like, we train in our lab with, like, much less data. And so I believe that, like, as you have much more of this data, your policies will end up getting much more robust as well and much more deployable in the real world. [00:52:31] S3: Yep. Any... That that that makes a lot of sense. Do you have any thoughts about how... What the ideal ratio is of, like, robot data to real data to whatever? Or can it stay fixed? Can you just need that 10? That's all you... That ten minutes. That's all you ever need. [00:52:44] S0: Yeah. Three one one. My exact question too. What's a code training mix? Yeah. What's your suspicion? [00:52:50] S3: It's it's [00:52:51] S0: Would it be 100 to one?
00:52:51 · Chris Paxton, Lawrence Zhu, Michael Cho
[00:52:51] S0: Would it be 100 to one? [00:52:54] S1: For like, code training, maybe I would say... Because there are actually different designs. For example, you can code train, you can pre train on a robot and fine train on a human, or pre train on human and fine train on a robot. But I think I think for co training, what we've been working on is around a one two three ratio because that's, like, the exact ratio of, like, how much faster human data collection compared to, like, robot data collection. But if we're talking about, like, yeah, how... If we're training a foundation model level of data, right, then it's really... I I don't really have any thoughts on this. [00:53:35] S3: But So sorry. Is that one to three, like, the per batch training ratio or the dataset rate coaching ratio or both? [00:53:42] S1: Dataset, the dataset contribution. So the per batch ratio, we strictly use one to one. Yeah.
00:53:42 · Lawrence Zhu, Pranav, Michael Cho
[00:53:42] S1: Dataset, the dataset contribution. So the per batch ratio, we strictly use one to one. Yeah. [00:53:47] S2: I actually really do like this question too because it's... It reminds me of something we were talking about in CORAL too. And it's like, is human data meant to replace robot data or is it meant to like extend robot data? Right? And so I think a big consensus is that like human data is not supposed to just completely outright replace robot data. It's supposed to like add to it. You can, like, generalize new situations or you can, like, do do things that are much more difficult to to do with the robot. And so you really, like, extend and, like... It's like if this is, like... If you have, a circle of, like, your robot data, it can be at, like, the edge and, like, add to more difficult situations like that. And so I think that's a really... I think that's probably the best place that human data can be used for. [00:54:31] S3: It's included. Yeah.
00:54:31 · Michael Cho, Lawrence Zhu
[00:54:31] S3: It's included. Yeah. [00:54:33] S1: Also, if we talk about one extension of, like, because we said, like, one of our acknowledged limitation is, like, by directly transferring, we can only do, like, manipulation, navigation switches. That's because, like, we don't have robot data during the navigation phase, right? So especially when we use ground control, you don't have such data, then how would you kind of deploy your commands? So in this case, I think recently there have been works for... There are works in data engine. For example, there's Phantom, there's Masquerade. And so what they're trying to do is they try to kind of augment human data to look like robots, right? I would say that's a very interesting direction to explore as well. So to, like, make, synthetic robot data from human demonstrations to, like, further mitigate this, visual gap. And, ideally, can... Like, we can actually also, like, solve inverse kinematics or, like, just buy the data engine solution. We can actually get the join commands of, like, performing the same, human motion or, like, the same object trajectory from, any robot environment. I'll say that would be some pretty cool direction to go as well. [00:55:50] S3: You think... Does that just add another thing into your co training mix? And now I have real data, robot data, synthetic data... Or, sorry, human data, [00:55:56] S1: which is synthetic data. Okay. From beta, it's always the better the... Yeah. Yeah. Yeah. It's always the more, the better. Yeah.