00:52:51 · Chris Paxton, Lawrence Zhu, Michael Cho
[00:52:51] S0: Would it be 100 to one? [00:52:54] S1: For like, code training, maybe I would say... Because there are actually different designs. For example, you can code train, you can pre train on a robot and fine train on a human, or pre train on human and fine train on a robot. But I think I think for co training, what we've been working on is around a one two three ratio because that's, like, the exact ratio of, like, how much faster human data collection compared to, like, robot data collection. But if we're talking about, like, yeah, how... If we're training a foundation model level of data, right, then it's really... I I don't really have any thoughts on this. [00:53:35] S3: But So sorry. Is that one to three, like, the per batch training ratio or the dataset rate coaching ratio or both? [00:53:42] S1: Dataset, the dataset contribution. So the per batch ratio, we strictly use one to one. Yeah.
00:53:42 · Lawrence Zhu, Pranav, Michael Cho
[00:53:42] S1: Dataset, the dataset contribution. So the per batch ratio, we strictly use one to one. Yeah. [00:53:47] S2: I actually really do like this question too because it's... It reminds me of something we were talking about in CORAL too. And it's like, is human data meant to replace robot data or is it meant to like extend robot data? Right? And so I think a big consensus is that like human data is not supposed to just completely outright replace robot data. It's supposed to like add to it. You can, like, generalize new situations or you can, like, do do things that are much more difficult to to do with the robot. And so you really, like, extend and, like... It's like if this is, like... If you have, a circle of, like, your robot data, it can be at, like, the edge and, like, add to more difficult situations like that. And so I think that's a really... I think that's probably the best place that human data can be used for. [00:54:31] S3: It's included. Yeah.
00:54:31 · Michael Cho, Lawrence Zhu
[00:54:31] S3: It's included. Yeah. [00:54:33] S1: Also, if we talk about one extension of, like, because we said, like, one of our acknowledged limitation is, like, by directly transferring, we can only do, like, manipulation, navigation switches. That's because, like, we don't have robot data during the navigation phase, right? So especially when we use ground control, you don't have such data, then how would you kind of deploy your commands? So in this case, I think recently there have been works for... There are works in data engine. For example, there's Phantom, there's Masquerade. And so what they're trying to do is they try to kind of augment human data to look like robots, right? I would say that's a very interesting direction to explore as well. So to, like, make, synthetic robot data from human demonstrations to, like, further mitigate this, visual gap. And, ideally, can... Like, we can actually also, like, solve inverse kinematics or, like, just buy the data engine solution. We can actually get the join commands of, like, performing the same, human motion or, like, the same object trajectory from, any robot environment. I'll say that would be some pretty cool direction to go as well. [00:55:50] S3: You think... Does that just add another thing into your co training mix? And now I have real data, robot data, synthetic data... Or, sorry, human data, [00:55:56] S1: which is synthetic data. Okay. From beta, it's always the better the... Yeah. Yeah. Yeah. It's always the more, the better. Yeah.
00:55:56 · Lawrence Zhu, Chris Paxton, Pranav, Michael Cho
[00:55:56] S1: which is synthetic data. Okay. From beta, it's always the better the... Yeah. Yeah. Yeah. It's always the more, the better. Yeah. [00:56:01] S0: Yeah. I I... Well, I mean, just just the three, and I I think partially, like, the thing with all these smart glasses is, like, the the the test data is so much. You you can probably build a reasonable world model, like, gen thing, and then you can also now have a world model. Then you, yeah, you can do that as well. [00:56:18] S1: Yeah. [00:56:19] S2: Yeah. So... Yeah. It's funny. Like, all of these things you're mentioning is, like, a lot of the active extensions we're, like, investigating right now with our team. So, like, like, as you're saying, like, with the world model is something we're also looking into. [00:56:31] S0: Yeah. It makes sense. I mean, you have the data. It it Mhmm. Probably should just build a world model on top of this. It's quite easy to to get a lot of it. Just want a pair of smart glasses. [00:56:40] S3: How many pairs of smart glasses do you have? Oh, yeah. Sorry. It's okay. Can can you just, like, all wear them around? Like, how big of dataset can you guys collect at Georgia Tech? [00:56:48] S1: I think we are we are, like, the top four in our lab. [00:56:52] S3: So I think so. [00:56:54] S1: We do... I think I think we have, like, 10 of these. [00:56:57] S0: Yeah. 10. [00:56:58] S1: Oh, yeah. How long do [00:56:59] S0: they go for? [00:57:00] S1: Like, I You can do way [00:57:00] S3: better than [00:57:01] S1: one to three. Wait. Wait. You mean battery. Right?
00:57:01 · Lawrence Zhu, Chris Paxton, Michael Cho
[00:57:01] S1: one to three. Wait. Wait. You mean battery. Right? [00:57:03] S0: Yeah. Battery. Like, mean, if you [00:57:05] S1: press record, how long does it go for? How long does it go? So continuous rec... We don't we don't do continuous recording because that will make the data has to like, super large. Right? When you process it, it will go, like, out of RAM... Memory. So we do, like, three... We usually do, like, three, four minutes per episode. And for EMMA, we did, like, think, five minute episode per demo, and then we'll just rerecord. But the data will last around... Not the data. Sorry. Like, the the glasses battery will last around one hour for, like, continuous data recording. And we recently invented something called data stations. Basically, we have a power station and a chair, and then we have some... We have, like, a power cable plugged into the glasses so you can collect the infinite data. [00:57:45] S0: Yeah. Yeah. This is this is how humans become cyborg. I I tell you. You can see those smart smart glasses, man. Let's plug in more. [00:57:52] S3: You know? You know? Yeah. I mean, I felt like the 10 glasses thing, though. That's like another part of the data angle. Right? Like, you said it was three times as fast to collect human data, but you have 10 times as many pairs of glasses as your robot. So it's about 30 times as fast. It's the [00:58:07] S0: I'm... Because I was gonna say that one to three is is very impressive. You guys are very good teleoperator, man. [00:58:12] S3: Yeah. That's a lot of robot data. Yep.
00:58:12 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:58:12] S3: Yeah. That's a lot of robot data. Yep. [00:58:15] S1: That's a lot of pretty fast robot data collection. So [00:58:19] S3: Yep. [00:58:20] S0: Yeah. I guess maybe, you know, just to wrap things up, like, you know, and any any other... We... I guess we we talk a fair bit about, like, you know, like, it's like a extension of work. Like, what what is possible? But what do you think are some limitations that, you know, that you just wanna highlight with with this work or maybe even going forward? I think, you know, Pranav, you already said that we will probably continue to need, like, some kind of robot data to augment all this huge amount of, like, smart glasses data. But any other limitations that you guys wanna highlight? [00:58:54] S1: I would say the key assumption would be, like, during a rollout, you have to be, like, looking at the direction of, like, your moving direction approximately. So it will be like a... This is, like, the first assumption we made at the beginning of this talk would be, like, a strict subset assumption. [00:59:12] S3: I had one other question that maybe is... Sorry. [00:59:15] S1: Mhmm. Go go for it. Yeah. No. Please go, Michael. Go for [00:59:17] S3: This is a slightly unfair question, I think. But it's like, how much of this work would have been made easier by you guys having an actual humanoid robot? Like, I noticed a lot of the things like the sidestepping, all of that kind of stuff. Yeah.