RoboPapers
00:08:31 · Michael Cho, Hongyi Jing, Chris Paxton
[00:08:31] S0: Mhmm. Mhmm. [00:08:31] S3: And, also, we found that the GR00T n 1.5 is the best policy among all these imitation learning methods because we think that the GR00T n 1.5 for pretrained on a lot of humanoid data, and we think such a humanoid pretrained prior can give us much more valuable information and can help us in this robot learning. And finally, we found that, like, all the policies actually perform very poorly on those high difficulty manipulation tasks, like the high precision task here, which is inserting inserting the rows into the bus. And we can see that only OpenVLA have a success rate with only, like, a 10% success rate here. So we found that data from the robot arms and... Which is only has, like, seven action dimensions. But in our humanoid settings, actually, the humanoid has an action dimension of, like, 28, which is rather different from the robot arms. [00:10:46] S0: Yeah. Yeah. Actually, I have a... Yeah. Before you move on to this... The eval part, yeah, I have so many questions. Right? So I I guess, firstly, how many g ones did you break in the process of doing all this? It sounds like, yeah, like, how many g ones? And and I I guess also, like, how long did it take for you guys to collect all this data? It sounds like a hackingless effort. [00:11:08] S6: Yeah. Because 3,000,000 frames is, like, thirty hours of data. That must have taken, like, a really... Right? Is that... Sorry. Go. Yeah.
00:11:08 · Chris Paxton, Yue Wang, Michael Cho, Hongyi Jing
[00:11:08] S6: Yeah. Because 3,000,000 frames is, like, thirty hours of data. That must have taken, like, a really... Right? Is that... Sorry. Go. Yeah. [00:11:15] S1: Yeah. I mean, a huge effort. So we started this project early January, and has been almost nine months to finish the whole data collection, the model training, everything with a couple of students, with Zhenyu, Hongyi, Xiawei, who were, like, fully committed to the project. And, also, we hired additional students to help us collect the data. We spent a lot of time on this. And in terms of number of G1s we broke, we broke, I guess, two G1s. And Unitree was kind enough to to help us fix them and or to extend the G1 for us. Yeah. That was lot of heavy lifting scene behind the scene. [00:12:01] S6: It's not so bad, actually. It's pretty good. [00:12:04] S0: So I guess... So I... In terms of just absolute number of hours, if you were to count by hours, is is it like what what Chris say? Maybe you mentioned this already, but is is it something like tens of hours? Is is that... That is dataset size now? I know it's very diverse across many category. [00:12:21] S1: How many hours do you have? Do [00:12:23] S3: have hours? I think we we have, like, probably, like, hundreds of hours, yeah, at least, to collect all the data.
00:12:23 · Hongyi Jing, Michael Cho, Zhenyu Zhao, Chris Paxton
[00:12:23] S3: have hours? I think we we have, like, probably, like, hundreds of hours, yeah, at least, to collect all the data. [00:12:30] S0: Well, so I I guess there are two numbers here. Right? One is the number of hours that is, you know, in your published dataset. And then I guess that's the second question, which is how many actual hours... Bad hours has been put in to tell you out? So those are two different things always, typically. You know? So I'm... I I think, firstly, like, how much is it in this dataset? Do you know? You know? [00:12:53] S2: So in total, we have, like, 3,000,000 frames, and it's taken at 30 hertz. So, like, 3,000,000... If you do the math, it's 3,000,000 divided by 30. [00:13:05] S6: That's about it's about thirty hours, I think. Right? Yeah. That's what I was doing the math there. [00:13:10] S0: I see. So so thirty hours. So... And then I guess... Yeah. Then... Yeah. It's around thirty hours. I just asked. It's around thirty hours. But but... Okay. But but that's, of course, like, you know, very, you know, very, very diverse, very clean, you know, like, all these modalities that you guys have. It sounds like you guys took nine months. So what would you say if you were to guess, like, you know, I don't know whether you guys even track that. Like, how many hours was... Did you... Did it take for for you guys to actually do the teleoperation in order to get this very precious, let's say, thirty hours? I mean, which is still a lot, to to be honest, in my view, but, you know, I'm just curious how how many hours did did it take you guys to do that?
00:13:10 · Michael Cho, Chris Paxton, Hongyi Jing
[00:13:10] S0: I see. So so thirty hours. So... And then I guess... Yeah. Then... Yeah. It's around thirty hours. I just asked. It's around thirty hours. But but... Okay. But but that's, of course, like, you know, very, you know, very, very diverse, very clean, you know, like, all these modalities that you guys have. It sounds like you guys took nine months. So what would you say if you were to guess, like, you know, I don't know whether you guys even track that. Like, how many hours was... Did you... Did it take for for you guys to actually do the teleoperation in order to get this very precious, let's say, thirty hours? I mean, which is still a lot, to to be honest, in my view, but, you know, I'm just curious how how many hours did did it take you guys to do that? [00:13:49] S6: Do do... [00:13:50] S3: Oh, there. Go ahead.
00:13:50 · Hongyi Jing, Xiawei Liu, Michael Cho, Chris Paxton
[00:13:50] S3: Oh, there. Go ahead. [00:13:51] S4: Yeah. So I think because we have 260 tasks, and usually, each task would take around under thirty minutes. Or, really, if we're doing high manipulate... High precision manipulation or locomotion, it would be, like, over thirty minutes, maybe forty minutes for 40 episodes. So, basically, if we do the math, let's take an average, maybe thirty five minutes times two sixty tasks. It's nine thousand one hundred minutes divided by 60. One hundred and fifty one hours. We can't do it like really twenty four hours a day, so if we do it like four, five hours a day, it's thirty days. But in reality, the thirty days are if we were to have like all the materials and all the outlines of task ready, and we just do it one by one. But in reality, we spend a lot of time really optimizing which task we're doing, and we take a lot of times, really developing the tasks so that it ensures training quality because we don't want too difficult tasks or too impossible tasks that have random... Like, too much randomness. So, yeah, we really spend a lot of time doing the planning of tasks. Yeah. [00:15:26] S0: Right. Right. [00:15:27] S6: Makes sense.
00:15:27 · Chris Paxton, Yue Wang
[00:15:27] S6: Makes sense. [00:15:27] S1: And as Xiawei mentioned, so doing the task maybe only take a couple of hundred hours, less than two hundred hours. But a lot of time, we need to design the task, and, also, we need to re... Redo a lot of tasks if the data quality is not high. And, also, when we have different people collecting the data, the quality is hard to control. So we need to do a lot of quality control, quality assurance afterward, after we see the data. So that actually takes much more data... Much more time. And at the same time, a lot of time, when we break the when we broke the robot or when something happens with the finger or with the arms, fixing those hours also took a lot of time. So that's why the project took almost nine months to finish. Yeah. And Yeah. Also, when we, like, trimmed the policy and we found that, okay, some tasks are hard to learn, sometimes skills are easier to learn. And we may need to rethink the task, and we may need to recollect the data. So coupling everything together, that takes a very long time.