RoboPapers
00:06:32 · Hongyi Jing
[00:06:32] S3: Yeah. Yeah. So so after collecting all this humanoid everyday data, we then decide to bring some policies, including, like, imitation learning method and the VOA models to test if our data can support can support our unit three g one robot to do this manipulation task. So you can see here all these videos. They are all the examples of our autonomous policies on on this humanoid robots. And so so these videos, they all represent, like, different categories in our data. Like, the first one is rotating the chair, which is the articulated manipulation. The second one is to folding this towel, which is the deformable manipulation. And the third one is inserting the flower into the bus, which is the high precision task. And then the robot comes closer to the door and tries to grab the handle, and we we make this the local manipulation task. And the fifth one, the the robot just tries to use the eraser to erase this task, and we call this the tool use task. And the final one, it is like a person hands over the dumpling toy to the robot, and the robot then takes it and places it on the desk. And we would limit the human robot interaction task. And and, Jing Yu, can you go to the next slide? Yeah. So we then test all these imitation learning methods on these seven task categories, and we run this benchmark, and and the results are shown here. And and first of all, we can see that on average, all these VOA models with pretrained priors, they output... Outperform all those imitation learning policies, like the open VOA, the open type policies, and the GR00T policies. They all have a better performance than the traditional imitation learning ones, like the DP, DP3, and the ACT ones here.
00:08:31 · Michael Cho, Hongyi Jing, Chris Paxton
[00:08:31] S0: Mhmm. Mhmm. [00:08:31] S3: And, also, we found that the GR00T n 1.5 is the best policy among all these imitation learning methods because we think that the GR00T n 1.5 for pretrained on a lot of humanoid data, and we think such a humanoid pretrained prior can give us much more valuable information and can help us in this robot learning. And finally, we found that, like, all the policies actually perform very poorly on those high difficulty manipulation tasks, like the high precision task here, which is inserting inserting the rows into the bus. And we can see that only OpenVLA have a success rate with only, like, a 10% success rate here. So we found that data from the robot arms and... Which is only has, like, seven action dimensions. But in our humanoid settings, actually, the humanoid has an action dimension of, like, 28, which is rather different from the robot arms. [00:10:46] S0: Yeah. Yeah. Actually, I have a... Yeah. Before you move on to this... The eval part, yeah, I have so many questions. Right? So I I guess, firstly, how many g ones did you break in the process of doing all this? It sounds like, yeah, like, how many g ones? And and I I guess also, like, how long did it take for you guys to collect all this data? It sounds like a hackingless effort. [00:11:08] S6: Yeah. Because 3,000,000 frames is, like, thirty hours of data. That must have taken, like, a really... Right? Is that... Sorry. Go. Yeah.
00:11:08 · Chris Paxton, Yue Wang, Michael Cho, Hongyi Jing
[00:11:08] S6: Yeah. Because 3,000,000 frames is, like, thirty hours of data. That must have taken, like, a really... Right? Is that... Sorry. Go. Yeah. [00:11:15] S1: Yeah. I mean, a huge effort. So we started this project early January, and has been almost nine months to finish the whole data collection, the model training, everything with a couple of students, with Zhenyu, Hongyi, Xiawei, who were, like, fully committed to the project. And, also, we hired additional students to help us collect the data. We spent a lot of time on this. And in terms of number of G1s we broke, we broke, I guess, two G1s. And Unitree was kind enough to to help us fix them and or to extend the G1 for us. Yeah. That was lot of heavy lifting scene behind the scene. [00:12:01] S6: It's not so bad, actually. It's pretty good. [00:12:04] S0: So I guess... So I... In terms of just absolute number of hours, if you were to count by hours, is is it like what what Chris say? Maybe you mentioned this already, but is is it something like tens of hours? Is is that... That is dataset size now? I know it's very diverse across many category. [00:12:21] S1: How many hours do you have? Do [00:12:23] S3: have hours? I think we we have, like, probably, like, hundreds of hours, yeah, at least, to collect all the data.
00:12:23 · Hongyi Jing, Michael Cho, Zhenyu Zhao, Chris Paxton
[00:12:23] S3: have hours? I think we we have, like, probably, like, hundreds of hours, yeah, at least, to collect all the data. [00:12:30] S0: Well, so I I guess there are two numbers here. Right? One is the number of hours that is, you know, in your published dataset. And then I guess that's the second question, which is how many actual hours... Bad hours has been put in to tell you out? So those are two different things always, typically. You know? So I'm... I I think, firstly, like, how much is it in this dataset? Do you know? You know? [00:12:53] S2: So in total, we have, like, 3,000,000 frames, and it's taken at 30 hertz. So, like, 3,000,000... If you do the math, it's 3,000,000 divided by 30. [00:13:05] S6: That's about it's about thirty hours, I think. Right? Yeah. That's what I was doing the math there. [00:13:10] S0: I see. So so thirty hours. So... And then I guess... Yeah. Then... Yeah. It's around thirty hours. I just asked. It's around thirty hours. But but... Okay. But but that's, of course, like, you know, very, you know, very, very diverse, very clean, you know, like, all these modalities that you guys have. It sounds like you guys took nine months. So what would you say if you were to guess, like, you know, I don't know whether you guys even track that. Like, how many hours was... Did you... Did it take for for you guys to actually do the teleoperation in order to get this very precious, let's say, thirty hours? I mean, which is still a lot, to to be honest, in my view, but, you know, I'm just curious how how many hours did did it take you guys to do that?
00:13:10 · Michael Cho, Chris Paxton, Hongyi Jing
[00:13:10] S0: I see. So so thirty hours. So... And then I guess... Yeah. Then... Yeah. It's around thirty hours. I just asked. It's around thirty hours. But but... Okay. But but that's, of course, like, you know, very, you know, very, very diverse, very clean, you know, like, all these modalities that you guys have. It sounds like you guys took nine months. So what would you say if you were to guess, like, you know, I don't know whether you guys even track that. Like, how many hours was... Did you... Did it take for for you guys to actually do the teleoperation in order to get this very precious, let's say, thirty hours? I mean, which is still a lot, to to be honest, in my view, but, you know, I'm just curious how how many hours did did it take you guys to do that? [00:13:49] S6: Do do... [00:13:50] S3: Oh, there. Go ahead.
00:13:50 · Hongyi Jing, Xiawei Liu, Michael Cho, Chris Paxton
[00:13:50] S3: Oh, there. Go ahead. [00:13:51] S4: Yeah. So I think because we have 260 tasks, and usually, each task would take around under thirty minutes. Or, really, if we're doing high manipulate... High precision manipulation or locomotion, it would be, like, over thirty minutes, maybe forty minutes for 40 episodes. So, basically, if we do the math, let's take an average, maybe thirty five minutes times two sixty tasks. It's nine thousand one hundred minutes divided by 60. One hundred and fifty one hours. We can't do it like really twenty four hours a day, so if we do it like four, five hours a day, it's thirty days. But in reality, the thirty days are if we were to have like all the materials and all the outlines of task ready, and we just do it one by one. But in reality, we spend a lot of time really optimizing which task we're doing, and we take a lot of times, really developing the tasks so that it ensures training quality because we don't want too difficult tasks or too impossible tasks that have random... Like, too much randomness. So, yeah, we really spend a lot of time doing the planning of tasks. Yeah. [00:15:26] S0: Right. Right. [00:15:27] S6: Makes sense.