RoboPapers
00:02:10 · Zhenyu Zhao
[00:02:10] S2: Yeah. So here we present Humanoid Everyday, which is a comprehensive open world dataset for humanoid robot learning. So... And in this dataset, we we have over 10.3 k trajectories over 3,000,000 frames and two sixty tasks all taken using the unit tree g one and h one. And these tasks include seven categories, like basic manipulation, deformable manipulation, toy use articulated manipulation, high precision manipulation, human robot interaction, and local manipulation. And you can see... You can get a little preview of what some of the tasks that we do that are contained in our dataset. But, yeah, all of these are are, like, high frequency teleoperated data that that that we collect manually. In order to enable this dataset, we first have to... We first made huge improvement of the original data collection pipeline offered by the unitary team. They released this, you know, OpenXR collaboration system, like, like, earlier... I think in the beginning of this year. But then it was a little bit too slow in in areas such as, like, collecting large stream of data. For example, we have a lot of modalities here. We have the depth, the the LiDAR, the RGB, and the tactile sensors, and some of the proprioceptions. So so all of these have to be taken in synchronous. So, to enable that, we rewrote a lot of the data collection, pipeline. So here, you can get the basic overview of, what our data collection pipeline is doing. So it takes risk pulses from, an Apple Vision Pro that the teleoperator requires. And, so the Vision Pro tracks the the hand post... Pulses and then, transfer them to a a viewer server, which is like a, you know, WebSocket server that, you know, gets the the VR data. And, using the VR data, we can do, you know, I I k solving, and then we get the the joint poses for the humanoid robots. And after these... This process, we have already collected the the action that we want to store in the output data streams. And in the meantime, we also have the robot that's streaming, you know, ecocentric view to the teleoperator also through this viewer. And we have also another... Like, we also have the the eagle view collected here. And these... And, also, we have the LiDAR server that's separate and some of the proprioception stuff that's already internally enabled by the unitary robots.
00:05:19 · Zhenyu Zhao, Xiawei Liu
And all these streams of data are, like, collected independently. And, however, they're also all synchronized at 30 hertz. We do that through the use of... By leveraging Python shared memory, and then we were able to, like, create different processes, like separate processes for each of the stream, and we were able to collect them synchronously. Yeah. And, also, we have this data viewer, and I believe Xiawei can jump in and talk a little bit more about this. [00:05:57] S4: Yeah. Sure. So in order for users to really know what data we have and be able to, you know, scan through different categories of data and see which one we... They could use, we made a data viewer that has a selected 50 task from the seven categories, and each of the task would have a text description, a third view video, an ecocentric video, and also point cloud and depth visual... Visualization at the bottom.
00:06:32 · Hongyi Jing
[00:06:32] S3: Yeah. Yeah. So so after collecting all this humanoid everyday data, we then decide to bring some policies, including, like, imitation learning method and the VOA models to test if our data can support can support our unit three g one robot to do this manipulation task. So you can see here all these videos. They are all the examples of our autonomous policies on on this humanoid robots. And so so these videos, they all represent, like, different categories in our data. Like, the first one is rotating the chair, which is the articulated manipulation. The second one is to folding this towel, which is the deformable manipulation. And the third one is inserting the flower into the bus, which is the high precision task. And then the robot comes closer to the door and tries to grab the handle, and we we make this the local manipulation task. And the fifth one, the the robot just tries to use the eraser to erase this task, and we call this the tool use task. And the final one, it is like a person hands over the dumpling toy to the robot, and the robot then takes it and places it on the desk. And we would limit the human robot interaction task. And and, Jing Yu, can you go to the next slide? Yeah. So we then test all these imitation learning methods on these seven task categories, and we run this benchmark, and and the results are shown here. And and first of all, we can see that on average, all these VOA models with pretrained priors, they output... Outperform all those imitation learning policies, like the open VOA, the open type policies, and the GR00T policies. They all have a better performance than the traditional imitation learning ones, like the DP, DP3, and the ACT ones here.
00:08:31 · Michael Cho, Hongyi Jing, Chris Paxton
[00:08:31] S0: Mhmm. Mhmm. [00:08:31] S3: And, also, we found that the GR00T n 1.5 is the best policy among all these imitation learning methods because we think that the GR00T n 1.5 for pretrained on a lot of humanoid data, and we think such a humanoid pretrained prior can give us much more valuable information and can help us in this robot learning. And finally, we found that, like, all the policies actually perform very poorly on those high difficulty manipulation tasks, like the high precision task here, which is inserting inserting the rows into the bus. And we can see that only OpenVLA have a success rate with only, like, a 10% success rate here. So we found that data from the robot arms and... Which is only has, like, seven action dimensions. But in our humanoid settings, actually, the humanoid has an action dimension of, like, 28, which is rather different from the robot arms. [00:10:46] S0: Yeah. Yeah. Actually, I have a... Yeah. Before you move on to this... The eval part, yeah, I have so many questions. Right? So I I guess, firstly, how many g ones did you break in the process of doing all this? It sounds like, yeah, like, how many g ones? And and I I guess also, like, how long did it take for you guys to collect all this data? It sounds like a hackingless effort. [00:11:08] S6: Yeah. Because 3,000,000 frames is, like, thirty hours of data. That must have taken, like, a really... Right? Is that... Sorry. Go. Yeah.
00:11:08 · Chris Paxton, Yue Wang, Michael Cho, Hongyi Jing
[00:11:08] S6: Yeah. Because 3,000,000 frames is, like, thirty hours of data. That must have taken, like, a really... Right? Is that... Sorry. Go. Yeah. [00:11:15] S1: Yeah. I mean, a huge effort. So we started this project early January, and has been almost nine months to finish the whole data collection, the model training, everything with a couple of students, with Zhenyu, Hongyi, Xiawei, who were, like, fully committed to the project. And, also, we hired additional students to help us collect the data. We spent a lot of time on this. And in terms of number of G1s we broke, we broke, I guess, two G1s. And Unitree was kind enough to to help us fix them and or to extend the G1 for us. Yeah. That was lot of heavy lifting scene behind the scene. [00:12:01] S6: It's not so bad, actually. It's pretty good. [00:12:04] S0: So I guess... So I... In terms of just absolute number of hours, if you were to count by hours, is is it like what what Chris say? Maybe you mentioned this already, but is is it something like tens of hours? Is is that... That is dataset size now? I know it's very diverse across many category. [00:12:21] S1: How many hours do you have? Do [00:12:23] S3: have hours? I think we we have, like, probably, like, hundreds of hours, yeah, at least, to collect all the data.
00:12:23 · Hongyi Jing, Michael Cho, Zhenyu Zhao, Chris Paxton
[00:12:23] S3: have hours? I think we we have, like, probably, like, hundreds of hours, yeah, at least, to collect all the data. [00:12:30] S0: Well, so I I guess there are two numbers here. Right? One is the number of hours that is, you know, in your published dataset. And then I guess that's the second question, which is how many actual hours... Bad hours has been put in to tell you out? So those are two different things always, typically. You know? So I'm... I I think, firstly, like, how much is it in this dataset? Do you know? You know? [00:12:53] S2: So in total, we have, like, 3,000,000 frames, and it's taken at 30 hertz. So, like, 3,000,000... If you do the math, it's 3,000,000 divided by 30. [00:13:05] S6: That's about it's about thirty hours, I think. Right? Yeah. That's what I was doing the math there. [00:13:10] S0: I see. So so thirty hours. So... And then I guess... Yeah. Then... Yeah. It's around thirty hours. I just asked. It's around thirty hours. But but... Okay. But but that's, of course, like, you know, very, you know, very, very diverse, very clean, you know, like, all these modalities that you guys have. It sounds like you guys took nine months. So what would you say if you were to guess, like, you know, I don't know whether you guys even track that. Like, how many hours was... Did you... Did it take for for you guys to actually do the teleoperation in order to get this very precious, let's say, thirty hours? I mean, which is still a lot, to to be honest, in my view, but, you know, I'm just curious how how many hours did did it take you guys to do that?