RoboPapers
00:12:23 · Hongyi Jing, Michael Cho, Zhenyu Zhao, Chris Paxton
[00:12:23] S3: have hours? I think we we have, like, probably, like, hundreds of hours, yeah, at least, to collect all the data. [00:12:30] S0: Well, so I I guess there are two numbers here. Right? One is the number of hours that is, you know, in your published dataset. And then I guess that's the second question, which is how many actual hours... Bad hours has been put in to tell you out? So those are two different things always, typically. You know? So I'm... I I think, firstly, like, how much is it in this dataset? Do you know? You know? [00:12:53] S2: So in total, we have, like, 3,000,000 frames, and it's taken at 30 hertz. So, like, 3,000,000... If you do the math, it's 3,000,000 divided by 30. [00:13:05] S6: That's about it's about thirty hours, I think. Right? Yeah. That's what I was doing the math there. [00:13:10] S0: I see. So so thirty hours. So... And then I guess... Yeah. Then... Yeah. It's around thirty hours. I just asked. It's around thirty hours. But but... Okay. But but that's, of course, like, you know, very, you know, very, very diverse, very clean, you know, like, all these modalities that you guys have. It sounds like you guys took nine months. So what would you say if you were to guess, like, you know, I don't know whether you guys even track that. Like, how many hours was... Did you... Did it take for for you guys to actually do the teleoperation in order to get this very precious, let's say, thirty hours? I mean, which is still a lot, to to be honest, in my view, but, you know, I'm just curious how how many hours did did it take you guys to do that?
00:13:10 · Michael Cho, Chris Paxton, Hongyi Jing
[00:13:10] S0: I see. So so thirty hours. So... And then I guess... Yeah. Then... Yeah. It's around thirty hours. I just asked. It's around thirty hours. But but... Okay. But but that's, of course, like, you know, very, you know, very, very diverse, very clean, you know, like, all these modalities that you guys have. It sounds like you guys took nine months. So what would you say if you were to guess, like, you know, I don't know whether you guys even track that. Like, how many hours was... Did you... Did it take for for you guys to actually do the teleoperation in order to get this very precious, let's say, thirty hours? I mean, which is still a lot, to to be honest, in my view, but, you know, I'm just curious how how many hours did did it take you guys to do that? [00:13:49] S6: Do do... [00:13:50] S3: Oh, there. Go ahead.
00:13:50 · Hongyi Jing, Xiawei Liu, Michael Cho, Chris Paxton
[00:13:50] S3: Oh, there. Go ahead. [00:13:51] S4: Yeah. So I think because we have 260 tasks, and usually, each task would take around under thirty minutes. Or, really, if we're doing high manipulate... High precision manipulation or locomotion, it would be, like, over thirty minutes, maybe forty minutes for 40 episodes. So, basically, if we do the math, let's take an average, maybe thirty five minutes times two sixty tasks. It's nine thousand one hundred minutes divided by 60. One hundred and fifty one hours. We can't do it like really twenty four hours a day, so if we do it like four, five hours a day, it's thirty days. But in reality, the thirty days are if we were to have like all the materials and all the outlines of task ready, and we just do it one by one. But in reality, we spend a lot of time really optimizing which task we're doing, and we take a lot of times, really developing the tasks so that it ensures training quality because we don't want too difficult tasks or too impossible tasks that have random... Like, too much randomness. So, yeah, we really spend a lot of time doing the planning of tasks. Yeah. [00:15:26] S0: Right. Right. [00:15:27] S6: Makes sense.
00:15:27 · Chris Paxton, Yue Wang
[00:15:27] S6: Makes sense. [00:15:27] S1: And as Xiawei mentioned, so doing the task maybe only take a couple of hundred hours, less than two hundred hours. But a lot of time, we need to design the task, and, also, we need to re... Redo a lot of tasks if the data quality is not high. And, also, when we have different people collecting the data, the quality is hard to control. So we need to do a lot of quality control, quality assurance afterward, after we see the data. So that actually takes much more data... Much more time. And at the same time, a lot of time, when we break the when we broke the robot or when something happens with the finger or with the arms, fixing those hours also took a lot of time. So that's why the project took almost nine months to finish. Yeah. And Yeah. Also, when we, like, trimmed the policy and we found that, okay, some tasks are hard to learn, sometimes skills are easier to learn. And we may need to rethink the task, and we may need to recollect the data. So coupling everything together, that takes a very long time.
00:16:34 · Michael Cho, Chris Paxton, Yue Wang
[00:16:34] S0: Yeah. I... Yeah. Completely under... Yeah. I I guess, like... Yeah. Because you you guys started in in earlier this year. I mean, I would say even now, like, body tally op is still, you know, is still not soft. Right? It's it's still very much research papers. You know, Chris and I have interviewed many authors. I'm curious, like, just doing... Having decent whole body tell you op in order for you to call it the data in the first way is nontrivial. Like, what what kind of, like, sec... Like, whole body tell you op do do you guys using? I don't know. Have have something internal or use some... One of these, I don't know, maybe clone or twist? Or what what do you guys end up using [00:17:12] S6: Mhmm. [00:17:12] S0: When you [00:17:13] S6: do the actual TileOp? Mhmm. [00:17:14] S1: Yeah. So we developed the VR based... Vision Pro based TileOp as a a... Hongyi mentioned at the beginning. So we tried to speed up the We tried to have a real time teleoperation. But, of course, at that time, there are still some limitations. For example, it's hard to control the pelvis and also the legs together because there are multiple degrees of freedoms in the same axis, so we didn't take care of that. But right now, as you know, there are... Very recently, there are many great teleoperation system coming out, like the TWIST2 or the one that was just released of Sonic. By Unitree. Yeah. Sonic is another one. And [00:17:55] S0: then Unitree is... Yeah.
00:17:55 · Michael Cho, Yue Wang, Hongyi Jing
[00:17:55] S0: then Unitree is... Yeah. [00:17:56] S1: Mhmm. Yeah. Unitree has a one. I I think Unitree hasn't really, like, open sourced their code. And, also, today, the NVIDIA releases Sonic. But still, I will see Teleop is, as you mentioned, is not a solved problem. It's still quite hard. And, also, if you want to add the dextro has to Teleoperation, so that is even harder. And we don't have a good device or a combination of multiple devices to really capture the whole body vision... Whole body emotion with the Dexter has with everything. [00:18:27] S0: So am I right to say that right now, the way you are... You do it is is using some kind of VR headset, but then the teleoperator just put it on the neck. Then... And then it's mostly whole body... Upper body, and then the lower body, the man... Magnification is using some of the choice. It's it's a kind of policy. Right? The upper... Okay. Right. [00:18:45] S1: Right. Right. [00:18:45] S3: Right. Yeah. [00:18:46] S1: The low body is a user of the shelf for unit fees policy. [00:18:50] S0: I'm also curious to to ask a follow-up. You you mentioned something about... You guys did a lot of, like, so called data cleaning. What are the factors that you consider when you say, okay. This is not a good enough data collection. Like like, what do you look for when you look at an episode? Okay. This is not good enough. Gonna redo again.