RoboPapers
00:15:27 · Chris Paxton, Yue Wang
[00:15:27] S6: Makes sense. [00:15:27] S1: And as Xiawei mentioned, so doing the task maybe only take a couple of hundred hours, less than two hundred hours. But a lot of time, we need to design the task, and, also, we need to re... Redo a lot of tasks if the data quality is not high. And, also, when we have different people collecting the data, the quality is hard to control. So we need to do a lot of quality control, quality assurance afterward, after we see the data. So that actually takes much more data... Much more time. And at the same time, a lot of time, when we break the when we broke the robot or when something happens with the finger or with the arms, fixing those hours also took a lot of time. So that's why the project took almost nine months to finish. Yeah. And Yeah. Also, when we, like, trimmed the policy and we found that, okay, some tasks are hard to learn, sometimes skills are easier to learn. And we may need to rethink the task, and we may need to recollect the data. So coupling everything together, that takes a very long time.
00:16:34 · Michael Cho, Chris Paxton, Yue Wang
[00:16:34] S0: Yeah. I... Yeah. Completely under... Yeah. I I guess, like... Yeah. Because you you guys started in in earlier this year. I mean, I would say even now, like, body tally op is still, you know, is still not soft. Right? It's it's still very much research papers. You know, Chris and I have interviewed many authors. I'm curious, like, just doing... Having decent whole body tell you op in order for you to call it the data in the first way is nontrivial. Like, what what kind of, like, sec... Like, whole body tell you op do do you guys using? I don't know. Have have something internal or use some... One of these, I don't know, maybe clone or twist? Or what what do you guys end up using [00:17:12] S6: Mhmm. [00:17:12] S0: When you [00:17:13] S6: do the actual TileOp? Mhmm. [00:17:14] S1: Yeah. So we developed the VR based... Vision Pro based TileOp as a a... Hongyi mentioned at the beginning. So we tried to speed up the We tried to have a real time teleoperation. But, of course, at that time, there are still some limitations. For example, it's hard to control the pelvis and also the legs together because there are multiple degrees of freedoms in the same axis, so we didn't take care of that. But right now, as you know, there are... Very recently, there are many great teleoperation system coming out, like the TWIST2 or the one that was just released of Sonic. By Unitree. Yeah. Sonic is another one. And [00:17:55] S0: then Unitree is... Yeah.
00:17:55 · Michael Cho, Yue Wang, Hongyi Jing
[00:17:55] S0: then Unitree is... Yeah. [00:17:56] S1: Mhmm. Yeah. Unitree has a one. I I think Unitree hasn't really, like, open sourced their code. And, also, today, the NVIDIA releases Sonic. But still, I will see Teleop is, as you mentioned, is not a solved problem. It's still quite hard. And, also, if you want to add the dextro has to Teleoperation, so that is even harder. And we don't have a good device or a combination of multiple devices to really capture the whole body vision... Whole body emotion with the Dexter has with everything. [00:18:27] S0: So am I right to say that right now, the way you are... You do it is is using some kind of VR headset, but then the teleoperator just put it on the neck. Then... And then it's mostly whole body... Upper body, and then the lower body, the man... Magnification is using some of the choice. It's it's a kind of policy. Right? The upper... Okay. Right. [00:18:45] S1: Right. Right. [00:18:45] S3: Right. Yeah. [00:18:46] S1: The low body is a user of the shelf for unit fees policy. [00:18:50] S0: I'm also curious to to ask a follow-up. You you mentioned something about... You guys did a lot of, like, so called data cleaning. What are the factors that you consider when you say, okay. This is not a good enough data collection. Like like, what do you look for when you look at an episode? Okay. This is not good enough. Gonna redo again.
00:18:50 · Michael Cho, Zhenyu Zhao
[00:18:50] S0: I'm also curious to to ask a follow-up. You you mentioned something about... You guys did a lot of, like, so called data cleaning. What are the factors that you consider when you say, okay. This is not a good enough data collection. Like like, what do you look for when you look at an episode? Okay. This is not good enough. Gonna redo again. [00:19:09] S2: Yeah. So so initially, our data collection pipeline, there were like some, little bugs because we have so many modalities. So, sometimes when you, when we are working on, you know, just taking the data, we we, you know, like at the time, during during the process, we we, you know, didn't really pay much attention to what... Like, we didn't, like, examine each episode after we... We've taken them. So for example, sometimes when we didn't, like, gracefully shut down the previous data collection, that there were, like, you know, like like, some, like, resource leakage due to the the fact that we used shared memories. So it it could cause sometimes maybe, like, the depth is, was not being recorded. So we had to, like, redo the the whole episode or or even the whole task if that was, and also and also sometimes, the camera view, the ego view does not, you know, see the hands really well. For example, sometimes, the the... I I think that's also one of the limitations of the g one robot because the real sense [00:20:18] S0: have an actuated neck.
00:20:18 · Michael Cho, Zhenyu Zhao, Chris Paxton
[00:20:18] S0: have an actuated neck. [00:20:19] S2: Use... Has a very narrow field of view. So a lot of times when when we, like, extend the arms to grab, something that's far away, the camera does not really see it that far, and we had to, like, also get rid of those. [00:20:36] S0: I see. I see. [00:20:38] S6: About how much... So... And also there's like operator error, like mistakes. [00:20:42] S2: Yeah. Exactly. [00:20:43] S6: About how much data did you collect versus include in the dataset? If you could get... If you know the answer to that. [00:20:49] S2: We we got rid of a lot of them. Maybe... I think maybe Xiawei can answer that.
00:20:49 · Zhenyu Zhao, Xiawei Liu, Chris Paxton, Michael Cho
[00:20:49] S2: We we got rid of a lot of them. Maybe... I think maybe Xiawei can answer that. [00:20:54] S4: Yeah. I think the main data that we abandoned were at the beginning of the project. I would say maybe around 20 to 30% of the data, we abandoned them because they were useless data with wrong timestamps or even wrong frames. But that was mainly just at the beginning. But later on, when we started to, you know, have some experience with data collection, we would actually plan it out, like, before we start taking the video. We... For example, me as the teleoperator, I would be thinking, oh, if we're doing this task called inserting a flower into the vase, it means that we need to take... Pick up the flower and insert it. And and since the g one's robot has very narrow field of view, I need to be aware that when I'm taking the data, I need to ensure the flower doesn't get out of the field of view, the camera view, so... To ensure the data quality. So, basically, it really is like a experience based tasks. So it requires the operator to know what they're doing from the robot's perspective, but not from their own eyes' perspective. So [00:22:25] S6: Makes sense. [00:22:26] S0: Yeah. I'm curiously... On the call task. Yes. I'm sorry, please. [00:22:30] S6: Oh, it's just because you mentioned the vase task. I was gonna ask, like, how... What was your success rate, Xiawei, at that? Because the the the models all really struggled at it.