RoboPapers
00:20:18 · Michael Cho, Zhenyu Zhao, Chris Paxton
[00:20:18] S0: have an actuated neck. [00:20:19] S2: Use... Has a very narrow field of view. So a lot of times when when we, like, extend the arms to grab, something that's far away, the camera does not really see it that far, and we had to, like, also get rid of those. [00:20:36] S0: I see. I see. [00:20:38] S6: About how much... So... And also there's like operator error, like mistakes. [00:20:42] S2: Yeah. Exactly. [00:20:43] S6: About how much data did you collect versus include in the dataset? If you could get... If you know the answer to that. [00:20:49] S2: We we got rid of a lot of them. Maybe... I think maybe Xiawei can answer that.
00:20:49 · Zhenyu Zhao, Xiawei Liu, Chris Paxton, Michael Cho
[00:20:49] S2: We we got rid of a lot of them. Maybe... I think maybe Xiawei can answer that. [00:20:54] S4: Yeah. I think the main data that we abandoned were at the beginning of the project. I would say maybe around 20 to 30% of the data, we abandoned them because they were useless data with wrong timestamps or even wrong frames. But that was mainly just at the beginning. But later on, when we started to, you know, have some experience with data collection, we would actually plan it out, like, before we start taking the video. We... For example, me as the teleoperator, I would be thinking, oh, if we're doing this task called inserting a flower into the vase, it means that we need to take... Pick up the flower and insert it. And and since the g one's robot has very narrow field of view, I need to be aware that when I'm taking the data, I need to ensure the flower doesn't get out of the field of view, the camera view, so... To ensure the data quality. So, basically, it really is like a experience based tasks. So it requires the operator to know what they're doing from the robot's perspective, but not from their own eyes' perspective. So [00:22:25] S6: Makes sense. [00:22:26] S0: Yeah. I'm curiously... On the call task. Yes. I'm sorry, please. [00:22:30] S6: Oh, it's just because you mentioned the vase task. I was gonna ask, like, how... What was your success rate, Xiawei, at that? Because the the the models all really struggled at it.
00:22:30 · Chris Paxton, Xiawei Liu, Michael Cho
[00:22:30] S6: Oh, it's just because you mentioned the vase task. I was gonna ask, like, how... What was your success rate, Xiawei, at that? Because the the the models all really struggled at it. [00:22:39] S4: For for Human, because we really optimize the teleoperation pipeline, so we could do very high precision task because if we are just, you know, not moving our head and just moving our hands, then the success rate would be around eighty percent, I would say. Yeah. [00:23:00] S0: And I guess, you know, you mentioned about the... A lot of it is is... Hinges on kind of the skills of of the tally operator. How, like, how's the actual experience for you guys who... Yeah. I I imagine each of you put in hours of, like, teleoperating. Right? How is it like like, it's... Is it really the case that... Or or rather how many hours of training or or practice does it... Before you guys feel that you are actually maximizing, like, pushing to the limit of what your, you know, whole body teleopset can do?
00:23:00 · Michael Cho, Xiawei Liu
[00:23:00] S0: And I guess, you know, you mentioned about the... A lot of it is is... Hinges on kind of the skills of of the tally operator. How, like, how's the actual experience for you guys who... Yeah. I I imagine each of you put in hours of, like, teleoperating. Right? How is it like like, it's... Is it really the case that... Or or rather how many hours of training or or practice does it... Before you guys feel that you are actually maximizing, like, pushing to the limit of what your, you know, whole body teleopset can do? [00:23:32] S4: I would say for teleoperators, there's a few tricks that someone could, you know, get really hands on really fast once they know it. Basically, maybe I would say a good two hour... Under two hours of training would would help, you know, students to get hands on with teleoperating. Because the really hard part with teleoperating is that since the g one's neck couldn't move, students needs to make sure their neck their neck doesn't move. Because if their hands are at the same place, if they move their neck, for g one, their hands would be moving to the other direction that the head is moving. So, yeah, it's it's really tough for students to not move their neck when they're teleoperating because if they wanna grab something, for example, on the right side of the desk, they would intentionally just, like, you know, rotate their neck, and it's really hard for them to adapt to that at first. But after one hour or two hours of training and calibrating, they would be getting used to it. Yeah.
00:23:32 · Xiawei Liu, Michael Cho
[00:23:32] S4: I would say for teleoperators, there's a few tricks that someone could, you know, get really hands on really fast once they know it. Basically, maybe I would say a good two hour... Under two hours of training would would help, you know, students to get hands on with teleoperating. Because the really hard part with teleoperating is that since the g one's neck couldn't move, students needs to make sure their neck their neck doesn't move. Because if their hands are at the same place, if they move their neck, for g one, their hands would be moving to the other direction that the head is moving. So, yeah, it's it's really tough for students to not move their neck when they're teleoperating because if they wanna grab something, for example, on the right side of the desk, they would intentionally just, like, you know, rotate their neck, and it's really hard for them to adapt to that at first. But after one hour or two hours of training and calibrating, they would be getting used to it. Yeah. [00:24:51] S0: How about... How do you guys think about the different additional sensors that you you guys put? Right? I I I noticed you guys also have a third person exocentric view, and then you you put some LiDAR. Could could you talk about, like, that process? How do you think about what, you know, modalities to collect, why they're useful, and things like that?
00:24:51 · Michael Cho, Zhenyu Zhao
[00:24:51] S0: How about... How do you guys think about the different additional sensors that you you guys put? Right? I I I noticed you guys also have a third person exocentric view, and then you you put some LiDAR. Could could you talk about, like, that process? How do you think about what, you know, modalities to collect, why they're useful, and things like that? [00:25:09] S2: So here we... For the LiDAR, it's it's really all baked in into the the G1 system. We were just trying to maximize all the modalities that we can take out of the the G1 and and the H1. So the... So they they both have the the LiDAR, yeah, at their head. [00:25:27] S0: I see. And and I suppose, like, you know, so you guys did so much work to to gather this dataset, and then you you guys then do this, like, actual testing, you know, as you show in this slide here. How has that dataset, like, you know, affected these results in your view? Is there anything surprising that you you didn't expect?