RoboPapers
00:24:51 · Michael Cho, Zhenyu Zhao
[00:24:51] S0: How about... How do you guys think about the different additional sensors that you you guys put? Right? I I I noticed you guys also have a third person exocentric view, and then you you put some LiDAR. Could could you talk about, like, that process? How do you think about what, you know, modalities to collect, why they're useful, and things like that? [00:25:09] S2: So here we... For the LiDAR, it's it's really all baked in into the the G1 system. We were just trying to maximize all the modalities that we can take out of the the G1 and and the H1. So the... So they they both have the the LiDAR, yeah, at their head. [00:25:27] S0: I see. And and I suppose, like, you know, so you guys did so much work to to gather this dataset, and then you you guys then do this, like, actual testing, you know, as you show in this slide here. How has that dataset, like, you know, affected these results in your view? Is there anything surprising that you you didn't expect?
00:25:27 · Michael Cho, Hongyi Jing
[00:25:27] S0: I see. And and I suppose, like, you know, so you guys did so much work to to gather this dataset, and then you you guys then do this, like, actual testing, you know, as you show in this slide here. How has that dataset, like, you know, affected these results in your view? Is there anything surprising that you you didn't expect? [00:25:50] S3: Yeah. So so I think, first, first of all, we we found that all the traditional imitation learning methods are are not working well on on on the majority of our humanoid data. And and we first think that we we we do not collect the the, you know, the correct data. Maybe maybe the data quality is not that high. So so we then try to, like, collect more accurate data, and we then train them with the same imitation learning method, but we we still find out that maybe maybe this imitation learning method just cannot work well on this, you know, humanoid dataset. So then when we turn to train all those VOA... Large VOA models, we found that, wow. This this OpenVLA actually works very well on the humanoid manipulation. And then we're to tune the, you know, the openpi methods, like the pi 0.5 here and the pi 0.5 model here. And we we also find that, well, actually, the robot can can, you know, do the manipulation task very well compared to the imitation learning methods. And and finally, when we try to train these good models here, we find that the NVIDIA GR00T model is the best among all these VLA models. And, yeah, I think that's what we are getting from the list training process.
00:25:50 · Hongyi Jing, Michael Cho
[00:25:50] S3: Yeah. So so I think, first, first of all, we we found that all the traditional imitation learning methods are are not working well on on on the majority of our humanoid data. And and we first think that we we we do not collect the the, you know, the correct data. Maybe maybe the data quality is not that high. So so we then try to, like, collect more accurate data, and we then train them with the same imitation learning method, but we we still find out that maybe maybe this imitation learning method just cannot work well on this, you know, humanoid dataset. So then when we turn to train all those VOA... Large VOA models, we found that, wow. This this OpenVLA actually works very well on the humanoid manipulation. And then we're to tune the, you know, the openpi methods, like the pi 0.5 here and the pi 0.5 model here. And we we also find that, well, actually, the robot can can, you know, do the manipulation task very well compared to the imitation learning methods. And and finally, when we try to train these good models here, we find that the NVIDIA GR00T model is the best among all these VLA models. And, yeah, I think that's what we are getting from the list training process. [00:27:06] S0: I'm curious. Do you have some ablation study where you you do this, like, just straight out the box? Like, can can GR00T or PIE do any of these?
00:27:13 · Hongyi Jing
[00:27:13] S3: Right? Actually, we we we do some ablation study here. And and, you can can you, like, switch it to the paper so we can we can talk about it? Yeah. Yeah. Yeah. So so, yeah, you can see here. So so we do this episode in this ablation experiment, we want to validate that our humanoid everyday data is really useful, can can serve as a useful prior in training those VOA models. So so, actually, when tuning those VOA models, we do this ablation study. The first the first the first baseline is that we directly tune this, like like like, I'll say, like, OpenVLA, direct tune it with new new task, like inserting the rows into the waste. And the second choice is to first tune this OpenVLA model with all the Humanoid Everyday Data, like, 260 tasks, and then tune it with this inserting the results into the waste task. So comparing these two methods can... Like, we can see that when when trained first... Pretrained first with our Humanoid Everyday Prior, we can see that our... The whole performance is actually better than the one without our Humanoid Everyday Prior. So, yeah, you can you can see it here. Like, so the... For the pi 0.5 model, if if... Without the Humanoid Everyday Prior, we can we can never get any success rate in this task. But but with our Humanoid Everyday Prior, we can achieve actually four out of 10 to succeeding this task. And, actually, for obvious VOA models, we all find the the same... You know?
00:28:51 · Michael Cho, Chris Paxton, Hongyi Jing
[00:28:51] S0: Yeah. I see. Yeah. This is really awesome. I mean, it it clearly moves the needle. I think it does make sense, but I I think the question always is, yeah, I guess how much data is needed. And, I mean... Yeah. I mean, clearly, getting even this thirty eight hours took took months of hard hard work, but then it clearly doesn't move move the needle. Right? [00:29:13] S6: It sounds like the next thirty hours would be faster if you wanted to. Hopefully, other people take up the pick up the torch, I guess. [00:29:20] S3: So Yeah. Do some. [00:29:21] S6: Because you guys used a pretty off the shelf Jiaguan. Right? Like, it's the Unitree Crippers, like, no custom hardware. Yeah. That's... Which means that's that's great because it means that other labs can more easily participate and test stuff, and I think it's a it's a really nice step. Really nice decision, I think. I did not ask about that. [00:29:37] S0: Mhmm. Sorry, Chris. Yeah. [00:29:38] S6: Yeah. Because you collected all this depth data, and I'm I'm sure that you you have plans on this one. But why... Like, none of these methods here take advantage of that at all. Right? Like, none of them take advantage of the LiDAR or the touch sensors, maybe d p three. But, like, like, like, they don't know. Right? So do you have any thoughts on that? Like, is there anything that... Do you think that you could do better by including that extra information?
00:29:58 · Hongyi Jing, Chris Paxton
[00:29:58] S3: Yeah. So so, actually, we we do do try, like like, introducing the tactile sensors into our training process, but it turns out that it actually... It's not working very well because, like... So for the g one hand, the tactile sensor has a dimension, like, about, like, 66 dimension. So... But but for the humanoid robot, actually, the action dimension is only, like, 28. So, actually, 28 is is really large for a VLA model compared to those robot arms. And if we we add this, you know, the tactile dimension into it, it... It's, like, it's nearly a 100 action dimension, so so that's really difficult for us to train on those VOA models. So, actually, later on, we we want to, like, solve this problem by introducing, like, the action tokenizer. Like, we can actually tokenize, like... For example, for a horizon of, like... For for a horizon of, like, six by 28 humanoid action horizon, like, we can tokenize it into, like, only 20 tokens. So in this way, it would be easier for the v v... VOA models to output the correct humanoid actions, and we are still still, you know, trying on this method. And and, hopefully, we can get some better results in the future, but that's our that's our future directions in this. [00:31:20] S6: Yeah. So... Cool. But I'm sorry. Why would why would the tactile sensor increase the action dimension?