RoboPapers
00:27:13 · Hongyi Jing
[00:27:13] S3: Right? Actually, we we we do some ablation study here. And and, you can can you, like, switch it to the paper so we can we can talk about it? Yeah. Yeah. Yeah. So so, yeah, you can see here. So so we do this episode in this ablation experiment, we want to validate that our humanoid everyday data is really useful, can can serve as a useful prior in training those VOA models. So so, actually, when tuning those VOA models, we do this ablation study. The first the first the first baseline is that we directly tune this, like like like, I'll say, like, OpenVLA, direct tune it with new new task, like inserting the rows into the waste. And the second choice is to first tune this OpenVLA model with all the Humanoid Everyday Data, like, 260 tasks, and then tune it with this inserting the results into the waste task. So comparing these two methods can... Like, we can see that when when trained first... Pretrained first with our Humanoid Everyday Prior, we can see that our... The whole performance is actually better than the one without our Humanoid Everyday Prior. So, yeah, you can you can see it here. Like, so the... For the pi 0.5 model, if if... Without the Humanoid Everyday Prior, we can we can never get any success rate in this task. But but with our Humanoid Everyday Prior, we can achieve actually four out of 10 to succeeding this task. And, actually, for obvious VOA models, we all find the the same... You know?
00:28:51 · Michael Cho, Chris Paxton, Hongyi Jing
[00:28:51] S0: Yeah. I see. Yeah. This is really awesome. I mean, it it clearly moves the needle. I think it does make sense, but I I think the question always is, yeah, I guess how much data is needed. And, I mean... Yeah. I mean, clearly, getting even this thirty eight hours took took months of hard hard work, but then it clearly doesn't move move the needle. Right? [00:29:13] S6: It sounds like the next thirty hours would be faster if you wanted to. Hopefully, other people take up the pick up the torch, I guess. [00:29:20] S3: So Yeah. Do some. [00:29:21] S6: Because you guys used a pretty off the shelf Jiaguan. Right? Like, it's the Unitree Crippers, like, no custom hardware. Yeah. That's... Which means that's that's great because it means that other labs can more easily participate and test stuff, and I think it's a it's a really nice step. Really nice decision, I think. I did not ask about that. [00:29:37] S0: Mhmm. Sorry, Chris. Yeah. [00:29:38] S6: Yeah. Because you collected all this depth data, and I'm I'm sure that you you have plans on this one. But why... Like, none of these methods here take advantage of that at all. Right? Like, none of them take advantage of the LiDAR or the touch sensors, maybe d p three. But, like, like, like, they don't know. Right? So do you have any thoughts on that? Like, is there anything that... Do you think that you could do better by including that extra information?
00:29:58 · Hongyi Jing, Chris Paxton
[00:29:58] S3: Yeah. So so, actually, we we do do try, like like, introducing the tactile sensors into our training process, but it turns out that it actually... It's not working very well because, like... So for the g one hand, the tactile sensor has a dimension, like, about, like, 66 dimension. So... But but for the humanoid robot, actually, the action dimension is only, like, 28. So, actually, 28 is is really large for a VLA model compared to those robot arms. And if we we add this, you know, the tactile dimension into it, it... It's, like, it's nearly a 100 action dimension, so so that's really difficult for us to train on those VOA models. So, actually, later on, we we want to, like, solve this problem by introducing, like, the action tokenizer. Like, we can actually tokenize, like... For example, for a horizon of, like... For for a horizon of, like, six by 28 humanoid action horizon, like, we can tokenize it into, like, only 20 tokens. So in this way, it would be easier for the v v... VOA models to output the correct humanoid actions, and we are still still, you know, trying on this method. And and, hopefully, we can get some better results in the future, but that's our that's our future directions in this. [00:31:20] S6: Yeah. So... Cool. But I'm sorry. Why would why would the tactile sensor increase the action dimension?
00:31:20 · Chris Paxton, Hongyi Jing, Michael Cho
[00:31:20] S6: Yeah. So... Cool. But I'm sorry. Why would why would the tactile sensor increase the action dimension? [00:31:26] S3: Yeah. Because, yeah, actually, if you are doing the prediction, so the input should be the, first is the humanoid robot action horizon, which is 28, and you have to also include the 66 tactile sensors into your model. And and I'll say, actually, in the VLA models, I think it's better, you know, to have the input and the output have the same dimension. And so if you, like, exclude the tactile dimension from the outputs, I think we we found that it's actually not working. [00:32:04] S6: Yeah. So to some extent, it's an architectural limitation. So is that Yeah. Okay. Yeah. That makes sense. [00:32:09] S0: Yeah. But I think it it does make sense to me that you guys... I mean, it's it's... I mean, if you guys are going all the work to do the tally up, might as well just collect the tactile and and a couple more modalities if you can. Eventually, the algorithm will catch up and make full use of these, So I'm glad you guys just just did that anyway. You know, someone in the community can figure that out. But actually, maybe that's a good segue. I I know you guys have this really cool thing that you guys wanna maybe open up to the community as well, some kind of, like, eval as a service in the real world. Do do you guys wanna talk about that? I I noticed that's the next slide. Yeah. Oh, yeah. Mhmm.
00:32:42 · Xiawei Liu, Michael Cho
[00:32:42] S4: Yeah. So we made this cloud evaluation platform basically because we think that most of the researchers right now, even though they could train models and get data on their computer running, but it's really hard for them to get hands on with real hardware, for example, the real G1 robot, because the cost is really expensive and there's limited access to it. So we made this website so that the user could train their model on their own computer and connect to our server through a a server using the... Just the server IP and port. And then, basically, it becomes a back and forth sending with their inference action and our pitch... Our picture and the robot state. Basically, we would send the RGB image, depth view, and robot states to the user, and the client would use these informations to do inference and send us back the next action that they want the robot to perform. And we've done a lot of local tests right here in our lab, and it turns out to be performing pretty good with connection. And it has a, you know, a strong connection for user and client because... So that we... Doesn't lose important frames. Yeah. [00:34:33] S0: So so what you're doing here, you... What you say is that the the actual inference, is it done... I mean, is it done on someone else, you know, GPU in a cloud? Or are you downloading... Oh, I see. Wow. That's crazy.
00:34:33 · Michael Cho, Xiawei Liu
[00:34:33] S0: So so what you're doing here, you... What you say is that the the actual inference, is it done... I mean, is it done on someone else, you know, GPU in a cloud? Or are you downloading... Oh, I see. Wow. That's crazy. [00:34:48] S4: Yeah. So Yeah. And we are actually planning on updating this so that in the future, the user could just send us a checkpoint that they created, and then we will just use that checkpoint to do local inference here. And, so it will be much more easier for the user, and, we could actually be more flexible with, when we're doing, when we'll be performing the experiments. And professor Wang also mentioned that we might be doing a hosting a a challenge with this website. So user would just train their own model and using our datasets, and we will be decide... We will be performing the checkpoints and see which trained model outperforms the others and give them a board. And...