RoboPapers
00:32:42 · Xiawei Liu, Michael Cho
[00:32:42] S4: Yeah. So we made this cloud evaluation platform basically because we think that most of the researchers right now, even though they could train models and get data on their computer running, but it's really hard for them to get hands on with real hardware, for example, the real G1 robot, because the cost is really expensive and there's limited access to it. So we made this website so that the user could train their model on their own computer and connect to our server through a a server using the... Just the server IP and port. And then, basically, it becomes a back and forth sending with their inference action and our pitch... Our picture and the robot state. Basically, we would send the RGB image, depth view, and robot states to the user, and the client would use these informations to do inference and send us back the next action that they want the robot to perform. And we've done a lot of local tests right here in our lab, and it turns out to be performing pretty good with connection. And it has a, you know, a strong connection for user and client because... So that we... Doesn't lose important frames. Yeah. [00:34:33] S0: So so what you're doing here, you... What you say is that the the actual inference, is it done... I mean, is it done on someone else, you know, GPU in a cloud? Or are you downloading... Oh, I see. Wow. That's crazy.
00:34:33 · Michael Cho, Xiawei Liu
[00:34:33] S0: So so what you're doing here, you... What you say is that the the actual inference, is it done... I mean, is it done on someone else, you know, GPU in a cloud? Or are you downloading... Oh, I see. Wow. That's crazy. [00:34:48] S4: Yeah. So Yeah. And we are actually planning on updating this so that in the future, the user could just send us a checkpoint that they created, and then we will just use that checkpoint to do local inference here. And, so it will be much more easier for the user, and, we could actually be more flexible with, when we're doing, when we'll be performing the experiments. And professor Wang also mentioned that we might be doing a hosting a a challenge with this website. So user would just train their own model and using our datasets, and we will be decide... We will be performing the checkpoints and see which trained model outperforms the others and give them a board. And...
00:35:51 · Michael Cho
[00:35:51] S0: Yeah. Yeah. I love I love I love this. I I don't know whether you guys know of this other project called RoboArena. Yeah. But I guess you guys are doing Humanoid, which I think is Yeah. Potentially a lot more difficult operationally. Yeah. Yeah. Yeah. I'm curious, like, about the cloud inference part. Because, I mean, definitely, once you do it through the cloud, there have to be some latency. I wonder, given that there's so many dorf in Humanoid, like, what is the limit of the type of task that you can still, you know, reasonably, you know, expect the policy to be able to do? Maybe the policy could do more, but it's just that the latency is is just there, and therefore, it can't do more. Like, how do you guys think about that? You you see what I'm saying? So so imagine if if someone have a, let's say, a very, capable policy supposedly, but then if you give that... If if there's any latency, the latency compiler, because you... Ideally, you wanna stream everything 30, and then literally in ideally zero latency, then it will be the full capability of the of the policy. So I wonder in your in your testing here, when you do, like, having some of this policy in the cloud, like like, is it is it, for example, is it only doing simpler, like, desktop manipulation tasks, or can it do... Really do the full set of things a full humanoid can do?
00:37:13 · Yue Wang, Michael Cho
[00:37:13] S1: Yeah. So I think, ideally, we can do some local manipulation tasks but with controlled environments. So, again, we want to unify the evaluation, and we want to make sure the evaluation is fair and reproducible. So that's why we need to... We still need to have this, controlled, confined lab environments. We want to make sure the evaluation is fair and reproducible. So we, we need this controlled, confined lab environments. So we need to design a certain number of tasks. And we hope that tasks can be, like, hard enough for some local manipulation tasks. And, that is our ultimate goal. For open world evaluation, I think that's a a stretch goal for the future, which is, fully the ideal case for for for for all the humanoid evaluation or, algorithm evaluation tasks. [00:38:11] S0: Yeah. I I mean, I imagine it's nontrivial, especially with this. Yeah. But I'm very glad you guys are even trying this. It's... Yeah. I mean, me and Chris talk about this the time. We we like benchmarks. We like, like, you know, more like a Yeah. Neutral eval... Real world eval, especially. So... But it... I can imagine why this is a natural next step because you guys see all the work doing the teleop. Doing eval, it's it's it's it's it's a natural thing to do next as well.
00:38:11 · Michael Cho, Hongyi Jing, Chris Paxton
[00:38:11] S0: Yeah. I I mean, I imagine it's nontrivial, especially with this. Yeah. But I'm very glad you guys are even trying this. It's... Yeah. I mean, me and Chris talk about this the time. We we like benchmarks. We like, like, you know, more like a Yeah. Neutral eval... Real world eval, especially. So... But it... I can imagine why this is a natural next step because you guys see all the work doing the teleop. Doing eval, it's it's it's it's it's a natural thing to do next as well. [00:38:41] S3: So in our everyday work, actually, for the local manipulation task, we only include the very simple high level lower body movement, such as, like, making the robot moving forward or turning around. But, actually, finally, we want to train some generalizable whole body policies for, like, for more expressive local manipulation tasks. And so, like, you can see here, this... The data we collected recently, and we developed our collection pipeline to lead this expressive whole body movement. Like, the robot will actually first pick up the dumpling point and then crouches and and to place this dumpling on the chair and then stand up. Yeah. So so in the future, we want to train our other VLA models and maybe develop some more more newer models and to see if we can train some really generalizable policies on this, yeah, whole body local manipulation tasks. [00:39:36] S6: Yeah. Does that mean we should all look forward to, like, a Humanoid Everyday too next year with full body manipulation?
00:39:36 · Chris Paxton, Michael Cho, Zhenyu Zhao
[00:39:36] S6: Yeah. Does that mean we should all look forward to, like, a Humanoid Everyday too next year with full body manipulation? [00:39:41] S0: Yeah. Yeah. Yeah. Okay. Yeah. This is awesome, man. I suppose this is the last slide, is it? Yeah. Yeah. Yeah. I guess... I mean, maybe you guys already kinda alluded to a couple of piece, like... But I'm curious, like, if you were to sum up the last, maybe nine months of doing this, like, what are some of the, like, maybe surprises and and maybe main takeaways that, you know, people reading your paper should should should also get... I mean, basically, learn from your experience. Like, what what would you say are some of the very key takeaways, especially if especially if they're surprising to you? [00:40:16] S2: Yeah. Yeah. We were really surprised that how, like, you know, sometimes the diffusion policies, like... Well well, from our experience, we found that diffusion policy did not work as well as the the viewing policy three d. So, using, depth as the, one of the states, the observation, actually works better than having just raw RGB image. Yeah. Yeah. I think that's very interesting.