RoboPapers
00:34:33 · Michael Cho, Xiawei Liu
[00:34:33] S0: So so what you're doing here, you... What you say is that the the actual inference, is it done... I mean, is it done on someone else, you know, GPU in a cloud? Or are you downloading... Oh, I see. Wow. That's crazy. [00:34:48] S4: Yeah. So Yeah. And we are actually planning on updating this so that in the future, the user could just send us a checkpoint that they created, and then we will just use that checkpoint to do local inference here. And, so it will be much more easier for the user, and, we could actually be more flexible with, when we're doing, when we'll be performing the experiments. And professor Wang also mentioned that we might be doing a hosting a a challenge with this website. So user would just train their own model and using our datasets, and we will be decide... We will be performing the checkpoints and see which trained model outperforms the others and give them a board. And...
00:35:51 · Michael Cho
[00:35:51] S0: Yeah. Yeah. I love I love I love this. I I don't know whether you guys know of this other project called RoboArena. Yeah. But I guess you guys are doing Humanoid, which I think is Yeah. Potentially a lot more difficult operationally. Yeah. Yeah. Yeah. I'm curious, like, about the cloud inference part. Because, I mean, definitely, once you do it through the cloud, there have to be some latency. I wonder, given that there's so many dorf in Humanoid, like, what is the limit of the type of task that you can still, you know, reasonably, you know, expect the policy to be able to do? Maybe the policy could do more, but it's just that the latency is is just there, and therefore, it can't do more. Like, how do you guys think about that? You you see what I'm saying? So so imagine if if someone have a, let's say, a very, capable policy supposedly, but then if you give that... If if there's any latency, the latency compiler, because you... Ideally, you wanna stream everything 30, and then literally in ideally zero latency, then it will be the full capability of the of the policy. So I wonder in your in your testing here, when you do, like, having some of this policy in the cloud, like like, is it is it, for example, is it only doing simpler, like, desktop manipulation tasks, or can it do... Really do the full set of things a full humanoid can do?
00:37:13 · Yue Wang, Michael Cho
[00:37:13] S1: Yeah. So I think, ideally, we can do some local manipulation tasks but with controlled environments. So, again, we want to unify the evaluation, and we want to make sure the evaluation is fair and reproducible. So that's why we need to... We still need to have this, controlled, confined lab environments. We want to make sure the evaluation is fair and reproducible. So we, we need this controlled, confined lab environments. So we need to design a certain number of tasks. And we hope that tasks can be, like, hard enough for some local manipulation tasks. And, that is our ultimate goal. For open world evaluation, I think that's a a stretch goal for the future, which is, fully the ideal case for for for for all the humanoid evaluation or, algorithm evaluation tasks. [00:38:11] S0: Yeah. I I mean, I imagine it's nontrivial, especially with this. Yeah. But I'm very glad you guys are even trying this. It's... Yeah. I mean, me and Chris talk about this the time. We we like benchmarks. We like, like, you know, more like a Yeah. Neutral eval... Real world eval, especially. So... But it... I can imagine why this is a natural next step because you guys see all the work doing the teleop. Doing eval, it's it's it's it's it's a natural thing to do next as well.
00:38:11 · Michael Cho, Hongyi Jing, Chris Paxton
[00:38:11] S0: Yeah. I I mean, I imagine it's nontrivial, especially with this. Yeah. But I'm very glad you guys are even trying this. It's... Yeah. I mean, me and Chris talk about this the time. We we like benchmarks. We like, like, you know, more like a Yeah. Neutral eval... Real world eval, especially. So... But it... I can imagine why this is a natural next step because you guys see all the work doing the teleop. Doing eval, it's it's it's it's it's a natural thing to do next as well. [00:38:41] S3: So in our everyday work, actually, for the local manipulation task, we only include the very simple high level lower body movement, such as, like, making the robot moving forward or turning around. But, actually, finally, we want to train some generalizable whole body policies for, like, for more expressive local manipulation tasks. And so, like, you can see here, this... The data we collected recently, and we developed our collection pipeline to lead this expressive whole body movement. Like, the robot will actually first pick up the dumpling point and then crouches and and to place this dumpling on the chair and then stand up. Yeah. So so in the future, we want to train our other VLA models and maybe develop some more more newer models and to see if we can train some really generalizable policies on this, yeah, whole body local manipulation tasks. [00:39:36] S6: Yeah. Does that mean we should all look forward to, like, a Humanoid Everyday too next year with full body manipulation?
00:39:36 · Chris Paxton, Michael Cho, Zhenyu Zhao
[00:39:36] S6: Yeah. Does that mean we should all look forward to, like, a Humanoid Everyday too next year with full body manipulation? [00:39:41] S0: Yeah. Yeah. Yeah. Okay. Yeah. This is awesome, man. I suppose this is the last slide, is it? Yeah. Yeah. Yeah. I guess... I mean, maybe you guys already kinda alluded to a couple of piece, like... But I'm curious, like, if you were to sum up the last, maybe nine months of doing this, like, what are some of the, like, maybe surprises and and maybe main takeaways that, you know, people reading your paper should should should also get... I mean, basically, learn from your experience. Like, what what would you say are some of the very key takeaways, especially if especially if they're surprising to you? [00:40:16] S2: Yeah. Yeah. We were really surprised that how, like, you know, sometimes the diffusion policies, like... Well well, from our experience, we found that diffusion policy did not work as well as the the viewing policy three d. So, using, depth as the, one of the states, the observation, actually works better than having just raw RGB image. Yeah. Yeah. I think that's very interesting.
00:40:48 · Yue Wang, Hongyi Jing, Michael Cho
[00:40:48] S1: Yeah. My, personal experience, when working with, students and when observing how they work on this project is that pilot operation is still quite hard, and it's not scalable at all. And we definitely need other source of data to scale up robotics to solve robotics. So as you saw from a journalist, blog post a couple days ago, So they use UMI style data to scale up, robotics to scale up manipulation. So I think we we want more of those work to really scale up robotics with, egocentric video with some egocentric human operations that can be transferred to the robotic space that can be used for pretraining the, robotic policy models. And with that being said, teleoperation is still quite important. We still need teleoperation for post training, for alignment to the robotic action space. But I guess the day scale compared to large scale human operation will be much smaller. And, and also in addition to this different source of data, we we need simulation for for evaluation for for data augmentation. And, also, we need generative models, for data augmentation and other things. So in summary, my personal lesson is that, as of now, we still see very small amount of, human oid, data available. So we are still quite early in the process, in the, in the journey. So we we definitely need to solve the data problems for robotics, then we can solve robotics. [00:42:25] S3: Cool. [00:42:27] S0: Anything from Xiawei and Hongyi?