RoboPapers
00:35:51 · Michael Cho
[00:35:51] S0: Yeah. Yeah. I love I love I love this. I I don't know whether you guys know of this other project called RoboArena. Yeah. But I guess you guys are doing Humanoid, which I think is Yeah. Potentially a lot more difficult operationally. Yeah. Yeah. Yeah. I'm curious, like, about the cloud inference part. Because, I mean, definitely, once you do it through the cloud, there have to be some latency. I wonder, given that there's so many dorf in Humanoid, like, what is the limit of the type of task that you can still, you know, reasonably, you know, expect the policy to be able to do? Maybe the policy could do more, but it's just that the latency is is just there, and therefore, it can't do more. Like, how do you guys think about that? You you see what I'm saying? So so imagine if if someone have a, let's say, a very, capable policy supposedly, but then if you give that... If if there's any latency, the latency compiler, because you... Ideally, you wanna stream everything 30, and then literally in ideally zero latency, then it will be the full capability of the of the policy. So I wonder in your in your testing here, when you do, like, having some of this policy in the cloud, like like, is it is it, for example, is it only doing simpler, like, desktop manipulation tasks, or can it do... Really do the full set of things a full humanoid can do?
00:37:13 · Yue Wang, Michael Cho
[00:37:13] S1: Yeah. So I think, ideally, we can do some local manipulation tasks but with controlled environments. So, again, we want to unify the evaluation, and we want to make sure the evaluation is fair and reproducible. So that's why we need to... We still need to have this, controlled, confined lab environments. We want to make sure the evaluation is fair and reproducible. So we, we need this controlled, confined lab environments. So we need to design a certain number of tasks. And we hope that tasks can be, like, hard enough for some local manipulation tasks. And, that is our ultimate goal. For open world evaluation, I think that's a a stretch goal for the future, which is, fully the ideal case for for for for all the humanoid evaluation or, algorithm evaluation tasks. [00:38:11] S0: Yeah. I I mean, I imagine it's nontrivial, especially with this. Yeah. But I'm very glad you guys are even trying this. It's... Yeah. I mean, me and Chris talk about this the time. We we like benchmarks. We like, like, you know, more like a Yeah. Neutral eval... Real world eval, especially. So... But it... I can imagine why this is a natural next step because you guys see all the work doing the teleop. Doing eval, it's it's it's it's it's a natural thing to do next as well.
00:38:11 · Michael Cho, Hongyi Jing, Chris Paxton
[00:38:11] S0: Yeah. I I mean, I imagine it's nontrivial, especially with this. Yeah. But I'm very glad you guys are even trying this. It's... Yeah. I mean, me and Chris talk about this the time. We we like benchmarks. We like, like, you know, more like a Yeah. Neutral eval... Real world eval, especially. So... But it... I can imagine why this is a natural next step because you guys see all the work doing the teleop. Doing eval, it's it's it's it's it's a natural thing to do next as well. [00:38:41] S3: So in our everyday work, actually, for the local manipulation task, we only include the very simple high level lower body movement, such as, like, making the robot moving forward or turning around. But, actually, finally, we want to train some generalizable whole body policies for, like, for more expressive local manipulation tasks. And so, like, you can see here, this... The data we collected recently, and we developed our collection pipeline to lead this expressive whole body movement. Like, the robot will actually first pick up the dumpling point and then crouches and and to place this dumpling on the chair and then stand up. Yeah. So so in the future, we want to train our other VLA models and maybe develop some more more newer models and to see if we can train some really generalizable policies on this, yeah, whole body local manipulation tasks. [00:39:36] S6: Yeah. Does that mean we should all look forward to, like, a Humanoid Everyday too next year with full body manipulation?
00:39:36 · Chris Paxton, Michael Cho, Zhenyu Zhao
[00:39:36] S6: Yeah. Does that mean we should all look forward to, like, a Humanoid Everyday too next year with full body manipulation? [00:39:41] S0: Yeah. Yeah. Yeah. Okay. Yeah. This is awesome, man. I suppose this is the last slide, is it? Yeah. Yeah. Yeah. I guess... I mean, maybe you guys already kinda alluded to a couple of piece, like... But I'm curious, like, if you were to sum up the last, maybe nine months of doing this, like, what are some of the, like, maybe surprises and and maybe main takeaways that, you know, people reading your paper should should should also get... I mean, basically, learn from your experience. Like, what what would you say are some of the very key takeaways, especially if especially if they're surprising to you? [00:40:16] S2: Yeah. Yeah. We were really surprised that how, like, you know, sometimes the diffusion policies, like... Well well, from our experience, we found that diffusion policy did not work as well as the the viewing policy three d. So, using, depth as the, one of the states, the observation, actually works better than having just raw RGB image. Yeah. Yeah. I think that's very interesting.
00:40:48 · Yue Wang, Hongyi Jing, Michael Cho
[00:40:48] S1: Yeah. My, personal experience, when working with, students and when observing how they work on this project is that pilot operation is still quite hard, and it's not scalable at all. And we definitely need other source of data to scale up robotics to solve robotics. So as you saw from a journalist, blog post a couple days ago, So they use UMI style data to scale up, robotics to scale up manipulation. So I think we we want more of those work to really scale up robotics with, egocentric video with some egocentric human operations that can be transferred to the robotic space that can be used for pretraining the, robotic policy models. And with that being said, teleoperation is still quite important. We still need teleoperation for post training, for alignment to the robotic action space. But I guess the day scale compared to large scale human operation will be much smaller. And, and also in addition to this different source of data, we we need simulation for for evaluation for for data augmentation. And, also, we need generative models, for data augmentation and other things. So in summary, my personal lesson is that, as of now, we still see very small amount of, human oid, data available. So we are still quite early in the process, in the, in the journey. So we we definitely need to solve the data problems for robotics, then we can solve robotics. [00:42:25] S3: Cool. [00:42:27] S0: Anything from Xiawei and Hongyi?
00:42:27 · Michael Cho, Hongyi Jing
[00:42:27] S0: Anything from Xiawei and Hongyi? [00:42:29] S3: Yeah. So so, actually, I think that when we are doing this humanoid everyday work, like like we... I mentioned here, we only use this... The high level, simple, you know, lower body movement. So... And... But when we collect this data, like the... Like, when we include some more complex movement like crouching, like standing or sitting, and and even maybe jumping in the future, So we we find that, actually, it's it's getting much more harder for for the current VOA models to train these policies. And we we found that, like, during inference, the robot is doing some random movements, especially in its lower body. Like like, in in this frame, it's still standing, but in the next frame, it's... It suddenly falls and, like, sits on the ground. So I think this is also another another work we can further optimize in the future. And I think the current existing frameworks cannot solve this problem very well. So maybe we can we we can do some optimizations and service in the near future. Yeah.