RoboPapers
00:37:13 · Yue Wang, Michael Cho
[00:37:13] S1: Yeah. So I think, ideally, we can do some local manipulation tasks but with controlled environments. So, again, we want to unify the evaluation, and we want to make sure the evaluation is fair and reproducible. So that's why we need to... We still need to have this, controlled, confined lab environments. We want to make sure the evaluation is fair and reproducible. So we, we need this controlled, confined lab environments. So we need to design a certain number of tasks. And we hope that tasks can be, like, hard enough for some local manipulation tasks. And, that is our ultimate goal. For open world evaluation, I think that's a a stretch goal for the future, which is, fully the ideal case for for for for all the humanoid evaluation or, algorithm evaluation tasks. [00:38:11] S0: Yeah. I I mean, I imagine it's nontrivial, especially with this. Yeah. But I'm very glad you guys are even trying this. It's... Yeah. I mean, me and Chris talk about this the time. We we like benchmarks. We like, like, you know, more like a Yeah. Neutral eval... Real world eval, especially. So... But it... I can imagine why this is a natural next step because you guys see all the work doing the teleop. Doing eval, it's it's it's it's it's a natural thing to do next as well.
00:38:11 · Michael Cho, Hongyi Jing, Chris Paxton
[00:38:11] S0: Yeah. I I mean, I imagine it's nontrivial, especially with this. Yeah. But I'm very glad you guys are even trying this. It's... Yeah. I mean, me and Chris talk about this the time. We we like benchmarks. We like, like, you know, more like a Yeah. Neutral eval... Real world eval, especially. So... But it... I can imagine why this is a natural next step because you guys see all the work doing the teleop. Doing eval, it's it's it's it's it's a natural thing to do next as well. [00:38:41] S3: So in our everyday work, actually, for the local manipulation task, we only include the very simple high level lower body movement, such as, like, making the robot moving forward or turning around. But, actually, finally, we want to train some generalizable whole body policies for, like, for more expressive local manipulation tasks. And so, like, you can see here, this... The data we collected recently, and we developed our collection pipeline to lead this expressive whole body movement. Like, the robot will actually first pick up the dumpling point and then crouches and and to place this dumpling on the chair and then stand up. Yeah. So so in the future, we want to train our other VLA models and maybe develop some more more newer models and to see if we can train some really generalizable policies on this, yeah, whole body local manipulation tasks. [00:39:36] S6: Yeah. Does that mean we should all look forward to, like, a Humanoid Everyday too next year with full body manipulation?
00:39:36 · Chris Paxton, Michael Cho, Zhenyu Zhao
[00:39:36] S6: Yeah. Does that mean we should all look forward to, like, a Humanoid Everyday too next year with full body manipulation? [00:39:41] S0: Yeah. Yeah. Yeah. Okay. Yeah. This is awesome, man. I suppose this is the last slide, is it? Yeah. Yeah. Yeah. I guess... I mean, maybe you guys already kinda alluded to a couple of piece, like... But I'm curious, like, if you were to sum up the last, maybe nine months of doing this, like, what are some of the, like, maybe surprises and and maybe main takeaways that, you know, people reading your paper should should should also get... I mean, basically, learn from your experience. Like, what what would you say are some of the very key takeaways, especially if especially if they're surprising to you? [00:40:16] S2: Yeah. Yeah. We were really surprised that how, like, you know, sometimes the diffusion policies, like... Well well, from our experience, we found that diffusion policy did not work as well as the the viewing policy three d. So, using, depth as the, one of the states, the observation, actually works better than having just raw RGB image. Yeah. Yeah. I think that's very interesting.
00:40:48 · Yue Wang, Hongyi Jing, Michael Cho
[00:40:48] S1: Yeah. My, personal experience, when working with, students and when observing how they work on this project is that pilot operation is still quite hard, and it's not scalable at all. And we definitely need other source of data to scale up robotics to solve robotics. So as you saw from a journalist, blog post a couple days ago, So they use UMI style data to scale up, robotics to scale up manipulation. So I think we we want more of those work to really scale up robotics with, egocentric video with some egocentric human operations that can be transferred to the robotic space that can be used for pretraining the, robotic policy models. And with that being said, teleoperation is still quite important. We still need teleoperation for post training, for alignment to the robotic action space. But I guess the day scale compared to large scale human operation will be much smaller. And, and also in addition to this different source of data, we we need simulation for for evaluation for for data augmentation. And, also, we need generative models, for data augmentation and other things. So in summary, my personal lesson is that, as of now, we still see very small amount of, human oid, data available. So we are still quite early in the process, in the, in the journey. So we we definitely need to solve the data problems for robotics, then we can solve robotics. [00:42:25] S3: Cool. [00:42:27] S0: Anything from Xiawei and Hongyi?
00:42:27 · Michael Cho, Hongyi Jing
[00:42:27] S0: Anything from Xiawei and Hongyi? [00:42:29] S3: Yeah. So so, actually, I think that when we are doing this humanoid everyday work, like like we... I mentioned here, we only use this... The high level, simple, you know, lower body movement. So... And... But when we collect this data, like the... Like, when we include some more complex movement like crouching, like standing or sitting, and and even maybe jumping in the future, So we we find that, actually, it's it's getting much more harder for for the current VOA models to train these policies. And we we found that, like, during inference, the robot is doing some random movements, especially in its lower body. Like like, in in this frame, it's still standing, but in the next frame, it's... It suddenly falls and, like, sits on the ground. So I think this is also another another work we can further optimize in the future. And I think the current existing frameworks cannot solve this problem very well. So maybe we can we we can do some optimizations and service in the near future. Yeah.
00:43:36 · Xiawei Liu, Chris Paxton, Yue Wang, Michael Cho
[00:43:36] S4: Yeah. I think for me... Because I did most of the teleoperation parts, and I felt that companies right now are more suitable for doing humanoid, you know, research and training right now because they have the money and they have the power and people to do all those data collection. So I would say that for researchers right now, the best direction is to develop actually, you know, whole body data collection pipeline or to develop policies on top of the datasets that company can train. Because I felt when we are taking the data, it's it's really hard for labs and research centers to do all all these works. Even though we were able to complete this, we took a lot... A long time, a lot of commitment to this project. And I felt that, yeah, companies could... If if companies could do more data collection in this industry, this this humanoid industry would definitely progress faster, and researchers would be able to perform better research and eventually a more robust and sophisticated system with humanoids. [00:45:14] S6: I'm guessing this is a request for companies to share more data publicly. Is that the... [00:45:18] S1: Yeah. Yeah. Yeah. [00:45:20] S6: Makes sense. It's a good question. [00:45:23] S0: Good request. Maybe Jiageng as well?