RoboPapers
00:40:48 · Yue Wang, Hongyi Jing, Michael Cho
[00:40:48] S1: Yeah. My, personal experience, when working with, students and when observing how they work on this project is that pilot operation is still quite hard, and it's not scalable at all. And we definitely need other source of data to scale up robotics to solve robotics. So as you saw from a journalist, blog post a couple days ago, So they use UMI style data to scale up, robotics to scale up manipulation. So I think we we want more of those work to really scale up robotics with, egocentric video with some egocentric human operations that can be transferred to the robotic space that can be used for pretraining the, robotic policy models. And with that being said, teleoperation is still quite important. We still need teleoperation for post training, for alignment to the robotic action space. But I guess the day scale compared to large scale human operation will be much smaller. And, and also in addition to this different source of data, we we need simulation for for evaluation for for data augmentation. And, also, we need generative models, for data augmentation and other things. So in summary, my personal lesson is that, as of now, we still see very small amount of, human oid, data available. So we are still quite early in the process, in the, in the journey. So we we definitely need to solve the data problems for robotics, then we can solve robotics. [00:42:25] S3: Cool. [00:42:27] S0: Anything from Xiawei and Hongyi?
00:42:27 · Michael Cho, Hongyi Jing
[00:42:27] S0: Anything from Xiawei and Hongyi? [00:42:29] S3: Yeah. So so, actually, I think that when we are doing this humanoid everyday work, like like we... I mentioned here, we only use this... The high level, simple, you know, lower body movement. So... And... But when we collect this data, like the... Like, when we include some more complex movement like crouching, like standing or sitting, and and even maybe jumping in the future, So we we find that, actually, it's it's getting much more harder for for the current VOA models to train these policies. And we we found that, like, during inference, the robot is doing some random movements, especially in its lower body. Like like, in in this frame, it's still standing, but in the next frame, it's... It suddenly falls and, like, sits on the ground. So I think this is also another another work we can further optimize in the future. And I think the current existing frameworks cannot solve this problem very well. So maybe we can we we can do some optimizations and service in the near future. Yeah.
00:43:36 · Xiawei Liu, Chris Paxton, Yue Wang, Michael Cho
[00:43:36] S4: Yeah. I think for me... Because I did most of the teleoperation parts, and I felt that companies right now are more suitable for doing humanoid, you know, research and training right now because they have the money and they have the power and people to do all those data collection. So I would say that for researchers right now, the best direction is to develop actually, you know, whole body data collection pipeline or to develop policies on top of the datasets that company can train. Because I felt when we are taking the data, it's it's really hard for labs and research centers to do all all these works. Even though we were able to complete this, we took a lot... A long time, a lot of commitment to this project. And I felt that, yeah, companies could... If if companies could do more data collection in this industry, this this humanoid industry would definitely progress faster, and researchers would be able to perform better research and eventually a more robust and sophisticated system with humanoids. [00:45:14] S6: I'm guessing this is a request for companies to share more data publicly. Is that the... [00:45:18] S1: Yeah. Yeah. Yeah. [00:45:20] S6: Makes sense. It's a good question. [00:45:23] S0: Good request. Maybe Jiageng as well?
00:45:23 · Michael Cho, Jiageng Mao
[00:45:23] S0: Good request. Maybe Jiageng as well? [00:45:28] S5: Yeah. I actually believe in the power of in the wild teachings because it's very hard to scale up the physical robot. So basically, every of the teleoperation tools, they need a physical robot with a human. And so you have to... The manufacturing of the physical robot is limited by the manufacturers, but the human's labors, it can be cross housed or they can be more scalable than those utility operators' physical robots. So I'm more interested in how do you apply those in the wild. One thing is... One dimension is how do you apply those in the wild data, especially you're using, like, egocentric manipulation, human manipulation videos to train your humanoid robots. So that is one thing I'm thinking about. And another thing is... So how do you design exoskeletons? Just like a UMI's, but it's UMI for humanoid robots. So they basically have arms and dexterous hand, exoskeletons about dexterous hands, also have cameras as an egocentric view. Humans can use those to collect data and just use those data to train humanoid robots. So in method, they will have a physical robot to teleoperate just using humans to test the data. I I believe that will be a more scalable ways, and I think we will spend more time in the future on these topics. Yeah.
00:47:03 · Michael Cho
[00:47:03] S0: Right. Perfect. Yeah. Maybe I I guess... I think we we are about to wrap up here. Maybe as a as a typical last question, Chris and I like to ask this. Like, is is is there any cool paper you guys came across recently you wanna give a shout out to? Doesn't have to be related to Humanoid Everyday. It could be computer vision or whatever. Yeah. In case there are some gems that me and Chris or the the listeners missed it, they can go and read up the... That paper. Yeah. Maybe, Yue Wang, you can... You can go first.
00:47:33 · Yue Wang, Michael Cho
[00:47:33] S1: Yeah. So as you... As we mentioned at the beginning, the Sonic paper from NVIDIA is a good one. And I guess it's pretty similar to the Twist two paper in terms of the device they use. They use a pickle two, I guess, to teleoperate the robot. That will unlock a lot of capabilities. And the other thing, as I mentioned at the beginning, which is another paper, is, Unitree's Teleoperation System, the... Which was released last week. That was very impressive because, I think they use, some motion tracking system, to do multi person tracking, and then they do, multi person, like a tidal operation with mocap system. So that one is also quite powerful. So I'm imagining in the future, if we can have a MoCap system that is mounted in a very large space on the ceiling and that can track multiple persons, then we can probably do some multi robot teleoperation to perform a even harder task. So that's... That one is very impressive. Yeah. Those are some interesting papers. And the other interesting paper is, I just want to give a shout to, Jiageng. So he released a paper called the phase word, which is trying to leverage some video generation model to to generate some demonstrations and then transfer those demonstrations from a general video domain to the robotic domain, which is a very interesting one from my perspective. [00:49:00] S0: What... What's it called? It's called the What's it called? [00:49:02] S1: Face Face word. Okay. Cool. [00:49:04] S0: Cool. Cool. Alright.