00:05:46 · Siyuan Huang
[00:05:46] S2: See ya. Okay. Okay. So basically the video is composed by two parts. The first part is the robot doing the very dexterous manipulation task. And second part is the robot doing very long horizon local manipulation. So basically it's about three minutes of uncut video and there are like nine atomic tests and more than like 30 steps for the robot to finish all these tests. So it's pretty challenging. For us, we have seen companies like Figure, they show a video like this, but it's really hard. As Michael also said, open, close this kind of task requires a lot of interaction with the articulated objects. It's pretty hard to connect those tasks because it should start from the end of the last atomic task. And this is difficult because we don't use any like SLAM based navigation still in this video. So the robot is purely driven by the model to do the navigation, to do the upper body interaction and lower body interactions, all of this stuff from one single model. So it's pretty challenging. And the blog introduces more details about our models. So first we want to introduce about how we like treated the problem of humanoid foundation model and why we think is really important for the, for us to tackle the problem of humanoid impetages. So basically, the humanoid form offers the possibility of working wherever people do. So we think that humanoid is a general form. It's better than a real robot, so it can do the works everywhere. And it shows that humanoid Human intelligence is what turns that possibility into useful and reliable actions. So basically we have human intelligence, we can drive all this human robot to do the useful and reliable actions. Right now we have seen a lot of demos and in the future, we want to turn these capabilities into deployments. And this is the vision of our companies, the data intelligence. And I think there are some challenges in Tackling this kind of female intelligence. So first we show the long-term local manipulations, right? So we expect the robot to do the long-term actions, not just some single pickup like actions in front of the table. So basically the challenge is like in long horizon tasks, like every So basically the surrounding will change, and the location will change, and the interaction mode will change, and there are a lot of dynamics.
00:08:34 · Siyuan Huang
So it's really hard to ensure the stabilities during the long horizon task. And we described more details about why the local manipulation is hard, and we treated the local manipulation as the most challenging And solve the problem in robotics, also the core problem of human intelligence. And there are some specific reasons of why the local manipulation is challenging. So the first is that the robot must control a moving body while perceiving and manipulating a changing world. So this is different from just doing the manipulation alone, right? So previously, we were usually trying to disentangle the problem of local manipulation into two parts, the local motion and the So for locomotion, we can drive the robot by a controller, an IR-based controller, so the robot can dance, can have very extreme movement, like the omni-extreme we described before in one of the episodes, and show really impressive dancing behavior from one single polycyst. And that's called locomotion and the whole body control. Another problem is the manipulations, right? We can have a fixed space and then you can use your robot anti-factor, gripper or dexterous hands to do some dexterous manipulations with a fixing hand. So the fixing base actually limits the action space of the robot. So we want a larger action space similar to the human. So we combine the locomotion and the manipulation problem towards the local manipulation. That means the robot needs to move while proceeding and manipulating the interacting with the changing world. And the action space is pretty large and it's really hard to solve the capital problem of the locomotion and manipulation. Pretty challenging. And we think this is the most challenging problem in the robotics and humanoid robot. And there are more reasons of why it's hard. And like the each action determines the starting condition for the nest. And that's one of the reasons why the long horizon task is still the holy grail of the robotics. Because if you're trying to connect the previous task and the second task, you can have some like GPT-6 for the You can decompose the task into some atomic task. But for robotics, if you want to actually try to connect the previous action and next action, you may have a very precise position and you need to have very good adjustments. So this is also pretty challenging. And the third is about the generalization.
00:11:26 · Siyuan Huang
So, uh, for each single atomic task, we have seen that the task must So we have seen that from the demo that the robot can brace itself from the dishwasher task. So the supporting condition and the forcing condition compliance is changing a lot during each atomic task. So this is really challenging for the robot to finish all these tasks and also have the capability to deal with the generalization problems. That means basically if you want to change a certain object, object categories, you want to change the color, change the lightening conditions for the thing, the model should handle all these kinds of problems. So the second part, I introduced local manipulation, why it is the most challenging to solve the problem. And I think we cover a lot in the paper channels. After that, we introduced about, officially introduced about Delta Zero, and we summarized the innovations or contributions in Delta Zero into six steps. We called it six steps towards humanoid intelligence. And to summarize that into one sentence is that the long horizon local manipulation, it actually demands coordinations, practical speed, transferable experience, executable control, corrective learning, and scalable evaluation together. And for the Q9 foundation model, we want to bring all this kind of stuff, all this kind of ingredients into one single model. And this is our main contributions. And to make it more details, the Delta zero actually does several The first is that the Delta Zero actually unlocks a humanoid's full dexterity across 69 degrees of freedom. So basically for the G1 initial robot, we have 29 degrees of freedom. So it's really hard for the system to control this kind of high degree of freedoms of RoboSystems, including the controller and also the planner, right? Of Heidegger Freedom System actually advance towards the human level, local manipulation and coordination. So it covers the action space of humans. So we have higher action space, higher coverage of action space. So the robot can be really general. And the second is that we use a single generative policies that performs diverse everyday tasks and near human speed. So we have seen from the demo that all of these actions, the long horizon tasks, they are performed at one We did not accelerate the video speed. So basically the robot has, they can already perform all these tasks at a near human speed.
00:14:19 · Siyuan Huang, Michael Cho
I think this is pretty important. And we have seen that emphasis from like, from Reward AI, from Dyna, from several companies, they all emphasize that the robot needs to have a near human speed so they can really can do the production. We can really calculate efficiency. We can really calculate how the robot can making the profit. They can helping the people to do the daily tasks. So for data intelligence, we want the humanoid robot, the bipedal robot. They can also have the speed of near human level. So that's the metric or that's the standard we view the humanoid robot should approach after the model development. So the second part... [00:15:07] S0: Sorry, I have one question. So I noticed that the way you guys set up the G1s right now, you guys have your own head with your presumably camera sensing suite, right? But that is not Activision, right? There's no neck. Have you guys thought about that? I presume you're using a full body to get to the view you want. Yeah. [00:15:29] S2: Yeah. So for the active part, like we, for the first version, we have the like moving, moving head, but after some experiments, we find out that for the current model we built, the Activision is not It's not that useful for the current G1 robot. I think in the future, we might add back the degree of freedom to the NAC, but currently we don't observe significant improvement of performance from the active vision part. I'm not sure, do you have any other different observations? Because we also see some different designs of the NACs from other companies, right, Michael?
00:15:29 · Siyuan Huang, Michael Cho
[00:15:29] S2: Yeah. So for the active part, like we, for the first version, we have the like moving, moving head, but after some experiments, we find out that for the current model we built, the Activision is not It's not that useful for the current G1 robot. I think in the future, we might add back the degree of freedom to the NAC, but currently we don't observe significant improvement of performance from the active vision part. I'm not sure, do you have any other different observations? Because we also see some different designs of the NACs from other companies, right, Michael? [00:16:08] S0: Yeah, I'm not sure. Because I guess in a way, you have active vision through the moving of the whole body, perhaps. And I suppose if the view of you is good enough, these days the cameras are quite good. So you might already have the information, not by moving the body too much. But in biology, pretty much all animals do have... You know, an hour would have to be able to move in almost like, you know, like, so I suppose it helps. But yeah. Yeah, I think in the current stage... I'm actually quite surprised at the... I was going to say, I'm quite surprised that in your case, right, it was so long horizon and you managed to achieve all that. Just like you mentioned, there are nine subtasks all string together, right, in the second part of the video. I was quite surprised that just, I mean, without the actuator neck, it can kind of string all this thing together with just kind of having a fixer. So I thought that was quite cool.
00:17:10 · Siyuan Huang
[00:17:10] S2: Yeah. Yeah, I think in the current stage, the body movement can compensate most of the camera movement from the Activision part. I think during the human, so that we are learning from the human data, right? So for the most tasks, when the human is familiar with the task, familiar enough with the task, it actually requires, don't require lots of activation for humans. So activation is... Mainly for exploration when you are in the new environment, or when you are trying to accelerate the movement, increase the efficiency of the movement. But for a daily task, if the humans are pretty familiar with task completions actually, Actually, the activation is quite incremental. Compared with this, the body movement can compensate the viewpoint change requirements for switching tasks from one to another. I think this is our current observation.