00:08:34 · Siyuan Huang
So it's really hard to ensure the stabilities during the long horizon task. And we described more details about why the local manipulation is hard, and we treated the local manipulation as the most challenging And solve the problem in robotics, also the core problem of human intelligence. And there are some specific reasons of why the local manipulation is challenging. So the first is that the robot must control a moving body while perceiving and manipulating a changing world. So this is different from just doing the manipulation alone, right? So previously, we were usually trying to disentangle the problem of local manipulation into two parts, the local motion and the So for locomotion, we can drive the robot by a controller, an IR-based controller, so the robot can dance, can have very extreme movement, like the omni-extreme we described before in one of the episodes, and show really impressive dancing behavior from one single polycyst. And that's called locomotion and the whole body control. Another problem is the manipulations, right? We can have a fixed space and then you can use your robot anti-factor, gripper or dexterous hands to do some dexterous manipulations with a fixing hand. So the fixing base actually limits the action space of the robot. So we want a larger action space similar to the human. So we combine the locomotion and the manipulation problem towards the local manipulation. That means the robot needs to move while proceeding and manipulating the interacting with the changing world. And the action space is pretty large and it's really hard to solve the capital problem of the locomotion and manipulation. Pretty challenging. And we think this is the most challenging problem in the robotics and humanoid robot. And there are more reasons of why it's hard. And like the each action determines the starting condition for the nest. And that's one of the reasons why the long horizon task is still the holy grail of the robotics. Because if you're trying to connect the previous task and the second task, you can have some like GPT-6 for the You can decompose the task into some atomic task. But for robotics, if you want to actually try to connect the previous action and next action, you may have a very precise position and you need to have very good adjustments. So this is also pretty challenging. And the third is about the generalization.
00:11:26 · Siyuan Huang
So, uh, for each single atomic task, we have seen that the task must So we have seen that from the demo that the robot can brace itself from the dishwasher task. So the supporting condition and the forcing condition compliance is changing a lot during each atomic task. So this is really challenging for the robot to finish all these tasks and also have the capability to deal with the generalization problems. That means basically if you want to change a certain object, object categories, you want to change the color, change the lightening conditions for the thing, the model should handle all these kinds of problems. So the second part, I introduced local manipulation, why it is the most challenging to solve the problem. And I think we cover a lot in the paper channels. After that, we introduced about, officially introduced about Delta Zero, and we summarized the innovations or contributions in Delta Zero into six steps. We called it six steps towards humanoid intelligence. And to summarize that into one sentence is that the long horizon local manipulation, it actually demands coordinations, practical speed, transferable experience, executable control, corrective learning, and scalable evaluation together. And for the Q9 foundation model, we want to bring all this kind of stuff, all this kind of ingredients into one single model. And this is our main contributions. And to make it more details, the Delta zero actually does several The first is that the Delta Zero actually unlocks a humanoid's full dexterity across 69 degrees of freedom. So basically for the G1 initial robot, we have 29 degrees of freedom. So it's really hard for the system to control this kind of high degree of freedoms of RoboSystems, including the controller and also the planner, right? Of Heidegger Freedom System actually advance towards the human level, local manipulation and coordination. So it covers the action space of humans. So we have higher action space, higher coverage of action space. So the robot can be really general. And the second is that we use a single generative policies that performs diverse everyday tasks and near human speed. So we have seen from the demo that all of these actions, the long horizon tasks, they are performed at one We did not accelerate the video speed. So basically the robot has, they can already perform all these tasks at a near human speed.
00:14:19 · Siyuan Huang, Michael Cho
I think this is pretty important. And we have seen that emphasis from like, from Reward AI, from Dyna, from several companies, they all emphasize that the robot needs to have a near human speed so they can really can do the production. We can really calculate efficiency. We can really calculate how the robot can making the profit. They can helping the people to do the daily tasks. So for data intelligence, we want the humanoid robot, the bipedal robot. They can also have the speed of near human level. So that's the metric or that's the standard we view the humanoid robot should approach after the model development. So the second part... [00:15:07] S0: Sorry, I have one question. So I noticed that the way you guys set up the G1s right now, you guys have your own head with your presumably camera sensing suite, right? But that is not Activision, right? There's no neck. Have you guys thought about that? I presume you're using a full body to get to the view you want. Yeah. [00:15:29] S2: Yeah. So for the active part, like we, for the first version, we have the like moving, moving head, but after some experiments, we find out that for the current model we built, the Activision is not It's not that useful for the current G1 robot. I think in the future, we might add back the degree of freedom to the NAC, but currently we don't observe significant improvement of performance from the active vision part. I'm not sure, do you have any other different observations? Because we also see some different designs of the NACs from other companies, right, Michael?
00:15:29 · Siyuan Huang, Michael Cho
[00:15:29] S2: Yeah. So for the active part, like we, for the first version, we have the like moving, moving head, but after some experiments, we find out that for the current model we built, the Activision is not It's not that useful for the current G1 robot. I think in the future, we might add back the degree of freedom to the NAC, but currently we don't observe significant improvement of performance from the active vision part. I'm not sure, do you have any other different observations? Because we also see some different designs of the NACs from other companies, right, Michael? [00:16:08] S0: Yeah, I'm not sure. Because I guess in a way, you have active vision through the moving of the whole body, perhaps. And I suppose if the view of you is good enough, these days the cameras are quite good. So you might already have the information, not by moving the body too much. But in biology, pretty much all animals do have... You know, an hour would have to be able to move in almost like, you know, like, so I suppose it helps. But yeah. Yeah, I think in the current stage... I'm actually quite surprised at the... I was going to say, I'm quite surprised that in your case, right, it was so long horizon and you managed to achieve all that. Just like you mentioned, there are nine subtasks all string together, right, in the second part of the video. I was quite surprised that just, I mean, without the actuator neck, it can kind of string all this thing together with just kind of having a fixer. So I thought that was quite cool.
00:17:10 · Siyuan Huang
[00:17:10] S2: Yeah. Yeah, I think in the current stage, the body movement can compensate most of the camera movement from the Activision part. I think during the human, so that we are learning from the human data, right? So for the most tasks, when the human is familiar with the task, familiar enough with the task, it actually requires, don't require lots of activation for humans. So activation is... Mainly for exploration when you are in the new environment, or when you are trying to accelerate the movement, increase the efficiency of the movement. But for a daily task, if the humans are pretty familiar with task completions actually, Actually, the activation is quite incremental. Compared with this, the body movement can compensate the viewpoint change requirements for switching tasks from one to another. I think this is our current observation.
00:18:12 · Xiaojian Ma
[00:18:12] S1: Maybe I'd add a signal on this. Actually, the activation thing can be broken down into two facets. The first one is during the data collection, how you actually can collect the activation data. I mean, if you are doing this tally operation stuff, that will be much easier because you can have a headset or something like this to just collect your head movement from your tally operator. But the real issue arising arose when you're trying to switch to egocentric data collection, or especially when you're trying to only collect the upper body of your egocentric collection like most other people do. What is interesting here is that your neck movement and your body movement are actually entangled with each other. So it's actually, you have no way, there's no way to actually tell your body movement from your head movement, just from an upper body kind of egocentric collection. The only way to Substack is something we actually will introduce later, which is our full body collection system that will simultaneously collect your head movement and your body movement in the same process. So you will be able to just disentangle these two parts. And to accurately retarget your head movement and your body movement to your humanoid robot at this point. So this is the first thing, the first thesis. And the second thesis is that there's actually a similar fix to For the activation part, because you look carefully, so in our robot configuration, we actually have an Insta360 X5 camera mounted to the head. So that one actually provides you with this panorama kind of perspective. So you'll be able to see almost everything around you without actually moving your neck. But that configuration actually comes with a cost, is that Insta360 is actually adds a significant amount of bump cost to our bone. So that is something that we should take care of later. But at least at this point, that will be a... Got it. If that makes sense. Yeah.