00:15:29 · Siyuan Huang, Michael Cho
[00:15:29] S2: Yeah. So for the active part, like we, for the first version, we have the like moving, moving head, but after some experiments, we find out that for the current model we built, the Activision is not It's not that useful for the current G1 robot. I think in the future, we might add back the degree of freedom to the NAC, but currently we don't observe significant improvement of performance from the active vision part. I'm not sure, do you have any other different observations? Because we also see some different designs of the NACs from other companies, right, Michael? [00:16:08] S0: Yeah, I'm not sure. Because I guess in a way, you have active vision through the moving of the whole body, perhaps. And I suppose if the view of you is good enough, these days the cameras are quite good. So you might already have the information, not by moving the body too much. But in biology, pretty much all animals do have... You know, an hour would have to be able to move in almost like, you know, like, so I suppose it helps. But yeah. Yeah, I think in the current stage... I'm actually quite surprised at the... I was going to say, I'm quite surprised that in your case, right, it was so long horizon and you managed to achieve all that. Just like you mentioned, there are nine subtasks all string together, right, in the second part of the video. I was quite surprised that just, I mean, without the actuator neck, it can kind of string all this thing together with just kind of having a fixer. So I thought that was quite cool.
00:17:10 · Siyuan Huang
[00:17:10] S2: Yeah. Yeah, I think in the current stage, the body movement can compensate most of the camera movement from the Activision part. I think during the human, so that we are learning from the human data, right? So for the most tasks, when the human is familiar with the task, familiar enough with the task, it actually requires, don't require lots of activation for humans. So activation is... Mainly for exploration when you are in the new environment, or when you are trying to accelerate the movement, increase the efficiency of the movement. But for a daily task, if the humans are pretty familiar with task completions actually, Actually, the activation is quite incremental. Compared with this, the body movement can compensate the viewpoint change requirements for switching tasks from one to another. I think this is our current observation.
00:18:12 · Xiaojian Ma
[00:18:12] S1: Maybe I'd add a signal on this. Actually, the activation thing can be broken down into two facets. The first one is during the data collection, how you actually can collect the activation data. I mean, if you are doing this tally operation stuff, that will be much easier because you can have a headset or something like this to just collect your head movement from your tally operator. But the real issue arising arose when you're trying to switch to egocentric data collection, or especially when you're trying to only collect the upper body of your egocentric collection like most other people do. What is interesting here is that your neck movement and your body movement are actually entangled with each other. So it's actually, you have no way, there's no way to actually tell your body movement from your head movement, just from an upper body kind of egocentric collection. The only way to Substack is something we actually will introduce later, which is our full body collection system that will simultaneously collect your head movement and your body movement in the same process. So you will be able to just disentangle these two parts. And to accurately retarget your head movement and your body movement to your humanoid robot at this point. So this is the first thing, the first thesis. And the second thesis is that there's actually a similar fix to For the activation part, because you look carefully, so in our robot configuration, we actually have an Insta360 X5 camera mounted to the head. So that one actually provides you with this panorama kind of perspective. So you'll be able to see almost everything around you without actually moving your neck. But that configuration actually comes with a cost, is that Insta360 is actually adds a significant amount of bump cost to our bone. So that is something that we should take care of later. But at least at this point, that will be a... Got it. If that makes sense. Yeah.
00:20:16 · Siyuan Huang
[00:20:16] S2: So for the data zero, actually, there are several key designs for the models. So first is that the brain and controller, they are co-designed to scale humanoid intelligence. So during years of research in robotics, we have significantly found out that the controller is still actually one of the bottlenecks for the robotic systems, including, I think, including the humanoid robot and also the wheel robot. So it's really... It's really critical to incorporate the design ideas of the controller into the design of the brain. So we call it co-design. We co-design the controller and the brain to scale the human intelligence. We also propose several ways of how to scale such kind of humanoid foundation models. So first is to portray on large-scale diverse full-body human data. As Xiaojian Ma mentioned, we designed our own data collection system. I think I discussed this with Michael previously. We have some shared insight in how to build up the egocentric videos data and also capture the human movement we'll introduce later. And second is the real-to-seem-to-real accelerated humanoid foundation model evaluation and enable humanoid agents. So this is a severe problem for humanoid robots because doing the experiment, repeated experiment or control experiment, ablations for the humanoid robot is really, really hard. I think you need to reset humana robot into a certain accurate space and you need to do the repeated trial for H-robot and for the G1-robot, the machine status is also a problem. In order to have rigorous evaluations, we need better infrastructures to do the experiments. I think that's the key to have a better or more efficient iterations for the foundation models. So we build up the real-to-sim-to-real pipeline to do this kind of repeated experiments and try to loop the development of the foundation model with the post-stream and evaluation I think this is quite mature in previous wheel robots, but still early for the humanoid robot. We'll describe this later. And the third one is the data action control. So we believe that in the future, if we want to really scale up the foundation model and we want to improve the performance towards deployable standards, like 99% of access rate or 100% of access rate, We need RL. We need real-world RL to train a humanoid robot to have a better human learning process.
00:23:14 · Siyuan Huang
So the current, most trackers, they use the trajectory-based tracking or controller. So that is not possible. It's possible for the human interventions while during the deployment or during the rollout of the human robot. So we design a Delta action-based controller so that we can enable the human intervention during every rollout or inference time of the foundation models. I think this is a key design. We'll describe detail. In the following stage. So basically these six points are the designer design principle we have proposed in the downtown zero models. And we can have a brief overview of the human intelligence tasks. And here are some like summaries of the tasks we have seen in the demos like operating our turntable. And so this kind of task is quite challenging because like The, like holding the record and put it into a hole, the very small hole is requires a pretty precise, um, uh, interactions. And also we have this kind of wujiheng. So it's really hard to move the base. And then move the arm, then insert that record into the small hole. And we find out this is one of the most challenging tasks for the zero models. And also making the bat. So this incorporates interactions with the fabrics doing the Moving the pillar and then move the joysticks, interacting with the fabrics and also tackling some randomization or the variations of the forms of the fabric materials. And there are some more tasks like, I can introduce this briefly, like picking up objects from the ground, requiring the robot to have the capability to do the balancing. And also operating the dishwasher. Yeah. Also, as Chris just mentioned, it needs the Delta Zero to brace its left hand against the shelf for this kind of support to have enough force to opening the door. Yeah.
00:25:28 · Michael Cho, Siyuan Huang, Chris Paxton
[00:25:28] S0: I'm curious, like this bracing against the... Sorry. Is that learned from the human data or... Because human probably don't need this, right? Human is strong enough. Our core muscle is strong enough. But probably the G1 needs it. Is this learned from the human data or is it something else? [00:25:42] S2: Yeah, this is learned and emerged from human data. So the humans have various ways of opening the, the dishwashers. Like we have a better core, so may not need the, uh, to do the bracing stuff, but, uh, if you want only using the G1 to do the, like, teleoperations, so we have, we need to have a better way for the robot to, to have a like stable core. So bracing is one of the, one of the way, or have this kind of Compliance to open the door. [00:26:13] S3: So the teleoperators basically are, or not teleoperators, data collectors were aware of the limitations of the G1 and collected that particular motion. Yeah. Yeah. Okay. Makes sense.