00:35:15 · Siyuan Huang, Chris Paxton, Michael Cho
That means the robot is not following the actual trajectory of the performer. It is learning or is adjusting from the delta actions of the anti-factor. [00:35:27] S3: So you're just like nudging its hand, basically, and it's still controlling its fingers and the rest of its body. Is that right? [00:35:33] S2: Yeah. Yeah. We use the Delta as the input, not the whole body trajectory. [00:35:39] S3: But the fingers were closing. I was like, how is it set? [00:35:42] S2: Yeah. Yeah. Finger is not designed in this version. So it's not like the buttons. [00:35:48] S3: It's like the policy is still deciding, like, this looks like the time I should close my fingers. Okay. No, that's cool. Thanks. [00:35:54] S2: Yeah. We're doing it in our next version. [00:35:58] S0: Yeah, so I guess what you're saying is that because this tracks the delta or just a change, and so the human literally, let's say the, it looks like the human is taking over from the robot when the robot is already bending down. The human himself don't have to bend down to simulate that. Exactly. Okay, okay. Okay, a lot more comfortable for the human to do the intervention.
00:35:58 · Michael Cho, Siyuan Huang
[00:35:58] S0: Yeah, so I guess what you're saying is that because this tracks the delta or just a change, and so the human literally, let's say the, it looks like the human is taking over from the robot when the robot is already bending down. The human himself don't have to bend down to simulate that. Exactly. Okay, okay. Okay, a lot more comfortable for the human to do the intervention. [00:36:18] S2: Okay, yeah, sure. Yeah, we can see Xiao Zhan in front here. And during the execution, even for a challenging task, we can start from here and then we can intervene and start from the human as well. So you can sit there operating a robot that's doing one-fit stand. Yeah, it's quite intensive. Another example? Yeah, I think we are the first one to really tackle this system, the database control system for the And here you can use it as an [00:36:49] S0: interface.
00:36:52 · Siyuan Huang
[00:36:52] S2: Because the reason is that if you're using the whole body as the input or the The joint input for the trackers, you need to synchronize the actions for the human and the robot whenever you want to do the intervention. And this is basically not possible for, you can do this, but the efficiency is really, really low. So it's basically not possible for the real training, real RL training for the human robot. So this, I think this system is a must for the future of the human robot system. Yeah. And then we also compare some metrics of the trackers. We call it execution fidelity on everyday task motions. We compare with other like SOTA controller, including the heft, the Mimic light, the new version of the sonic, and the scale BFM. We compare with them in terms of the global root errors and body position errors. These are some tracking metrics. We evaluate these trackers on the daily tasks for the local manipulation. So these are not for the... Like the tracking dataset, we're using it on our daily local manipulation tasks. So it's actually proving that it can have a That has performance over this kind of daily interactions. And then we have, um, uh, more, uh, design for the, for the brain part. So basically I, I tried to go through this briefly is, um, a latent word action models and we design certain dynamics. So basically the vision And then we train the action branch and also the MMDIT, the multimodal diffusion models, tension models. And then we have certain dynamics, including the forward dynamics. Inverse dynamics, regional planning, and policy only. That corresponds to certain learning dynamics of the training according to different training rewards. We try to explore, fully explore the range of the data. So we designed these dynamics schemes. And then long-horizon execution is done by stage-by-stage instructions. So this stage can be provided by higher-level models. Currently, people are discussing whether GPT-6 can really replace the robot foundation models. But actually, first, it can serve as good high-level models. Maybe you can call it system two. So it can, it can decompose this kind of high level task into some atomic, atomic instructions for every, for more detailed sub steps. It can be a higher level models, an agent or a human. So this long horizon task can be staged.
00:39:47 · Siyuan Huang
And it can have some adaptations or can have some transitions from transition appearance or behaviors from task to another. And the exception is specified by the objectives. So delta 0 must learn how to reposition its body. So the position must be accurate before starting from the next actions. So this is the design we consider about the So the next section is about how we scaling the reliable humanoid intelligence. And this is quite interesting. We decompose this into three parts. The first part is we want to scale the data. We want learning from human data. I think from the first principle, we Doing all of this foundation model for humanoid robots is that we believe that human data is the richest source of robot learning. So we try to make use of this kind of high quality or low quality human data, including the upper body and also the lower body. We try to make use of this kind of data for the learning of the Delta Zero. And this motivates the design of the action space and motivate design of the data recipes from the Delta Zero. So actually we design our own devices of collecting such data. And here is a brief view of the collection devices. So it's basically an egocentric collection system with an extra fisheye camera aiming at the body part. So here is the fisheye cameras. So S is towards and down, towards the human body. So we can have a better view and extra view. Like this one is a fisheye camera. We can have an extra view of the human body. So this can increase the coverage. Because we have seen a lot of occlusion during the data collection system for the standard egocentric data collection system. So we want to adjust this kind of occlusion. Uh, in, in, in like, you know, in a better way. So we use this kind of visual camera. So we'd have a better coverage. We can use this view as input. We train a better model to estimate the whole body poses. I think this is one of the most promising way to scaling out the whole body data. Because right now we are using systems like motion capture systems for in-house data collections, whole body, and we use Pico, right? The motion capture systems, they are not scalable. And the Pico, honestly, is not very accurate, especially for sound. You need to do the calibration, and for the feed, it's not accurate along the way, a long time. So we designed our O-system to own...
00:42:44 · Siyuan Huang
Our system to capture the fisheye input and we estimate the body pose at the right side. So we can train a model to continually improve the estimation accuracies, including using some body prior. We have a better way of capturing the whole body actions, including the feet. And I think this is a critical step towards how we scaling up this kind of human data, because currently we have seen like maybe 90% or 99% of egocentric data is bound to upper body. So we have the hand pose. But we don't have the lower body information. So I think the device from the data can actually fill the gap between how we can collect the upper body data towards collecting the real full body data. So from the left side, the top left side, we can see that in the future, we want to develop the system, the data collection system, so we can have the full body and the whole 3D thing. And I think this is a good idea. This is a very promising way to really scaling up and expanding the horizons of how we can collect the human data, the interaction data. Imagine that, but currently we can train the foundation model with one million hours, right? More is about tabletop assortment and some long-range movement because we cannot reconstruct the accurate body movement from data. So the lower body is basically not used in this kind of videos. So in the future, we can have millions and maybe more videos for this kind of whole body movement. So then the humanoid robot can, we can train the foundation model. We can let the humanoid robot can directly learn from this kind of whole body data and behaviors can emerge. Like in context learning behavior or zero shot behavior can emerge from this kind of whole body data. I think this is our vision. I think Michael also shared with me about some of the work you are doing, trying to estimate the whole body data. I think that's definitely very, very promising. Maybe we can do some collaboration together to try to push this direction into a better shape.
00:45:18 · Michael Cho, Siyuan Huang, Chris Paxton
[00:45:18] S0: I'm curious what you think about the, I mean, even, so I completely agree, like ego to whole body is something that definitely the community should pay more attention to now. But there is still always the question about occlusion, right? So for example, right here, Yeah, you can kind of see the leak, but then if you can see the human stigma thing, it's not precise enough. Do you think it's just a matter of having enough data over time so that the model can just infer these things, even despite the occlusion? [00:45:51] S2: Yeah, I think the occlusion cannot be solved entirely by the fisheye camera. So in the future, we will also have a version so we can put the tracker onto the feed. So then we can have a combined solutions. The hardware is a tracker and software is a model estimations. So we can fuse the input from the tracker and also the individual input. We try to make the post estimation as accurate as possible and deal with occlusion. Yeah. That makes sense. Yep. [00:46:21] S3: I'm curious. So you, you said your, your robots using like the wuchi hands you were, you were saying earlier. So like you don't use the gloves or any tactile information or anything like, you know, cause that would also help handle occlusions, right? [00:46:33] S2: Yeah. You mean, you mean tactile, tactile sensor input? [00:46:36] S3: Yeah. I was curious, like, so, so you're, you're finding that most of you just need to do something with data?