00:39:47 · Siyuan Huang
And it can have some adaptations or can have some transitions from transition appearance or behaviors from task to another. And the exception is specified by the objectives. So delta 0 must learn how to reposition its body. So the position must be accurate before starting from the next actions. So this is the design we consider about the So the next section is about how we scaling the reliable humanoid intelligence. And this is quite interesting. We decompose this into three parts. The first part is we want to scale the data. We want learning from human data. I think from the first principle, we Doing all of this foundation model for humanoid robots is that we believe that human data is the richest source of robot learning. So we try to make use of this kind of high quality or low quality human data, including the upper body and also the lower body. We try to make use of this kind of data for the learning of the Delta Zero. And this motivates the design of the action space and motivate design of the data recipes from the Delta Zero. So actually we design our own devices of collecting such data. And here is a brief view of the collection devices. So it's basically an egocentric collection system with an extra fisheye camera aiming at the body part. So here is the fisheye cameras. So S is towards and down, towards the human body. So we can have a better view and extra view. Like this one is a fisheye camera. We can have an extra view of the human body. So this can increase the coverage. Because we have seen a lot of occlusion during the data collection system for the standard egocentric data collection system. So we want to adjust this kind of occlusion. Uh, in, in, in like, you know, in a better way. So we use this kind of visual camera. So we'd have a better coverage. We can use this view as input. We train a better model to estimate the whole body poses. I think this is one of the most promising way to scaling out the whole body data. Because right now we are using systems like motion capture systems for in-house data collections, whole body, and we use Pico, right? The motion capture systems, they are not scalable. And the Pico, honestly, is not very accurate, especially for sound. You need to do the calibration, and for the feed, it's not accurate along the way, a long time. So we designed our O-system to own...
00:42:44 · Siyuan Huang
Our system to capture the fisheye input and we estimate the body pose at the right side. So we can train a model to continually improve the estimation accuracies, including using some body prior. We have a better way of capturing the whole body actions, including the feet. And I think this is a critical step towards how we scaling up this kind of human data, because currently we have seen like maybe 90% or 99% of egocentric data is bound to upper body. So we have the hand pose. But we don't have the lower body information. So I think the device from the data can actually fill the gap between how we can collect the upper body data towards collecting the real full body data. So from the left side, the top left side, we can see that in the future, we want to develop the system, the data collection system, so we can have the full body and the whole 3D thing. And I think this is a good idea. This is a very promising way to really scaling up and expanding the horizons of how we can collect the human data, the interaction data. Imagine that, but currently we can train the foundation model with one million hours, right? More is about tabletop assortment and some long-range movement because we cannot reconstruct the accurate body movement from data. So the lower body is basically not used in this kind of videos. So in the future, we can have millions and maybe more videos for this kind of whole body movement. So then the humanoid robot can, we can train the foundation model. We can let the humanoid robot can directly learn from this kind of whole body data and behaviors can emerge. Like in context learning behavior or zero shot behavior can emerge from this kind of whole body data. I think this is our vision. I think Michael also shared with me about some of the work you are doing, trying to estimate the whole body data. I think that's definitely very, very promising. Maybe we can do some collaboration together to try to push this direction into a better shape.
00:45:18 · Michael Cho, Siyuan Huang, Chris Paxton
[00:45:18] S0: I'm curious what you think about the, I mean, even, so I completely agree, like ego to whole body is something that definitely the community should pay more attention to now. But there is still always the question about occlusion, right? So for example, right here, Yeah, you can kind of see the leak, but then if you can see the human stigma thing, it's not precise enough. Do you think it's just a matter of having enough data over time so that the model can just infer these things, even despite the occlusion? [00:45:51] S2: Yeah, I think the occlusion cannot be solved entirely by the fisheye camera. So in the future, we will also have a version so we can put the tracker onto the feed. So then we can have a combined solutions. The hardware is a tracker and software is a model estimations. So we can fuse the input from the tracker and also the individual input. We try to make the post estimation as accurate as possible and deal with occlusion. Yeah. That makes sense. Yep. [00:46:21] S3: I'm curious. So you, you said your, your robots using like the wuchi hands you were, you were saying earlier. So like you don't use the gloves or any tactile information or anything like, you know, cause that would also help handle occlusions, right? [00:46:33] S2: Yeah. You mean, you mean tactile, tactile sensor input? [00:46:36] S3: Yeah. I was curious, like, so, so you're, you're finding that most of you just need to do something with data?
00:46:41 · Siyuan Huang
[00:46:41] S2: Yeah, we actually tried some tactile solutions, but we think current is not very stable. I'm not sure how you change this. We tried some solutions, including some hardware companies in China, But we didn't find it is very stable and very efficient. So we did not use tactile in the current version. But actually in the future, the contact and also the tactile, the whole body contact, right? The elbow, the feet, we can put a contact sensor on each part of the human. And then we can put the tactile sensor on the fingers. I think they are all useful. But we are not sure how useful they are in the current stage because I think the roadmap is that we want to try to explore the limit of the visual input. And then we combine the visual input with the contact sensor, contact input, and the tactile input. And then we can see how they really improve the performance based on the model utilizing the visual input. Right. Because in our experiments, including the, using Sharpa hand, using WUJI hand, we found out that the improvement of the tactile is quite marginal for the task that is, that can already have huge amount of data when you have the visual input. So it's not very clear right now how this accurate tactile data can actually help the visual modalities. Except for this extremely dexterous task, I think the tactile is pretty cool. But for daily interactions, I'm not sure how good the tactile can really improve based on the visual input. And then we can, so here... This is the data recipe we want to emphasize for the data zero. So basically what we did is that we used the open source, the robotics data, the dual arm, the ego, the UMI, the whole body motion, the human robot motions. And we want to convert this kind of embodiment into one action space. So we actually separate them into the left, right arm, and also the body command space. And we convert this kind of robot data and human data into a single... So we can pre-train based on this action space. And we do some evaluation based on the tracking trackers and also the scaling law of the tracker and also the brain from the data zero models. On the left image that shows the scaling law for the The X axis is the motion training coverage for the controller and the Y axis is the high precision tracking average.
00:49:38 · Siyuan Huang, Michael Cho
So basically evaluate the coverage percentage of the motions that tracked zero shot with mean wrist error within two centimeters. We think this is a good measure for how this tracker can really track the whole body local manipulations. And we see a scaling effect that proves if you have more motion data and if you want to, if you can really by scaling up the motion models, you can have a better tracking performance. I think this is critical because For human or robot, we want a better controller. So we can have a proof that we can scan out the motion amount, then we can have a lower and lower tracking performance because if the If a tracking error is extremely low, then you can have a better bridge between the brain and the controller. And you can even switch the controller if the tracking, switching the controller, you are using the combined controller solution if the tracking error is low enough. [00:50:52] S0: Siyuan, I have a quick question. Just now, you were saying that you basically combined the different, just above, you have Ego, you have UMI, basically various kind of data sources. I think you kind of unify them in the same action space, right? I'm curious, like, for things such as UMI, how much does it actually help when it comes to humanoid controller? Or is it, do you think, actually, Ego and just bet on Ego? You mean how UMI can help the controller? Yeah, because you have a mix, right? It seems like in your training data, you have a mix of UMI, you have a lot of ego.
00:51:29 · Siyuan Huang
[00:51:29] S2: Yeah, the mix data is for the planner, basically for the brain, it's not for the controller. So for the controller, it's mostly about human whole body data, whole body captures. So the experiment is conducted on the whole body data. On the right side, we have ablations of the training data amount and the scaling effect of the planner. So basically the X axis is about pre-trained data usage and the Y axis is the success rate. We compared in Like the bimanual depthless task and also whole body depthless task, local manipulation task. We see a steadily increase along the usage, more usage of the return data. And in this version, we... We focus on the data recipe. We mainly try to incorporate the whole body data into the training. So we use about 10,000 hours and we currently are trying to skim up this amount to like 100,000 hours. And try to see like how this can differ or how this can really behave when we scale up this kind of data into a huge amount. And we show some experiments about how the robot can be robustly behaving. So we're changing the lighting conditions and we're using the lighting as an interactions, interruptions. So we test the robustness of the models. And it actually shows that 0 is quite robust to the lighting condition variations and certain interventions. So we also did some experiments about putting some interventions into the task. For example, we add an extra cup onto the table. I want to see if Robo can continue to the task. When you see one cup in the tray and one cup is still on the table, but actually the model can successfully doing this kind of task with different scenario from the training set, from the post-trained data. And also we... I tried to do the intervention. We closed the trash can after the robot finished the task and the robot can still doing the task robustly, stepping on the paddle. And doing this repeatedly, this really fun. So this kind of experiment actually proved that the model can have certain robustness to lighting conditions, to different interventions. So it can provide a basis for the fusion deployment for some indoor scenarios and can have more safer behaviors, can have safe behaviors when you are doing this kind of interventions. And the second part of scaling is about evaluations.