00:46:41 · Siyuan Huang
[00:46:41] S2: Yeah, we actually tried some tactile solutions, but we think current is not very stable. I'm not sure how you change this. We tried some solutions, including some hardware companies in China, But we didn't find it is very stable and very efficient. So we did not use tactile in the current version. But actually in the future, the contact and also the tactile, the whole body contact, right? The elbow, the feet, we can put a contact sensor on each part of the human. And then we can put the tactile sensor on the fingers. I think they are all useful. But we are not sure how useful they are in the current stage because I think the roadmap is that we want to try to explore the limit of the visual input. And then we combine the visual input with the contact sensor, contact input, and the tactile input. And then we can see how they really improve the performance based on the model utilizing the visual input. Right. Because in our experiments, including the, using Sharpa hand, using WUJI hand, we found out that the improvement of the tactile is quite marginal for the task that is, that can already have huge amount of data when you have the visual input. So it's not very clear right now how this accurate tactile data can actually help the visual modalities. Except for this extremely dexterous task, I think the tactile is pretty cool. But for daily interactions, I'm not sure how good the tactile can really improve based on the visual input. And then we can, so here... This is the data recipe we want to emphasize for the data zero. So basically what we did is that we used the open source, the robotics data, the dual arm, the ego, the UMI, the whole body motion, the human robot motions. And we want to convert this kind of embodiment into one action space. So we actually separate them into the left, right arm, and also the body command space. And we convert this kind of robot data and human data into a single... So we can pre-train based on this action space. And we do some evaluation based on the tracking trackers and also the scaling law of the tracker and also the brain from the data zero models. On the left image that shows the scaling law for the The X axis is the motion training coverage for the controller and the Y axis is the high precision tracking average.
00:49:38 · Siyuan Huang, Michael Cho
So basically evaluate the coverage percentage of the motions that tracked zero shot with mean wrist error within two centimeters. We think this is a good measure for how this tracker can really track the whole body local manipulations. And we see a scaling effect that proves if you have more motion data and if you want to, if you can really by scaling up the motion models, you can have a better tracking performance. I think this is critical because For human or robot, we want a better controller. So we can have a proof that we can scan out the motion amount, then we can have a lower and lower tracking performance because if the If a tracking error is extremely low, then you can have a better bridge between the brain and the controller. And you can even switch the controller if the tracking, switching the controller, you are using the combined controller solution if the tracking error is low enough. [00:50:52] S0: Siyuan, I have a quick question. Just now, you were saying that you basically combined the different, just above, you have Ego, you have UMI, basically various kind of data sources. I think you kind of unify them in the same action space, right? I'm curious, like, for things such as UMI, how much does it actually help when it comes to humanoid controller? Or is it, do you think, actually, Ego and just bet on Ego? You mean how UMI can help the controller? Yeah, because you have a mix, right? It seems like in your training data, you have a mix of UMI, you have a lot of ego.
00:51:29 · Siyuan Huang
[00:51:29] S2: Yeah, the mix data is for the planner, basically for the brain, it's not for the controller. So for the controller, it's mostly about human whole body data, whole body captures. So the experiment is conducted on the whole body data. On the right side, we have ablations of the training data amount and the scaling effect of the planner. So basically the X axis is about pre-trained data usage and the Y axis is the success rate. We compared in Like the bimanual depthless task and also whole body depthless task, local manipulation task. We see a steadily increase along the usage, more usage of the return data. And in this version, we... We focus on the data recipe. We mainly try to incorporate the whole body data into the training. So we use about 10,000 hours and we currently are trying to skim up this amount to like 100,000 hours. And try to see like how this can differ or how this can really behave when we scale up this kind of data into a huge amount. And we show some experiments about how the robot can be robustly behaving. So we're changing the lighting conditions and we're using the lighting as an interactions, interruptions. So we test the robustness of the models. And it actually shows that 0 is quite robust to the lighting condition variations and certain interventions. So we also did some experiments about putting some interventions into the task. For example, we add an extra cup onto the table. I want to see if Robo can continue to the task. When you see one cup in the tray and one cup is still on the table, but actually the model can successfully doing this kind of task with different scenario from the training set, from the post-trained data. And also we... I tried to do the intervention. We closed the trash can after the robot finished the task and the robot can still doing the task robustly, stepping on the paddle. And doing this repeatedly, this really fun. So this kind of experiment actually proved that the model can have certain robustness to lighting conditions, to different interventions. So it can provide a basis for the fusion deployment for some indoor scenarios and can have more safer behaviors, can have safe behaviors when you are doing this kind of interventions. And the second part of scaling is about evaluations.
00:54:23 · Siyuan Huang
So, um, we, we're trying to build up a real-to-sim-to-real pipeline for evaluating the Shimano robot because doing the Shimano robot experiment is so costly, especially when the However, it's still not that good enough in the current stage. So we try to reconstruct the 3D things into a simulation, into a simulator, and try to using the performance in the simulator to do a coloration test compared with the real world experiments. So actually the left side is the demo room, and right side is a replica, the 3D replica of this room. So we can use tools like, including the GPT-6 extra, To do multi-run verifications or improvement of this kind of environment and have a very accurate replica, including the turntable and also the records, this kind of stuff. Basically, we can build up all these scenarios into these simulations. And then we can actually do a rollout of the same policy in simulator and also in the real world. So this is a dishwasher scenario. So actually we are using the exactly the same models that is tested in the real scenario. So here is an example and the task is absolutely the same, like picking up the cup and put it into the dishwasher and then close. And interestingly, Previously, I think Chris knows a lot about the 3D reconstruction. It's really hard to verify or to identify the articulations of this kind of objects in the simulators. But with the tools like Astra, we can actually do the automatic, the physical system identifications about the This kind of articulation structures and about the stiffness damping because we can using the demos from the humans or demos from the robot as input to do the identification. So the real-to-sim system can actually search a parameter, the articulation parameter or the physical parameters that can match the real scenario from the demo-sims. But this robot does test a bad to have a very accurate replica. Including the... Including the articulated objects from the real world. And then we can do the real to same evaluations. We're trying to find the coloration between the simulator and the real world. So then if you have a new model, you can iterate the model in a faster speed. We can do the coloration test. So the process is that we do the exact, using the exactly the same models, but doing the repeated trials in simulator and the real world.
00:57:21 · Siyuan Huang, Michael Cho, Chris Paxton
And we compared the coloration and find out the linear, Linear relations between the models. So in the future, when we have a pre-trained model, we can do all the Austrian experiments in the simulators. Then it can prove how good it is in the real-world experiment. We do not need to use the Jiva robot or A3 robot to do thousands of trials in the real world because it's too costly for the current stage. [00:57:52] S0: Yeah, I just want to say, like, I'm surprised by the absolute difference. Like, there's very, very little difference between the simulation and real world. I mean, technically, I don't know, other papers we've seen, usually, the gap is quite wide. I mean, usually, the simulation would give you a sense of the trend, but in terms of absolute, usually, they are still far apart. This one is very close. Yeah. It's pretty close. 89, 86. [00:58:16] S2: Yeah. Yeah. Does it look like it's for every task? Yeah. We are not sure if it's for a task because for this task called the dishwasher, actually we did a lot of cross-verifications. We tuned the physical parameters. We are definitely expanding this kind of experiment to different other tasks. [00:58:38] S3: And all that tuning the physical parameters, was that done by the agent? Was that done by Astra or did you guys manually? [00:58:44] S2: I think it's 90% is done by the agent and 10% by some human neighbors.
00:58:44 · Siyuan Huang, Xiaojian Ma
[00:58:44] S2: I think it's 90% is done by the agent and 10% by some human neighbors. [00:58:58] S1: So the way we're doing this is actually we're asking the GD6 extra to do this real-to-scene simulator parameter tuning. We actually set a goal for this agent model is that we're actually providing this agent with a trajectory, collecting the real world. The trajectory including both the robot movement and also what the robot actually sees. So basically the camera observation. So what we're asking the agents to do is to try its best to replicate the same motion in the simulator to not just So that kind of metric actually forced the agent to do his best to try to tune the parameters to make sure that the motion can be replicated in a faithful kind of way. But another thing you can see in this is that we are actually tuning the physical parameter of this simulator for this certain motion, for this certain task. So what we're trying right now is that we're trying to expand this approach to other set of tasks, other set of motions well, but what I'm not sure is that maybe when the agent's trying to just replicate this very different kind of motions, it might lead to some contradicting parameters at some point. But we're still working on that.