The Robot Report Podcast
00:49:39 · Eugene Demetri
[00:49:39] S3: I hear a lot about emergent behavior, but so much, like you said, comes down to that probabilistic modeling and and weighing of of outcomes. So let's go down a couple of levels. We read a little bit about how both Atlas' arms and legs have been controlled by the same model, but with feet as additional hands. New approach and how does that work? There must be instances where behaviors for one might not work for the other.
00:50:08 · Scott Kuindersma
[00:50:08] S4: Yeah. Basically, what we did was so we have this teleoperation interface that we built that's based on a Vive style VR system. So there's a a head mounted display, and so people are looking into a VR headset. And so they don't have line of sight to the robot. They can only see through the robot's camera sensors. They have trackers on their wrists and torso and feet, and then they have some kind of controller for the hands. And we have two different things. One is a VR style controller where we can use button presses to switch between different hand shapes and then use triggers in order to open and close the hands. Or we have motion capture gloves that allow people to do finger level control of the hands for more dexterous things. And so when we say that the hands are... That the feet are essentially like additional hands, what we mean is that the trackers that are attaching to the feet and hands are essentially the same, and those get turned into just desired reference streams into an underlying controller that's coordinating the entire body. So part of the teleoperation system is resting on the years of work that we did in whole body control. So we're able to basically just take these raw streams of information from the VR system and throw it into this whole body controller. And it's able to do all the coordination, maintaining balance, avoiding self collisions. All that stuff is just happening underneath the teleoperation so that the human being doesn't have to worry about all that. A person's not literally trying to balance the robot through moving their own body, or they're not really worried about, well, if I move my foot from here to here too quickly or whatever, it sort of doesn't matter because the underlying controller is gonna sort of do it in a way that doesn't knock the robot over. And then once we've done that and we collect the data, the policy uses the exact same interface as the teleop system. So from the underlying control system, it doesn't really know if it's a teleoperator or if it's a policy. And so that's also been a really flexible way to architect the thing.
00:52:22 · Steve Crowe
[00:52:22] S0: Are there things holding back how quickly you guys could scale these behaviors on Atlas to more tasks or even broader tasks? Like, I think you guys use the new Jetson Thor platform, but is it computing power? Is it training data and diversity of that data? Like are those sort of, I don't mean this in a negative way, but what are those bottlenecks that are holding you guys back from taking this even further more quickly?
00:52:50 · Scott Kuindersma
[00:52:50] S4: Yeah, for sure it's data. That's just because in order to collect, like the highest quality data that you can get is data for the tasks you care about on the robot you care about. And there's just a limited amount of that. So you only have so many humanoids. Humanoids are among the most complex robots that anyone's trying to do this kind of work on. So, I think as you look at other types of robot platforms like table mounted arms and things like that, I think your ability to scale it up to just massive data collect facilities is a little better than it is with humanoids. So, I think that's one thing. But one of the really exciting things in this space is that there's other scaling bets that we can leverage here. So I think if the only answer we believe was true is that you needed on hardware experience in order to train these models and nothing else would work. That would be pretty problematic because I think in order to get to even like GPT, I don't know, two level scale would be a tremendous undertaking, right? And so what people tend to talk about is something like a data pyramid. And so you could think about the tip of that pyramid as being the highest quality data that if you just had more and more of that, things are only gonna get better. And so that's sort of like the teleoperation on Atlas. And then as you go down that pyramid, the scalability proposition gets a lot better, but the embodiment gap maybe gets bigger as well. So maybe if I go down a little bit, I can talk about simulation data, right? So if I go down somewhere in the pyramid, having simulations of my robot doing manipulation tasks could be used as co training data. So this is data that we throw into the mix with the real robot data. We've actually found in our experience that, and other groups have too, that incorporating simulation data as part of the training mixture actually improves performance on the physical robot, which is really exciting. One of the other opportunities about humanoids specifically is maybe there's more of an opportunity to use human demonstration data that isn't teleop. So if I have recordings of people, people wearing instrumentation, egocentric videos of people doing manipulation tasks, if their bodies are not too dissimilar from my robot's body, that may be an easier thing to translate into and inform robot behavior.
00:55:17 · Scott Kuindersma, Eugene Demetri
And then you go all the way down to probably the bottom of the pyramids like YouTube. So it's like very unstructured video of people doing things and it's just a harder thing probably to leverage into real concrete improvements in robot behavior. But it'll probably be part of the story. So from our perspective, we're thinking about that whole set of things. And I think, you know, so is many other groups. And I think everyone's sort of taking different bets on the relative proportions that they're expecting to be part of the right answer, and they're investing resources in different parts of that pyramid. So you see people with like giant factories filled with teleoperators, and then some groups with like relatively modest tele operations, but a lot of investment around egocentric data collector simulation or things like this. [00:56:10] S3: And you mentioned before that obviously, Toyota Research Institute as a partner has been helping with that. Can you talk about that ongoing relationship for a moment? Isn't it something that you're expecting to continue?
00:56:10 · Eugene Demetri, Scott Kuindersma
[00:56:10] S3: And you mentioned before that obviously, Toyota Research Institute as a partner has been helping with that. Can you talk about that ongoing relationship for a moment? Isn't it something that you're expecting to continue? [00:56:21] S4: Yeah, for sure. Yeah, we're continuing to work together. I think us working with them has been just so much fun. They're an awesome team. I think we are very similar in terms of our outlook and what we think is important. The structure of how we've been working together is, we have a special space in our building that's kind of just like a research lab, and there's research scientists from TRI who are mixed in with our own research scientists in there, and they all kind of just work as one big team. We'll get some Atlas robots in there. So it's been really great, and I'm looking forward to continuing to work with [00:56:56] S3: And are you also working with the AI robotics institute as well?