The Robot Report Podcast
00:29:51 · Steve Crowe, Scott Kuindersma
When Boston Dynamics released this video of its new Atlas humanoid doing some work with spot parts, picking up legs and faceplates and putting them in one bin and adding them to another bin. Fascinating video, over a million views on YouTube at this point. So fortunate enough to be joined by Scott Kuindersma. He's the VP of robotics research at Boston Dynamics. He's been there for almost seven and a half years now. He's obviously seen some fascinating things. So he's going to help us unpack as much as he can large behavior models and how they're helping Atlas learn new skills and generalize to new environments. So Scott, thanks so much for being there. How are doing, man? [00:30:34] S4: I'm doing well. Thanks for having me. [00:30:36] S0: Yeah, and all the pleasure is ours. Open invitation anytime you guys want to come on the podcast. We're here for you. So the video that we're trying to break down, was called the Spot Workshop, I guess the task that you guys were undertaking here. And I always find it fascinating, I shouldn't, maybe there's a very simple reason that you guys pick this specific task. I always love when robots are working with other robots or picking pieces or parts of other robots. So again, you guys are using Atlas to pick parts of the Spot quadruped. Why this task? Why was this important enough to undertake and film? Explain the backstory as much as you can.
00:31:17 · Scott Kuindersma, Steve Crowe
[00:31:17] S4: Yeah, for sure. So oftentimes when we pick tasks to work on to develop new algorithms, new ideas at Boston Dynamics, we're usually trying to isolate some features, like some some behavioral features or some technical properties of those tasks. And then the semantic content of those tasks can just sort of be whatever. And sometimes we'll decide to make it fun or sometimes make it, like, on brand, like, with the Spot parts. So here, what we are trying to do is pick tasks that involved doing bimanual manipulation over the entire robot's workspace, so being able to pick things up from very low, put them up high, and move around with environmental constraints. So shelves that are close to the robot or bins that have to be pulled between the legs. And reason for that was the strategy that we're using to program the robot here is a demonstration based programming, right? So we're using teleoperation as a way to demonstrate examples of the behavior that we want the robot to perform. And then we're using basically machine learning to take that data and turn it into a model that can perform many of the tasks that we demonstrated. And one of the things we're interested in is, of course, we've seen a lot of people using these types of techniques in the last couple of years. And a lot of that's focused on kind of largely sort of tabletop style manipulation. And [00:32:49] S0: one of
00:32:49 · Steve Crowe, Scott Kuindersma
[00:32:49] S0: one of [00:32:49] S4: the things we're excited about with humanoids, of course, is the full breadth of manipulation skills we know they can perform. And so what we wanted to do was show that we could first build an interface to the robot that would allow us to do a really expressive set of data collection. So I could teleoperate the robot to do almost anything that it could physically do, right? Stepping, crouching, picking up things with one or both hands, hand level manipulation, things like this. And then once we were able to do that and collect that data, show that the same sort of techniques that have been gaining traction in the larger behavior models literature also translate to this task space where it's now not just moving hands around and moving things on a tabletop, but actually coordinating an entire humanoid robot body to do a manipulation task. So once we decided, you know, this is the thing we wanted to do, it was really just a matter of walking around our building and trying to figure out what are some tasks that kind of tick these boxes. And we literally walked into the Spot Repair Shop and just saw their shelves of all these Spot Parts. And we saw what the people in that shop were doing. And we thought, okay, we could just pull all these elements together and create a task in our lab based on it. And that's what we did.
00:34:05 · Eugene Demetri
[00:34:05] S3: And that's the cool thing about Boston Dynamics is doing with humanoid robots, right? That we're taking full advantage of real world surroundings and applications and not just having a humanoid do what you have some other robot can already do, but to take advantage of the humanoid form factor. But you mentioned a large behavior models, and we've heard a lot about large language models, but maybe not everyone's familiar with LBMs. Can you explain what they are and how are they helping Atlas learn new skills?
00:34:35 · Scott Kuindersma
[00:34:35] S4: Yeah. So the naming of it, which I think was coined by our collaborator CRI, is meant to draw an analogy to large language models. And so basically the story that we all know from large language models and vision language models and all these multimodal variants is that by pulling together large amounts of diverse data and training a single model, basically using supervised learning, sometimes other types of learning to do additional fine tuning after. But by compiling this all into one model, you can actually use that same model to do many different downstream tasks. We can build large language models that are really good at just being chatbots, but also good at solving math problems or coding problems and things like that. Observation is that, well, the same sort of generalization So, should be available to physical there's nothing inherently special, we think, about language or about images or anything like that, that if we also had a similar sized corpus of embodied data that involved robot state observations and actions that the robot took to change the state of itself and of the world, that if you fed these models that same sort of, that kind of data, that they should be able to generalize in similar ways over behavior. And so that's the sort of hypothesis. And so what we were doing here is using primarily teleoperation data on a couple of different robot platforms over many, many tasks, and then compiling that all together to train a model that is able to take information from the robot sensors, information from the robot cameras, and language commands, and turn that into closed loop manipulation policies like we showed in the video.
00:36:29 · Steve Crowe
[00:36:29] S0: So we'll link to this blog that you guys wrote. It's fascinating. It gets pretty in-depth to all this stuff as well. And in it, you mentioned there's a four stage pipeline you use to, I guess, train these LBMS, Teleop, to collect the data and there's data processing and annotation model training and evaluation. Can you quickly take us through that four stage effort for this specific task? I mean, how intensive is it? How long does it take to collect and fine tune all the data that was necessary to put this together?