Going Direct Conversations
00:30:04 · Jordan Wolfe, Michael Cho, Chris Paxton
[00:30:04] S2: Let's go over to RoboPapers. Thank you for that. So actually, before we do that, I wanna, I guess if we were to step back, when you think about all these different training methods, and obviously several folks working on a variety of both open source and their own closed source robot foundational models and multimodal models and world foundational models, whatever you wanna call it, I want to get each of your perspectives on, on, on, on how do we, how do we ultimately solve this training gap? Like what is the thing that we can do today to move the field forward in the most effective way, I guess, uh, around the training gap. If there's one thing that you pick, you pick, we could do. Gosh, [00:30:46] S1: uh, I think you need, yeah, it's very hard to pick one thing because like, it's a, it's very unclear which one is actually going to scale. Like all, all these, uh, [00:30:55] S0: I mean, if it's like wave your magic wand, infinite money. Sorry. Go ahead. I have an answer.
00:30:55 · Chris Paxton, Michael Cho, Jordan Wolfe
[00:30:55] S0: I mean, if it's like wave your magic wand, infinite money. Sorry. Go ahead. I have an answer. [00:31:01] S1: I have talked to so many different researchers now. Different researchers have different, like, kind of, let's say, I don't know, like, inclination of biases. Some people think, you know, TeddyWarp is really the way. Some people are really betting on, like, role model. I mean, I should say something about simulation, right? So to me, Simulation data. Simulation have kind of two types now. You have the traditional simulation engine, and then you have the wall model, which is the new stuff. Because the wall model is, in a way, a simulator as well. It just behaves very, very differently and trains very differently. I think simulation has got to... Okay, if I were to bet, I feel that simulation alone can get us 80%, 90% there. It's my guess. Okay. But because you need 99.9999% so it's not enough therefore. But it can probably move us all near to 90 something percent. The issue with simulation, I don't know whether Chris you agree with me, at least with the classical simulation engine, is that I believe contrary to popular belief, we're not bottlenecked by compute actually, I think. I think that in a way, the its capabilities is also its curse, right? Because with a very good simulation engine, you can literally simulate anything you want. And that's a big problem because the search space or the exploration space is nearly infinite. [00:32:26] S2: It's too wide, yeah.
00:32:28 · Michael Cho
[00:32:28] S1: Yeah, you can imagine, you can use a simulation engine to, let's say, simulate a robot frying egg, let's say. You can imagine in your head maybe an infinite way to set up that scene, that task. Different lighting, different robots, different eggs and whatnot. And so it will take a crack researchers like Chris to drastically shrink that search space into something that is tractable with the compute budget that he has. But as researchers do that, well, so, you know, they are missing a lot of actually potential edge cases that may happen in the real world that they could have simulated, but they just don't know. I mean, they would have messed up. And so I think that's a big part of it. So if I look at it that way, actually the bottleneck to a simulation, a big bottleneck is having more people like Chris, honestly, because that process is very curated. Like it's very handcrafted. Like you need, like Chris will have to come up with very like a man, like handcrafted, like the rewards functions for the reinforcement learning, things like that. It's all very handcrafted. So, which means actually the human in the loop, and unique researchers to do it. Like, you know, it's a very, very specialized skill set. That's a big bottleneck. World model, on the other hand, all this video generation, I think it's also very, very interesting. I think they potentially can move the needle a lot. But you can imagine to build a world model, you also need a lot of probably like video data and tele-op data too. So you still need a lot of data, a huge amount of data to build a decent world model, right? Yeah, so it's hard to pick one. I feel that tele-op data, video data, simulation, all are needed. Like we, we need all. And there's a lot of papers that we've come across, you know, that we've interviewed that do co-training of various sorts, right? They'll, they'll, they'll, they'll, they'll scale it using some video or like, you know, do the real to seem the real thing.
00:34:24 · Jordan Wolfe, Chris Paxton
[00:34:24] S2: Yeah. So, yep. Sure. What's your thought, Chris? If there was kind of what, if you have one thing that kind of stands out in your mind, if someone who's actually in there doing, you know, working on it every day, [00:34:34] S0: I mean, I think that if you were going to wave a magic wand and had infinite money, then I think you'd just have lots and lots of robots doing. I think that they worked. But practically speaking, I think Michael's correct. It seems like you need a little bit of everything, because they're all good at slightly different things. And they all have different pluses and minuses. Sure. Yeah. So the real solution is going to be a mix. So, Chris, I actually saw a recent tweet that you put out that caught my eye. [00:35:07] S2: And I don't have it in front of me, so I might butcher it a little bit. But basically, the premise was the reality of it is if we want low-cost generalized robots, we need a bunch of robots in the field learning and actually doing stuff. That's,
00:35:07 · Jordan Wolfe, Chris Paxton
[00:35:07] S2: And I don't have it in front of me, so I might butcher it a little bit. But basically, the premise was the reality of it is if we want low-cost generalized robots, we need a bunch of robots in the field learning and actually doing stuff. That's, [00:35:23] S0: yeah, that's increasingly kind of what I, my opinion on a lot of this is just like, it'll be solved with that. Like we have, and that's one of the things is that like the robots are so much better. Like, so I, I've, I've been doing this since taken one form or another since like 2014. Right. So like I've been doing robotics for a long time. And when, when I started, we paid like $40,000 for a UR5 industrial arm that failed like every few days. And like we had a, we had another, like this really, really, not so great robot arm that like, They had hand-tuned PID controllers where one of the grad students stayed up and slammed into the wall and broke it into a million pieces and that kind of stuff. And so this was the reality. And that shuts down the lab for months, many months. So that was the reality for a long time. And now you can buy a $20,000 unitree robot from Walmart. Or you can 3D print one of those SO-100 or What's it? Vector's robot. I forget that. The robot. The various little robot things. You can 3D print. You can build these things. We're so much closer to that world with infinite robots everywhere, I think, than we used to be. So that's...
00:36:42 · Jordan Wolfe, Michael Cho
[00:36:42] S2: Yeah, it's cool. It's cool seeing it all come together. And as someone who, I guess, without that kind of background in robotics and the more folks that I speak to, especially on the research side, and someone always thinking about like market, market application and how to get adoption and useful application of these different robotic systems. Yeah. It's very clear that it's such an exciting time. It's very clear that there's a lot of movement and it's moving fast, but yet I'm challenged because most of the robotics companies that I see, I think the path to any type of adoptability or market adoptability is very challenging. still at this case. And so I don't know if you guys think from like a market viability standpoint that the vast majority is still not viable or do you actually see kind of a window here where it's going to start becoming really viable with any type of scale? Well, I think it's absolutely viable if you look at it as an entertainment thing, right? [00:37:40] S1: So Unitree is putting up... I love it. [00:37:43] S2: He's pulling in the entertainment plug. Give it to us.