00:50:13 · Ademi Adeniji
[00:50:13] S2: Yeah. I I guess I can... I have a view on this, and then maybe, Vincent, you also have a view. So I guess my perspective is ultimately in order to get, I think superhuman embodied AI and really the capabilities that I think we all, we'd all dream about, like, and we see in sci fi, I really think you want autonomous real world learning. I think sort of learning, I think for me in terms of the research I've done is exploration is a key problem in AI. And a lot of our reinforcement learning algorithms are built around the assumption that our exploration algorithms are somewhat efficient and we can get data that's surprising or there's high bits of information to learn from. I think once we can get into the regime where we have some sort of prior or inductive bias around how to operate in a very basic way in the world, I think we then want to be looking towards autonomous learning in order to push the manifold of manipulation and hopefully have robots begin to even exceed our own capabilities. So I guess to answer your question directly, I think I'm less sort of... The big question in my mind is not okay. Are we going to be teleoping robots once they're deployed? But how can we get into the regime where robots can basically just operate with their own intelligence and be curious about things like humans are and begin to autonomously develop sensory motor capabilities. I think that's sort of the end game here that I'm most excited about.
00:51:52 · Vincent Liu
[00:51:52] S1: Yeah. I I generally agree. I think, like, there's... I agree with that sort of breakdown of there being, like, this bottleneck, which is chicken and egg problem, which is how can robots efficiently collect and explore on their own... Or collect data and explore on their own. And I think I actually had read one of Chris's Substack posts, and I kind of agree with this, which is like, it's not necessarily is it human data or sim data or teleop data. Like, there's probably efficiency gains that can be... That, like, each of these possesses. And I guess for for our perspective, it's like, how far can you get with just human data? And, you know, because that sort of has some nice properties here. And so for the second part, you know, after you're able to get, like, this very good base model, I like largely... I like largely agree with the idea that okay, then your robots are autonomous enough to go like explore and learn on their own. And I don't know. I'm a I'm a... I kinda like that idea, but recently, I read, like, this limits of RLVR paper, which I've been sort of thinking about. And this is sort of like a language model paper where, you know, you have this idea of a base model that is very powerful, and you then sort of put it in a RO training on tasks where you can verify, like, success or not. So it's like coding or, like, math perhaps. And what they showed is that when you start with a strong base model, which is is kind of like a requirement for efficient exploration, and then you sort of, like, train it with reinforcement learning in, like, some simulator, they showed that RO basically doesn't produce any new, like, superhuman behaviors in a way. And they sort of demonstrate this by plotting the... They basically sent... They increased the PASAT k of these base models. So, basically, you know, if you just let the base model retry, like, a 100 times, it ends up performing in many cases better than the RL trained model. And so, you know, I don't I don't know, like, what... If we're never... We're not even close to, like, that regime in robotics, I suppose. But in my head, I'm like, okay. Maybe practically, there's again some algorithmic things that would prevent you from just this idea of once you're past the chicken and the egg problem that you can just do RL and you become superhuman.
00:54:05 · Vincent Liu, Chris Paxton
It's like, maybe that does exist, but it's like that... The point at which you do that, you might already be hitting, like, five significant digits of, like, nines, right, in your success rate. [00:54:16] S3: I do think that was a super interesting paper, but at least in the robotics space, I feel like like in language, they always have... ROVR is always, like, discrete success terms, right, which we never use in robotics for RL. Right? Like, it's like like, you can always, like, penalize energy or speed or something. Like, it should always... I I... I'm more optimistic about robotics RL, I think, than that. But [00:54:39] S1: I mean, what [00:54:39] S3: is say it doesn't have its limits either. [00:54:42] S1: Right. But I mean, like, if you... I mean, if I if I imagine, like... I'm relatively new to the robotics field. Right? But if I were to imagine, you know, if you had RL in real, like, learning, you'd probably be also just assigning zeros and ones to whether it completed the task or not. Right? And in a way you wouldn't expect it to explore in a very, like, in a very... In a way that, like, diverges from its base behavior.
00:55:04 · Ademi Adeniji
[00:55:04] S2: I think... Can I chip in here? Like, I I have, like, maybe the other side of Chris, what you said. I think reward shaping is maybe one way that you can coerce certain behaviors, but I think is maybe subject to the same ideology that Vincent is describing. Of course, this is one study. Think generally, you need a collection of works for us to be convinced of a direction. Think my take is actually that we should go the opposite direction and use less supervision in this regime. I think there's so much interesting work in the unsupervised RL community and the general exploration that shows that you can learn surprising behaviors by doing things like trying to optimize for model uncertainty or trying to optimize for going to areas where you have high disagreement about what the future holds. And so I think these sorts of algorithmic approaches that are less constrained are the ones that maybe are going to lead us to the capabilities that surprise us and do things that maybe humans wouldn't do, or maybe humans can't specify concisely with a reward function. But again, you do need a really good prior because these methods are like, not only are they RL methods, but they're also like without sort of a direction in some sense. So the exploration problem is is very high dimensional and inefficient.
00:56:28 · Vincent Liu, Chris Paxton, Ademi Adeniji
[00:56:28] S1: Wait. So, Demi, what do think would be missing from, like, the limits of ROVR work, right, where it is... You know, you're able to explore efficiently because you're in a simulator effectively, and you are allowing it to discover things by just kind of... I'm assuming they do do, like, some form of farts rewards. But then again, they show that it... That it's not actually able to explore meaningfully, so maybe that's the problem. [00:56:48] S3: So verifiable rewards are things like like, does your code compile? Right? So they're they're always so they're always fairly, like, what we would consider sparse to be in robotics. So they don't solve the explode. They don't do anything on the exploration side, which I think is the important thing to remember. Whereas, like, in most RL robotics context, your reward function does help exploration, or you have a a prior or something, like, you're doing a motion policy prior that'll, like, guide exploration. So it's a it's [00:57:18] S2: I don't know. I don't know much [00:57:19] S3: about the language side either, so we'll see.
00:57:21 · Michael Cho
[00:57:21] S0: Well, I guess that the other real challenge in the real world is that if you try to really let the robot try to explore in the real world, it's gonna run into trouble, and it can't respond itself, know, unlike in a simulation. Right? So, you know, there's this recent paper also from this guy,, I think he's gonna come on to report soon Yeah. About Ariel, you know, of... Or policy Ariel. So, you know, I do think maybe, like he said, we also need some advancement, especially on, like, the horizon Yeah. For for off off policy RL. So that maybe some of these, we we don't have to wait until we have a very, very good... Because if the chicken and egg is is not trivial in my view, maybe by the time you have it so good, you actually have soft soft it. Yeah. Which is maybe what the the RM guys are finding out. Oh, the bottle is so good. Actually, you don't need RL. But, I mean, robot is very, very long way from that in my view because it's it's it's so hard to get that. I mean, if the robot fail, like, you know, you need a human to go and fix it. Right? So the way, for example, I look at Tesla, to me, it... You know, it's got a one to... One human to one robot ratio. Right? It still have a one one to one ratio. Even now with the robotaxi, you look at it, it still have a dude sitting in the car, not at the driver's seat, but, yeah, so there's still a human there. And I suspect maybe for... Especially things that that... That's gonna break in the real world, that it's just... Not just safety. Right? But it's just, like, whether it's pragmatic to reset. And, yeah, it's... It might be very, very hard to do to get that traditional RL thing going in the real world. And, of course, you're gonna do all sorts of this experiment in the simulation, but the question is always like, people would dub you, okay, can you... Can this really, really do seem to real? Then you get back to, oh, okay. You still have to test in the real world. Yeah.