00:55:04 · Ademi Adeniji
[00:55:04] S2: I think... Can I chip in here? Like, I I have, like, maybe the other side of Chris, what you said. I think reward shaping is maybe one way that you can coerce certain behaviors, but I think is maybe subject to the same ideology that Vincent is describing. Of course, this is one study. Think generally, you need a collection of works for us to be convinced of a direction. Think my take is actually that we should go the opposite direction and use less supervision in this regime. I think there's so much interesting work in the unsupervised RL community and the general exploration that shows that you can learn surprising behaviors by doing things like trying to optimize for model uncertainty or trying to optimize for going to areas where you have high disagreement about what the future holds. And so I think these sorts of algorithmic approaches that are less constrained are the ones that maybe are going to lead us to the capabilities that surprise us and do things that maybe humans wouldn't do, or maybe humans can't specify concisely with a reward function. But again, you do need a really good prior because these methods are like, not only are they RL methods, but they're also like without sort of a direction in some sense. So the exploration problem is is very high dimensional and inefficient.
00:56:28 · Vincent Liu, Chris Paxton, Ademi Adeniji
[00:56:28] S1: Wait. So, Demi, what do think would be missing from, like, the limits of ROVR work, right, where it is... You know, you're able to explore efficiently because you're in a simulator effectively, and you are allowing it to discover things by just kind of... I'm assuming they do do, like, some form of farts rewards. But then again, they show that it... That it's not actually able to explore meaningfully, so maybe that's the problem. [00:56:48] S3: So verifiable rewards are things like like, does your code compile? Right? So they're they're always so they're always fairly, like, what we would consider sparse to be in robotics. So they don't solve the explode. They don't do anything on the exploration side, which I think is the important thing to remember. Whereas, like, in most RL robotics context, your reward function does help exploration, or you have a a prior or something, like, you're doing a motion policy prior that'll, like, guide exploration. So it's a it's [00:57:18] S2: I don't know. I don't know much [00:57:19] S3: about the language side either, so we'll see.
00:57:21 · Michael Cho
[00:57:21] S0: Well, I guess that the other real challenge in the real world is that if you try to really let the robot try to explore in the real world, it's gonna run into trouble, and it can't respond itself, know, unlike in a simulation. Right? So, you know, there's this recent paper also from this guy,, I think he's gonna come on to report soon Yeah. About Ariel, you know, of... Or policy Ariel. So, you know, I do think maybe, like he said, we also need some advancement, especially on, like, the horizon Yeah. For for off off policy RL. So that maybe some of these, we we don't have to wait until we have a very, very good... Because if the chicken and egg is is not trivial in my view, maybe by the time you have it so good, you actually have soft soft it. Yeah. Which is maybe what the the RM guys are finding out. Oh, the bottle is so good. Actually, you don't need RL. But, I mean, robot is very, very long way from that in my view because it's it's it's so hard to get that. I mean, if the robot fail, like, you know, you need a human to go and fix it. Right? So the way, for example, I look at Tesla, to me, it... You know, it's got a one to... One human to one robot ratio. Right? It still have a one one to one ratio. Even now with the robotaxi, you look at it, it still have a dude sitting in the car, not at the driver's seat, but, yeah, so there's still a human there. And I suspect maybe for... Especially things that that... That's gonna break in the real world, that it's just... Not just safety. Right? But it's just, like, whether it's pragmatic to reset. And, yeah, it's... It might be very, very hard to do to get that traditional RL thing going in the real world. And, of course, you're gonna do all sorts of this experiment in the simulation, but the question is always like, people would dub you, okay, can you... Can this really, really do seem to real? Then you get back to, oh, okay. You still have to test in the real world. Yeah.
00:59:16 · Ademi Adeniji, Michael Cho
[00:59:16] S2: Maybe. Yeah. I don't know. I like to use the humans as like, to ground how I think about these things. Like... And so maybe, like, let's not think about unsupervised. Like, let's just take traditional supervised RL. You have a reward function, something you're trying to do. Like, let's say you're trying to, like, be really good at basketball or, like, maybe you're trying to perfect a free kick in soccer. Like like, there's there's maybe a few people that are very, very good at free kicks, like that can score every time or like close to every time. Right? But everyone knows how to pick the ball out of the back of the goal or, you know, walk, walk around, you know, like you may say you missed the goal. You can go walk, put it back on the, on the spot, try again. Right? So that reset that reset behavior is much, I would say, simpler in some sense or much more easy to learn than the very fine grained sort of, like, motor skill of, like, hitting the ball with the perfect angle, the perfect amount of force to get the curve right. So I think, like [01:00:10] S0: I see. [01:00:10] S2: Really what we're looking at is you wanna build a set of primitives that you can bootstrap on so that you can efficiently, like, reset and efficiently, like, be somewhat close to the the more complex behavior. And I think that's pretty I think that's pretty within scope. I think it's just how do you get those primitives and those inductive biases to begin with.
01:00:10 · Ademi Adeniji, Vincent Liu, Michael Cho
[01:00:10] S2: Really what we're looking at is you wanna build a set of primitives that you can bootstrap on so that you can efficiently, like, reset and efficiently, like, be somewhat close to the the more complex behavior. And I think that's pretty I think that's pretty within scope. I think it's just how do you get those primitives and those inductive biases to begin with. [01:00:28] S1: But you'd you'd also want, like, a, like, reasonably good policy for... In the freak example, right, to to like start off of. And so I guess you're still at the point you still go back to the fact that like, okay, if you can imitation learning your way or however however they do it these days, your way to like something good, then you sort of start from there. And then that that itself is maybe a chicken and egg as well. [01:00:49] S0: Yeah. Well, very fascinating stuff. Okay. Cool. I I think, you know, it's... Maybe we should do do a wrap up. Is there anything, like, you know, final one or two points that you guys wanna say? Like, you know, what is some of the biggest lesson that you think you guys have learned? I think you shared a couple, but if you can summarize for us. And, also, maybe if you... This is a common question we like to ask you. If you, let's say, have... You guys have another three months to work on this paper, like, what what would be something that you want to have done, like, to squeeze it into this paper?
01:01:21 · Vincent Liu
[01:01:21] S1: Oh, should I go first? Oh, yeah. Sure. Okay. And then maybe you could add stuff after I'm... After I wrap up. So I guess I'll start with the second question, which is what will we do in three more months? We had actually planned to sort of explore, like we mentioned, like like a VLA style training. But I guess the idea being, like, given what we've seen about the trade offs of using three d points, is there a way we can extract the benefits of that into, like, something like a VLM or like an LLM, which has a very strong reasoning prior, and maybe get like some generalist behavior. I think that's very interesting. And I think we've already seen a lot of work come out in the last month or so, pursuing this direction. In terms of like, you know, the one to two points, you know, I'll probably say something a little oversimplified, but I think it's sort of shaped by my sort of research in other domains more than robotics and also some time at Tesla, which is I feel like, you know, you basically for deep learning systems to really work in the real world, they've only really worked when the test set is basically the same domain as the training set. And so I kind of have this extreme perspective that you should start with that sort of hypothesis and then ask yourself, how can I basically collect data from every single, like, inference time scenario that I care about? And then you run massive self supervised learning or behavior cloning at scale, which is kind of what Tesla does. And in robotics, you have this morphology gap issue if you're doing this from the human perspective, but I feel like, you know, those algorithm... Algorithmic things will sort of fall away with time. And so I don't know. I've kind of maybe a little pessimistic view on on research where if you just do this at scale, at the the entire global scale, you will get very far to the point where maybe you even solve it.