00:47:25 · Ademi Adeniji
[00:47:25] S2: Possibly. Yeah. Yeah. Possibly. I think I think the general concept of an ambient artificial intelligence is really exciting. Also there's a lot to think about there too. If you're operating within the world, there's maybe privacy asterisks around all of these things. Yeah. I think as far as incentives go, I think maybe the way to start is to think about what is the natural end of the AI system you would build on top of this sort of data. So in this case, we're moving towards generalist manipulation. That's sort of like the, the North Star. Right? And so, you know, what are maybe... Who are stakeholders or people that would benefit from this sort of data and these sorts of models? Right? And then they in turn would be incentivized to collect the data because the outcome or the output of whatever is being built is sort of like, is benefiting them already. Right. Like I think, and then also there, you're worrying less about privacy because there's like, I guess a common understanding between the technology user and the technology developer about what are like sort of the parameters of the use of the data. Right. Yeah, well, I think the ambient AI direction is maybe something really cool for the future. I think there's some practical maybe roadblocks there in terms of privacy and other things.
00:48:51 · Vincent Liu, Michael Cho
[00:48:51] S1: Yeah. I'm gonna add to that, like a cool a cool direction would be, you know, if if home robots, home humanoids were like even somewhat capable, it's like the cell could be that all you do is put your glasses on. And if you do stuff in your house, your robot will end up learning them from you as well. So it's like a very nice way to sort of transfer, like, human intelligence through, like, this ambient intelligence into, an embodied intelligence. [00:49:15] S0: Yeah. I mean, I I'm definitely... Maybe maybe it's a minority if I saw so I have devices like this. Right? Like, you know, basically, just record what I'm talking and then just transcribe. So I I I mean, I... Like I said, I got a bunch of ray... Metal ray bands. So I I... But I believe I'm the 1%, maybe. My wife thinks it's very creepy, so I don't use it anywhere near her. Right? So and she definitely don't allow me to, like, you know, film when my kids around, you know, privacy. So so even at home. Right? So I think I... Yeah. There's all these social constraints for sure. I do have a question, like, jump... Jumping back to the earlier chart about teleop. Right? Do you think there's a version of the world where... Let's say, okay. Something like this. I take your point. Okay. Maybe this is gonna scale way better than something like DexUMI. How about tally op data, though? Like, do you think there would still be a role for, like, tally op data with with real robots to kinda do maybe the last mile, maybe co training.
00:50:13 · Ademi Adeniji
[00:50:13] S2: Yeah. I I guess I can... I have a view on this, and then maybe, Vincent, you also have a view. So I guess my perspective is ultimately in order to get, I think superhuman embodied AI and really the capabilities that I think we all, we'd all dream about, like, and we see in sci fi, I really think you want autonomous real world learning. I think sort of learning, I think for me in terms of the research I've done is exploration is a key problem in AI. And a lot of our reinforcement learning algorithms are built around the assumption that our exploration algorithms are somewhat efficient and we can get data that's surprising or there's high bits of information to learn from. I think once we can get into the regime where we have some sort of prior or inductive bias around how to operate in a very basic way in the world, I think we then want to be looking towards autonomous learning in order to push the manifold of manipulation and hopefully have robots begin to even exceed our own capabilities. So I guess to answer your question directly, I think I'm less sort of... The big question in my mind is not okay. Are we going to be teleoping robots once they're deployed? But how can we get into the regime where robots can basically just operate with their own intelligence and be curious about things like humans are and begin to autonomously develop sensory motor capabilities. I think that's sort of the end game here that I'm most excited about.
00:51:52 · Vincent Liu
[00:51:52] S1: Yeah. I I generally agree. I think, like, there's... I agree with that sort of breakdown of there being, like, this bottleneck, which is chicken and egg problem, which is how can robots efficiently collect and explore on their own... Or collect data and explore on their own. And I think I actually had read one of Chris's Substack posts, and I kind of agree with this, which is like, it's not necessarily is it human data or sim data or teleop data. Like, there's probably efficiency gains that can be... That, like, each of these possesses. And I guess for for our perspective, it's like, how far can you get with just human data? And, you know, because that sort of has some nice properties here. And so for the second part, you know, after you're able to get, like, this very good base model, I like largely... I like largely agree with the idea that okay, then your robots are autonomous enough to go like explore and learn on their own. And I don't know. I'm a I'm a... I kinda like that idea, but recently, I read, like, this limits of RLVR paper, which I've been sort of thinking about. And this is sort of like a language model paper where, you know, you have this idea of a base model that is very powerful, and you then sort of put it in a RO training on tasks where you can verify, like, success or not. So it's like coding or, like, math perhaps. And what they showed is that when you start with a strong base model, which is is kind of like a requirement for efficient exploration, and then you sort of, like, train it with reinforcement learning in, like, some simulator, they showed that RO basically doesn't produce any new, like, superhuman behaviors in a way. And they sort of demonstrate this by plotting the... They basically sent... They increased the PASAT k of these base models. So, basically, you know, if you just let the base model retry, like, a 100 times, it ends up performing in many cases better than the RL trained model. And so, you know, I don't I don't know, like, what... If we're never... We're not even close to, like, that regime in robotics, I suppose. But in my head, I'm like, okay. Maybe practically, there's again some algorithmic things that would prevent you from just this idea of once you're past the chicken and the egg problem that you can just do RL and you become superhuman.
00:54:05 · Vincent Liu, Chris Paxton
It's like, maybe that does exist, but it's like that... The point at which you do that, you might already be hitting, like, five significant digits of, like, nines, right, in your success rate. [00:54:16] S3: I do think that was a super interesting paper, but at least in the robotics space, I feel like like in language, they always have... ROVR is always, like, discrete success terms, right, which we never use in robotics for RL. Right? Like, it's like like, you can always, like, penalize energy or speed or something. Like, it should always... I I... I'm more optimistic about robotics RL, I think, than that. But [00:54:39] S1: I mean, what [00:54:39] S3: is say it doesn't have its limits either. [00:54:42] S1: Right. But I mean, like, if you... I mean, if I if I imagine, like... I'm relatively new to the robotics field. Right? But if I were to imagine, you know, if you had RL in real, like, learning, you'd probably be also just assigning zeros and ones to whether it completed the task or not. Right? And in a way you wouldn't expect it to explore in a very, like, in a very... In a way that, like, diverges from its base behavior.
00:55:04 · Ademi Adeniji
[00:55:04] S2: I think... Can I chip in here? Like, I I have, like, maybe the other side of Chris, what you said. I think reward shaping is maybe one way that you can coerce certain behaviors, but I think is maybe subject to the same ideology that Vincent is describing. Of course, this is one study. Think generally, you need a collection of works for us to be convinced of a direction. Think my take is actually that we should go the opposite direction and use less supervision in this regime. I think there's so much interesting work in the unsupervised RL community and the general exploration that shows that you can learn surprising behaviors by doing things like trying to optimize for model uncertainty or trying to optimize for going to areas where you have high disagreement about what the future holds. And so I think these sorts of algorithmic approaches that are less constrained are the ones that maybe are going to lead us to the capabilities that surprise us and do things that maybe humans wouldn't do, or maybe humans can't specify concisely with a reward function. But again, you do need a really good prior because these methods are like, not only are they RL methods, but they're also like without sort of a direction in some sense. So the exploration problem is is very high dimensional and inefficient.