The People's AI: The Decentralized AI Podcast
00:03:47 · Bayley Wang, Jeff Wilser
[00:03:47] S1: Yeah. Yeah. So robot robotics is an interesting field, right, Because it combines both difficult hardware challenges and difficult software challenges. And uniquely among any of these fields, robotics... In robotics, the hardware challenges got solved before the software challenges. So between, I would say, about 2010 and 2019, the industry put in a huge flurry flurry of work on actuation and generating the forces need to make the... Needed to make the robots work. So nowadays, you... As you said, you see these crazy YouTube videos of robots doing parkour and front flips, back flips, barrel rolls, but you don't really see the robots being in the wild because they don't really have the intelligence or planning to be able to complete real tasks. [00:04:28] S0: So am I hearing you right? The actual hardware is maybe surprisingly far along. It's a software that's lagging, which surprises me because my thought naively is that it's the physical world in the hardware that makes us all more complicated than Gen AI, which is growing by leaps and bounds in the past, you know, two plus years? [00:04:51] S1: I would say certainly the necessity to own hardware is a barrier to scaling, but the hardware problem itself is solved. Like, you ask me to design you a robot of some shape, there is a set of off the shelf or nearly off the shelf parts you could put together to make it happen.
00:04:51 · Bayley Wang, Jeff Wilser
[00:04:51] S1: I would say certainly the necessity to own hardware is a barrier to scaling, but the hardware problem itself is solved. Like, you ask me to design you a robot of some shape, there is a set of off the shelf or nearly off the shelf parts you could put together to make it happen. [00:05:05] S0: Got it. So what are the main kind of pain points of why we're not seeing more... I mean, I I go to AI conferences and you you often see little prototype robot scurrying around. They look like dogs almost. Do you see things like that in very crude prototypes? They're clearly nowhere near ready for prime time. So what's holding us back?
00:05:29 · Bayley Wang
[00:05:29] S1: Yeah. So I think I think really what's holding us back is automation. Like having these robots be able to go out and do things and complete tasks on their own without any sort of human intervention. And for that, we need better AI that can interact with the physical world. Now, this AI that interacts with the real world is a tricky thing. Right? When you go use ChatGPT, if the answer is wrong, you're like, oh, that's kind of stupid, and you try again. If you go deploy a model like that on a robot, like it's trying to clean your kitchen and you come back and 10% of the dishes have been thrown on the ground, you're going to be very angry and you're definitely like try again is not really what you want to do. You're gonna be very angry I don't know, like never use that robot again. So the accuracy requirements are are harder. The real world is also very diverse. Like even just saying this robot runs in a kitchen, there's a whole lot of different shapes and sizes that the robot has to be able to navigate. And some people are trying to do it with like a rules based approach where they use machine vision to generate like a list of items in the scene, then a VLM to go try to plan the tasks. But you have a lot of, like, rules written by humans in there. Mhmm. And those rules tend to break down when the people who wrote them didn't expect a new type of scenario.
00:06:44 · Jeff Wilser
[00:06:44] S0: Am I am I right that rules based approach, is that analogous to before kind of LLMs and the transformer architecture in Gen AI, the idea was we will label. Right? We'll say we'll say, hey, this is a cat. This is a dog. And so then they realize, okay, the kind of transformer breakthrough, realize you don't have to do that. You can just feed a shitload of text and data, and the transformer will kind of figure it out and make these connections on its own. Is it something similar happening in robotics that that that instead of having all of these kind of rules based logic that there's more elegant way to to get these robots to move their their hands and claws?
00:07:28 · Bayley Wang
[00:07:28] S1: Yeah. So modern rules based approaches are a little more sophisticated than the ones from like the early mid twenty tens because we do have LLMs and BLMs now, and they can pick up a little bit of the fuzziness fuzziness. Right? So if you want a robot to clean a kitchen, you can use the LLM to go plan out a set of actions and then use a more conventional approach to find the objects in the scene that those actions might correspond to and have the robot go like move its arms to the coordinate of the cup and then move it to the dishwasher. But the actual like planning was done by an LLM. So it does account to account for some level of fuzziness. But still fundamentally, this process is a rules based approach. The humans are writing rules and making assumptions about what the environment will look like. And those assumptions tend to break down. It's one of those unfortunate things that even the best teams of developers are nowhere near omniscient, so they can't plan for everything. And that's why the LLMs are so powerful. Right? No one writes the rules for the LLMs. They see a cross section consisting of, like, tens of percent of the entire Internet. And by doing so, they're able to capture a little bit of everything. So while LLMs sometimes hallucinate, they very rarely are horrifically incorrect. Like, they generate semantically and syntactically correct English. They generate plausible outputs. It's sort of like the difference in robotics between the robot picking up the dish and like throwing it on the ground versus the robot say picking up the wrong dish and putting it in the wrong container. Right? Like the the first one is not very good, but the second one you can at least sort of like put up with. Or in language modeling terms, it's the difference between getting the wrong answer versus getting something which is just a string of words that don't even make any sense next to each other.
00:09:14 · Jeff Wilser, Bayley Wang
[00:09:14] S0: I'd love to get a little more into your personal background and kinda what brought you to PrismaX. And I think that might give listeners a good sense of where are the particular pain points in robotics and and how data fits into this. Tell us a little more about your journey from MIT to what you're building now. [00:09:32] S1: Yeah. Yeah. So I was a robotics researcher at MIT, and I was part of one of the teams, the mini Cheetah team, that helped solve this hardware problem by making the actuators a heck of a lot cheaper and easier to get. But when it came time to decide whether we took that robot and we made a commercial spin off or we just, like, let it rest and went went back to doing doing research and doing other things. We made the very active decision to not try to sell the mini cheetah. The MIT robotics team is probably one of the only great academic robotics teams without a commercial spin off. And the reason we did that is because we we couldn't figure out how to how to get the robot to do anything useful. [00:10:11] S0: For those who are not familiar, what was the mini cheetah and talk about kinda what it did and and what it Yeah. Yes. Yeah.