The People's AI: The Decentralized AI Podcast
00:01:17 · Jeff Wilser
[00:01:17] S0: he worked on many of the early cutting edge robot prototypes. He eventually realized that for robotics to truly scale, we need better data. So we launched PrismaX, which is essentially a platform to connect real world data provided by users with the robotics companies that are hungry for this data. And broadly speaking, PrismaX is partnering with Vana to help bring user data into the fold and create better robotics. PrismaX recently came out of stealth at an a 16 z demo day, and the idea is that you can be rewarded for sending videos of real world actions like folding laundry or even operating actual robots remotely. You log in to your laptop and you virtually control some robot a thousand miles away. As Bayley describes in her conversation, right now, today, there are billion dollar organizations that are begging for people to virtually operate their robots, kind of help them move around and and get insights, get data, and they're having a hard time making this happen. PrismaX is building a system to enable this at scale. Zooming out the lens, for me, robots almost symbolize something magical about the future, something straight from sci fi. I personally think of them as more Jetsons than Terminator, and I love the vision of a future where a robot will clean my bathroom, scramble my eggs, fold my laundry. How far are we from that future, and what will it take to get there? This is what we will explore in today's conversation with Bayley Wang, the cofounder of PrismaX. Bayley, welcome to The People's AI. Thanks for having me. So I am fascinated by robotics. I know many of our listeners are too. So we're gonna get deep into the nuance of what you guys are building and how data fits into this. But let's start at a more macro lens. What is the current state of play of robotics? Right. You occasionally see YouTube videos of cool things happening, but you also don't see any mainstream adoption. How would you describe where we're at at this moment with robotics? What are robots good at right now, and what are they not so good at? What's your high level look at that?
00:03:47 · Bayley Wang, Jeff Wilser
[00:03:47] S1: Yeah. Yeah. So robot robotics is an interesting field, right, Because it combines both difficult hardware challenges and difficult software challenges. And uniquely among any of these fields, robotics... In robotics, the hardware challenges got solved before the software challenges. So between, I would say, about 2010 and 2019, the industry put in a huge flurry flurry of work on actuation and generating the forces need to make the... Needed to make the robots work. So nowadays, you... As you said, you see these crazy YouTube videos of robots doing parkour and front flips, back flips, barrel rolls, but you don't really see the robots being in the wild because they don't really have the intelligence or planning to be able to complete real tasks. [00:04:28] S0: So am I hearing you right? The actual hardware is maybe surprisingly far along. It's a software that's lagging, which surprises me because my thought naively is that it's the physical world in the hardware that makes us all more complicated than Gen AI, which is growing by leaps and bounds in the past, you know, two plus years? [00:04:51] S1: I would say certainly the necessity to own hardware is a barrier to scaling, but the hardware problem itself is solved. Like, you ask me to design you a robot of some shape, there is a set of off the shelf or nearly off the shelf parts you could put together to make it happen.
00:04:51 · Bayley Wang, Jeff Wilser
[00:04:51] S1: I would say certainly the necessity to own hardware is a barrier to scaling, but the hardware problem itself is solved. Like, you ask me to design you a robot of some shape, there is a set of off the shelf or nearly off the shelf parts you could put together to make it happen. [00:05:05] S0: Got it. So what are the main kind of pain points of why we're not seeing more... I mean, I I go to AI conferences and you you often see little prototype robot scurrying around. They look like dogs almost. Do you see things like that in very crude prototypes? They're clearly nowhere near ready for prime time. So what's holding us back?
00:05:29 · Bayley Wang
[00:05:29] S1: Yeah. So I think I think really what's holding us back is automation. Like having these robots be able to go out and do things and complete tasks on their own without any sort of human intervention. And for that, we need better AI that can interact with the physical world. Now, this AI that interacts with the real world is a tricky thing. Right? When you go use ChatGPT, if the answer is wrong, you're like, oh, that's kind of stupid, and you try again. If you go deploy a model like that on a robot, like it's trying to clean your kitchen and you come back and 10% of the dishes have been thrown on the ground, you're going to be very angry and you're definitely like try again is not really what you want to do. You're gonna be very angry I don't know, like never use that robot again. So the accuracy requirements are are harder. The real world is also very diverse. Like even just saying this robot runs in a kitchen, there's a whole lot of different shapes and sizes that the robot has to be able to navigate. And some people are trying to do it with like a rules based approach where they use machine vision to generate like a list of items in the scene, then a VLM to go try to plan the tasks. But you have a lot of, like, rules written by humans in there. Mhmm. And those rules tend to break down when the people who wrote them didn't expect a new type of scenario.
00:06:44 · Jeff Wilser
[00:06:44] S0: Am I am I right that rules based approach, is that analogous to before kind of LLMs and the transformer architecture in Gen AI, the idea was we will label. Right? We'll say we'll say, hey, this is a cat. This is a dog. And so then they realize, okay, the kind of transformer breakthrough, realize you don't have to do that. You can just feed a shitload of text and data, and the transformer will kind of figure it out and make these connections on its own. Is it something similar happening in robotics that that that instead of having all of these kind of rules based logic that there's more elegant way to to get these robots to move their their hands and claws?
00:07:28 · Bayley Wang
[00:07:28] S1: Yeah. So modern rules based approaches are a little more sophisticated than the ones from like the early mid twenty tens because we do have LLMs and BLMs now, and they can pick up a little bit of the fuzziness fuzziness. Right? So if you want a robot to clean a kitchen, you can use the LLM to go plan out a set of actions and then use a more conventional approach to find the objects in the scene that those actions might correspond to and have the robot go like move its arms to the coordinate of the cup and then move it to the dishwasher. But the actual like planning was done by an LLM. So it does account to account for some level of fuzziness. But still fundamentally, this process is a rules based approach. The humans are writing rules and making assumptions about what the environment will look like. And those assumptions tend to break down. It's one of those unfortunate things that even the best teams of developers are nowhere near omniscient, so they can't plan for everything. And that's why the LLMs are so powerful. Right? No one writes the rules for the LLMs. They see a cross section consisting of, like, tens of percent of the entire Internet. And by doing so, they're able to capture a little bit of everything. So while LLMs sometimes hallucinate, they very rarely are horrifically incorrect. Like, they generate semantically and syntactically correct English. They generate plausible outputs. It's sort of like the difference in robotics between the robot picking up the dish and like throwing it on the ground versus the robot say picking up the wrong dish and putting it in the wrong container. Right? Like the the first one is not very good, but the second one you can at least sort of like put up with. Or in language modeling terms, it's the difference between getting the wrong answer versus getting something which is just a string of words that don't even make any sense next to each other.