The People's AI: The Decentralized AI Podcast
00:21:58 · Bayley Wang
[00:21:58] S1: Yeah. So then then looking at how this exactly works, the idea is that if you had, say, ten thousand hours of videos folding sheets... Of people folding sheets, you get, a lot of diversity to be able to address like all kinds of environments. But at the same time, the task is repetitive enough that it's easy for the AI model to pick up on how to do it. Right? It's it's similar to like if if you if you watch a hundred hours of videos of someone doing the same task over and over again, you're... It's much easier to learn that task than if you went and watched a hundred hours of random tasks, you would learn basically nothing. And the exact way this works is not to get, like, too deeply technical is the model will go sort of watch those videos and learn the repetitive patterns in those videos without ever having them labeled. Like, it doesn't need to really know what a sheet is or what an arm is or what a hand is. It just sees the same things happening in the videos over and over again and builds a strong internal representation of those patterns. Those patterns, the the the the latents they call them are, like, very compact. They they contain a lot of information, and the model has learned to extract that information. That then gets decoded by, like, a a... They call it in body specific specific model, in order to directly output things like the gripper positions or the joint forces needed to actually complete the task. So that way you can take your huge dataset, train a big model once to generate these, like, representations. Then for each of the robots you want to ship against, you would train a much smaller model on a much smaller amount of robot specific data in order to be able to in order to be able to do the, like, robot specific action decoding.
00:23:42 · Jeff Wilser, Bayley Wang
[00:23:42] S0: So minor follow-up on that. This sounds very maybe, like, silly question, but does the angle matter? I was imagining if most people when folding sheets... If they're... I imagine if I'm if I'm videoing myself holding sheets, maybe I like... If I hold it with my hand, I I do it one way or if I put it across the room their way. But if all of the videos the robots are seeing are filmed a certain way, are there blind spots? So as you have to get, like, do you encourage users? Say, we we are... What's really useful is videos from reverse angle and or bot below angle. And I know it's a kind of a a bizarre question, but I think it gets at some of the complications and nuance here of the the complications of the real world. Right? So like if if, you know, how much variety and richness do you need in different kinda like angles of folding sheets, for example? I think everything in the real world probably has similar types of of complexity of of how do you get enough, like, 360 degree views to get the robot a good sense of, okay, here's what's actually going on. [00:24:52] S1: I think sheet folding is a somewhat, I I would say, not not unique, but like in the minority of examples where the sheet is likely to obscure a large part of the [00:25:03] S0: Yeah.
00:25:03 · Bayley Wang
[00:25:03] S1: View for a while. So for this particular example, I think video from the front and back would definitely be helpful. But in general, the robot... The the... If the robot sees enough video, it should gain an understanding of how the world works in order to be able to sort of guess the obscured views from the, visible views. For for, like, the specific sheet folding example where we're collecting data to address a single task, you probably do need multiple views in order to be able to do the synthesis because you don't have a lot of that data in, like, of other tasks to be able to learn what the hidden parts look like. But if you can imagine if you had enough data, would have view like when the robot would learn when its arm was being obscured, what it was doing. Like, it would learn to guess from like videos where the... There... Its arms weren't obscured. So I think in general for a foundation scale modeling, you don't really need an explicit, like, every task needs to be recorded from every angle. And this sort of like jumps into another another interesting question of like, where does the camera need to be? Yeah. And a lot of people like, oh, yeah. You need egocentric video. You you need like specialized sensors. I I I don't think so. I think if you go set a camera on on a desk and then like do things, that contains a lot of information already. And sure, it doesn't match the point of view of the robot's cameras. But that's fine because the model should be able to learn how to extract the actions anyway from that point of view. Right? Like, I once again, like doing the biomimetic analogy, you don't need to watch a first person view video of someone completing a task to learn how to do it. You're smart enough to deduce where your arms go from like a video from any angle. And what's important is that the like the like view is relatively unobscured and that's like reasonably high resolution. Resolution.
00:26:54 · Jeff Wilser, Bayley Wang
[00:26:54] S0: Makes sense. And... Okay. So we've we've covered sheet folding. What are some other concrete examples of of use cases that that people are feeding data through? [00:27:03] S1: I think probably the most interesting one in the short term is going to be warehousing and restocking. It has really good alignment with what, the types of communities that engage with decentralized projects are are doing anywhere. Like, there's a lot of people who are like gig workers or like do... Working like convenience stores or retail. And it also has a very immediate use case that there are plenty of companies that are either trying to do last mile light warehousing or restocking or, like, some sort of, like, light mix of domestic commercial. Ease... Easy data to collect. You could wear a camera, like, on your chest as you go about doing your doing your work and just passively collect that collect that data. [00:27:42] S0: Tell us a little more about what this looks like as far as incentives and compensation and where are we in the timeline? To what extent is this already out there in the wild? And give us a sense of kinda what the adoption has been like so far.
00:27:42 · Jeff Wilser, Bayley Wang
[00:27:42] S0: Tell us a little more about what this looks like as far as incentives and compensation and where are we in the timeline? To what extent is this already out there in the wild? And give us a sense of kinda what the adoption has been like so far. [00:27:57] S1: Yeah. So for our project specifically, we've really been working more on the more on the demand side with robotics companies. And for the supply side, we're going... I I I think initially we're going to work with, you know you know, like there there's folks like folks like Vana that already have data enthusiast communities. There is also guilds guilds like YGG which were focused on gate... More gaming type things but want to move into something which has more real world value. And these people are already used to the structure of like I... Incent... Incentive hunting either for tokens or for points that convert to tokens in the future. And I think for for, for the data collection stuff, it's it's really no different. The interesting thing I would say is for robot teleoperation. Like, part of our network and our protocol also covers directly being able to connect to a robot and operate the robot to complete tasks. [00:28:51] S0: Is that remote for... I'm trying to visualize this remotely or literally they go to a place and they they put their hands
00:28:57 · Bayley Wang
[00:28:57] S1: on on a robot? Remotely. Remotely. Okay. You you remotely operate the robot through like it could be as simple as a keyboard and a joystick or VR or something. Okay. And that that type of that type of task has a more interesting incentive structure right now because there are customers that are willing to like they're paying $50 an hour for that service right now, but they can't scale. So you you... That's like a very robust business. Right? Like they're they're like not only do they not want to pay $50 an hour, but more to the point they do not know how to scale these robot operators. Like the team is like like you you have companies with a valuation of over $1,000,000,000 running a team of eight people to do their data collection right now. Like one of the people is like a founding engineer. That kind of situation. So being able to offer something which is a lot more scalable is really helpful. And there's a lot of interesting things you can do there. Like, we encourage guild formation because guilds cannot only go and like organize the operators on the ground. They can also do things like invest in a robot that can be operated to generate data and mine incentives. And that's really helpful for the robotics companies as well because that means that they can, like, at least break even on the hardware, like, sell it at cost to a guild for the data collection rather than having to, like, raise dilutive capital for a manufacturing round in order to, like, get to the point where they can even start collecting data.