The People's AI: The Decentralized AI Podcast
00:20:50 · Jeff Wilser, Bayley Wang
[00:20:50] S0: I heard about one concrete task where users are submitting videos of themselves like folding sheets. You know? Can you talk a little bit? Yeah. Yeah. Is that an accurate example? What are what are things people are doing? And then you kinda connect the dots of how that kind of video they're filming can eventually feed into robotics in action. [00:21:13] S1: Yeah. So the sheet the sheet folding the sheet folding project, we're doing it we're doing it because domestic robot... Robotics is a field which is just very interesting in the robotics industry right now. There are a lot of companies that are trying to tackle this. It aligns well with, you know, hospitality use cases, hotel use cases where there's like a pretty clean GTM for the robotics. And it's something which everyone has. Like, everyone has towels or sheets they can fold. So it's very accessible for the... [00:21:44] S0: It's also low stakes. Right? If the robot Yeah. Yeah. Screws up the sheet folding, no one's going to die. Right? Unlike self driving cars where if they make a mistake, oops, pedestrian is killed. So [00:21:55] S1: Yeah. Exactly. Yeah. Exactly. [00:21:56] S0: It's a great low hanging fruit. I agree.
00:21:58 · Bayley Wang
[00:21:58] S1: Yeah. So then then looking at how this exactly works, the idea is that if you had, say, ten thousand hours of videos folding sheets... Of people folding sheets, you get, a lot of diversity to be able to address like all kinds of environments. But at the same time, the task is repetitive enough that it's easy for the AI model to pick up on how to do it. Right? It's it's similar to like if if you if you watch a hundred hours of videos of someone doing the same task over and over again, you're... It's much easier to learn that task than if you went and watched a hundred hours of random tasks, you would learn basically nothing. And the exact way this works is not to get, like, too deeply technical is the model will go sort of watch those videos and learn the repetitive patterns in those videos without ever having them labeled. Like, it doesn't need to really know what a sheet is or what an arm is or what a hand is. It just sees the same things happening in the videos over and over again and builds a strong internal representation of those patterns. Those patterns, the the the the latents they call them are, like, very compact. They they contain a lot of information, and the model has learned to extract that information. That then gets decoded by, like, a a... They call it in body specific specific model, in order to directly output things like the gripper positions or the joint forces needed to actually complete the task. So that way you can take your huge dataset, train a big model once to generate these, like, representations. Then for each of the robots you want to ship against, you would train a much smaller model on a much smaller amount of robot specific data in order to be able to in order to be able to do the, like, robot specific action decoding.
00:23:42 · Jeff Wilser, Bayley Wang
[00:23:42] S0: So minor follow-up on that. This sounds very maybe, like, silly question, but does the angle matter? I was imagining if most people when folding sheets... If they're... I imagine if I'm if I'm videoing myself holding sheets, maybe I like... If I hold it with my hand, I I do it one way or if I put it across the room their way. But if all of the videos the robots are seeing are filmed a certain way, are there blind spots? So as you have to get, like, do you encourage users? Say, we we are... What's really useful is videos from reverse angle and or bot below angle. And I know it's a kind of a a bizarre question, but I think it gets at some of the complications and nuance here of the the complications of the real world. Right? So like if if, you know, how much variety and richness do you need in different kinda like angles of folding sheets, for example? I think everything in the real world probably has similar types of of complexity of of how do you get enough, like, 360 degree views to get the robot a good sense of, okay, here's what's actually going on. [00:24:52] S1: I think sheet folding is a somewhat, I I would say, not not unique, but like in the minority of examples where the sheet is likely to obscure a large part of the [00:25:03] S0: Yeah.
00:25:03 · Bayley Wang
[00:25:03] S1: View for a while. So for this particular example, I think video from the front and back would definitely be helpful. But in general, the robot... The the... If the robot sees enough video, it should gain an understanding of how the world works in order to be able to sort of guess the obscured views from the, visible views. For for, like, the specific sheet folding example where we're collecting data to address a single task, you probably do need multiple views in order to be able to do the synthesis because you don't have a lot of that data in, like, of other tasks to be able to learn what the hidden parts look like. But if you can imagine if you had enough data, would have view like when the robot would learn when its arm was being obscured, what it was doing. Like, it would learn to guess from like videos where the... There... Its arms weren't obscured. So I think in general for a foundation scale modeling, you don't really need an explicit, like, every task needs to be recorded from every angle. And this sort of like jumps into another another interesting question of like, where does the camera need to be? Yeah. And a lot of people like, oh, yeah. You need egocentric video. You you need like specialized sensors. I I I don't think so. I think if you go set a camera on on a desk and then like do things, that contains a lot of information already. And sure, it doesn't match the point of view of the robot's cameras. But that's fine because the model should be able to learn how to extract the actions anyway from that point of view. Right? Like, I once again, like doing the biomimetic analogy, you don't need to watch a first person view video of someone completing a task to learn how to do it. You're smart enough to deduce where your arms go from like a video from any angle. And what's important is that the like the like view is relatively unobscured and that's like reasonably high resolution. Resolution.
00:26:54 · Jeff Wilser, Bayley Wang
[00:26:54] S0: Makes sense. And... Okay. So we've we've covered sheet folding. What are some other concrete examples of of use cases that that people are feeding data through? [00:27:03] S1: I think probably the most interesting one in the short term is going to be warehousing and restocking. It has really good alignment with what, the types of communities that engage with decentralized projects are are doing anywhere. Like, there's a lot of people who are like gig workers or like do... Working like convenience stores or retail. And it also has a very immediate use case that there are plenty of companies that are either trying to do last mile light warehousing or restocking or, like, some sort of, like, light mix of domestic commercial. Ease... Easy data to collect. You could wear a camera, like, on your chest as you go about doing your doing your work and just passively collect that collect that data. [00:27:42] S0: Tell us a little more about what this looks like as far as incentives and compensation and where are we in the timeline? To what extent is this already out there in the wild? And give us a sense of kinda what the adoption has been like so far.
00:27:42 · Jeff Wilser, Bayley Wang
[00:27:42] S0: Tell us a little more about what this looks like as far as incentives and compensation and where are we in the timeline? To what extent is this already out there in the wild? And give us a sense of kinda what the adoption has been like so far. [00:27:57] S1: Yeah. So for our project specifically, we've really been working more on the more on the demand side with robotics companies. And for the supply side, we're going... I I I think initially we're going to work with, you know you know, like there there's folks like folks like Vana that already have data enthusiast communities. There is also guilds guilds like YGG which were focused on gate... More gaming type things but want to move into something which has more real world value. And these people are already used to the structure of like I... Incent... Incentive hunting either for tokens or for points that convert to tokens in the future. And I think for for, for the data collection stuff, it's it's really no different. The interesting thing I would say is for robot teleoperation. Like, part of our network and our protocol also covers directly being able to connect to a robot and operate the robot to complete tasks. [00:28:51] S0: Is that remote for... I'm trying to visualize this remotely or literally they go to a place and they they put their hands