00:06:10 · Maya, Priya
[00:06:10] S0: But but... Keep going. [00:06:11] S1: Yeah. And then after that, it's simulated data. So there's a lot... That's a whole category of its own startups that are working on being able to simulate data. NVIDIA Cosmos is like one of the big platforms that does that. And it's basically just synthetic data. The reason that people use it is because you can actually get it at scale. It's super cheap. Like the first two that we discussed are super expensive and difficult to get. And so at scale, it's just like the best thing that you can get. And then the last one is egocentric data. So that's what, like, all of these startups that have been in the news recently are focused on, and that's when you have a camera either strapped on your head or like metaclasses or something like that to record what's going on. But then like Maya was saying, you don't get the force... Was it forced work? Yeah. You don't get forced work. You don't have depth perception. You don't have all access to all of that. And so it's not ideal data. It's like the worst quality. But when you have millions of people in India with cameras on them, it's scalable.
00:07:02 · Maya, Anika, Priya
[00:07:02] S0: Yeah. And then there's the final thing I would actually say is third person view where it's just like a video of like people moving through a room. And so this is just... People will like like people... Data labelers will label like what the light is, what a rug is, what a cup is, and then someone moving a cup from left to right. And this is mainly for like planning or understanding how like things move in an environment. And that's like, what creates like a world model. And you need like a world model in order to like have a humanoid work because you need to understand like how things move in a three d space rather than just like texts. But yeah. Yeah. [00:07:34] S2: Yeah. It's funny. One of my friends has a robotics data labeling company. And some of his workers in India were whattapping him, they were like, oh, like, we've realized it's, like, more sturdy and a better angle if we just set it up with a tripod. [00:07:47] S1: And he was like, no. No. No. [00:07:50] S0: The phone of it has to be egocentric. [00:07:52] S1: Yeah. Yeah. That's so funny. And I wanted to talk a [00:07:56] S2: little And then it turned out it was just because they were getting headaches from the camera. I know. That sucks. It's really sad. [00:08:02] S1: Wait. How many hours do they have it on a day? [00:08:05] S2: The whole time. He said, like, he can see them going to the bathroom going to the bathroom doing drugs, like watching, like That's crazy. Inappropriate things on their phone. [00:08:14] S1: And do they clean it at all or they send it all to the labs?
00:08:14 · Priya, Anika, Maya
[00:08:14] S1: And do they clean it at all or they send it all to the labs? [00:08:17] S2: They don't send it to the labs. Like Okay. Like, he does a really low level stuff where, like, he has another job and he's just, like, doing this for cash on the side. So, like, he doesn't do anything, but he sends it to, like, other companies. Like, what... For example, like, y c companies that have... Like, basically, there's, like, a whole chain, right, with the labs at the top. Yeah. He's at the bottom, and then there's, like, companies in the middle there's companies in the middle that are, like, they do some of the labeling, the cleaning [00:08:48] S1: Yeah. [00:08:48] S2: The... And then they're kinda like going up in the chain. Right? Like, they're more like a Mercor. Like, I don't know if Mercor wholesale buys video footage, but I'm sure they do. Because it's like, the people that have contracts with, like, labs, physical intelligence, like, all these big companies, they actually usually will have like really high demand, more high demand than what they can produce with their own operations. [00:09:09] S0: Yeah. Yeah. Like, the... It it makes sense to split out the actual collection to the labeling, to the cleaning, to the processing. [00:09:16] S2: Right. Like everyone wants to do as much of it as they can. But realistically, not every company can do all of Yeah.
00:09:16 · Anika, Maya, Priya
[00:09:16] S2: Right. Like everyone wants to do as much of it as they can. But realistically, not every company can do all of Yeah. [00:09:22] S0: Makes sense. And I wanted to talk about how this data all comes together into a model. Right now, what we see at least Figure do is they have like a dual model architecture system where one model is used for planning and one model is used for action. So the planning model basically looks around and it's like, okay, I wanna move that cup from left to right. And then the action model actually decides what the actual action is, meaning I'm gonna open up my left hand, close it, and then move it to the right. Woah. [00:09:47] S1: Sorry, I meant filming a day in the life, so I thought I would get you an action. [00:09:50] S0: Oh, okay. Well, now I'm awkward. [00:09:53] S1: Oh my god. Okay, ignore me.
00:09:56 · Maya
[00:09:56] S0: There's like one extra camera on me made me so camera shy. But yeah, so this is like one way to deploy a humanoid model or make a humanoid model. Another way is to build what they call a end to end policy learning model, which means it's just like one model that understands the entire environment. And then also decides what the action is going to be. The harder thing about making this model is that there's just not enough data to feed into making a model like this. Meaning there's not enough ecocentric data. When you have a dual model system, there's enough like third person view data. For example, like all the movies you watch, can... Or all the YouTube videos out there, that's all third person data. And that can be used to train like a planning model, which is really, really big. And just like understands contextually, like what the environment is around them. But then the smaller model, which is like the action model, you can basically make it extremely fast, low latency and specifically based on like POV, forced work, all of this like extra data that we talked about. It's just harder to now make these two models basically cohesive and work together at scale. And so then if you make it all into one model, then you have other challenges. And so we're seeing like humanoid companies in this space go after both. And I think sometimes the choices that they make is just like the lack of data. And so like what I'm really curious about is if you have all the data that you wanted in the world, what would be the better choice to go
00:11:16 · Priya, Maya
[00:11:16] S1: after? Yeah. Like what is actually the best architecture for like the end result that we're trying to get to? Yeah. So who's doing the second approach where it's like one combined model right now? [00:11:24] S0: Physical intelligence. Okay. Yeah. So their first version that they released is the combined approach. Is it [00:11:29] S1: like pie zero? Yeah. That's what it's called? [00:11:31] S0: Yeah. Pie zero. Nice. It's not... These things are not easy. Yeah. And the reason why it's not easy is because like LLMs are really good at generalizing. They're really good at taking the average. But when you think about an action, like if you wanna move a cup from left to right, there's like literally twenty hundred, maybe thousands of different ways to move... [00:11:49] S1: Think about like the average of how it would get there. It's like so funny. [00:11:53] S0: There's no there is no average. Right? There's it's actually there's like one... There's actually like an action and either works or it doesn't. And there's like a 100 actions at work and there's like a thousand actions that don't. Right? And so you need to actually produce like your output instead of being one general answer or one average answer, it should be like a 100 answers. And then you pick the one that works. And so like at the actual architecture layer, what this new model is doing is literally pushing out like a 100 options and then they're testing which option is the best one and then going and doing that option.