00:18:18 · Steve Xie
[00:18:18] S0: So I would say, visual fidelity is also quite important. The major, yeah. Yeah. Yeah. The major reason there is, you know, we are... So, so, basically, though the VR and VAs, you know, they are getting much more, visually data trained, you know, they still need to, you know, look at exactly, the kind of, fridge, to find on the handle and other, places, in order to manipulate. But but here's the thing. Right? The visual quality is not that difficult to build. Mhmm. Right? Because, you know, there... There's al... Already the gaming industry. There's already the the self driving industry. So it is it is... In my opinion, it is a solved problem. It is a solved problem. The difficulty there is more in, I would say, the optimization of efficiency. Right? So you need to understand about the best trade off between the visual quality versus, you know, the the efficiency of, you know, the the the simulation, the data. Mhmm. So that is the most important thing. Right? If we put, you know, a extremely high quality visual there, for example, a movie level visual, it would look extremely realistic. But the problem there is that it would take too much compute, and it wouldn't be super efficient to run at large parallelizable jobs there. And so from our perspective, you know, we are actually closing a loop in that for every new asset we introduce, we actually close the whole loop of SIM to real, but also real to SIM, meaning that, you know, we would actually leverage, we would actually use the data generated from the simulation to train the algorithm and deploy the algorithm in SIM and also in real of the real robotic arm and manipulate about the SIEM object to test, you know, the efficiency of the data and also the quality of the data. By this, we actually understand, you know, where is the, the best combination that we we we should have good visual, but no need to have the best visual there.
00:20:43 · Jonathan Stephens
[00:20:43] S1: Yeah. And so I I see you... I see your background at Cruise almost informing this, where it's You had a you you built simulation, we got to figure out how useful it is. Same here, we're creating this SimReady Assets in Sim and then you gotta you have to then test it in Sim and then back in world, got to go in and out of simulation to make sure every time you do that transfer, you're still getting the same result. Right? So you're not, you're not thinking that you have a good asset, that you actually have to prove you have a good asset before you can release it to to the world to use. Okay, so then getting back then, okay, so we've we've got the physics engine, we had the SimReady Assets where which is very important. Like, you don't have those things, you really can't run simulation, right? We all we all as you've been discussing the infrastructure layer that we're building here at Lightwheel, that's all encompassed in just one part that is, that is the world, right? So we have it also an environment, so we have to put them in an environment, a room, somewhere, right? And so then that's just one part. And then the next part is behavior, as you've described it, as part of an infrastructure. Can you break down what that behavior component is?
00:21:59 · Steve Xie, Jonathan Stephens
[00:21:59] S0: Yeah. Totally. Yeah. Thank you so much, Jonathan, for bringing up this word behavior and also evolve, framework that we use. Right? So I would say the... Yeah. Exactly. The word is basically how we leverage simulation, right, to build the physically accurate and also diverse words and also the similar AI size. Right? The behavior is, where we use the word to automatically generate data. And, the data should be the robotics trajectories. So for example, robotics, right, manipulating the fridge, the robotics opening up, you know, the oven, the robotics inserting cables, things like that, which is also extremely important. So behavior is actually where we generate the data. Right? And, also, we provide this data, to our customers in order for them to train their foundation models. Mhmm. [00:22:59] S1: So you need you need physics accurate data at scale, and so we're using simulation so you can, as you said, you are automating a lot of this collection, right? To go put a robot out in a data center and have it pull a bunch of cables and reinsert them. You have to either go to a different data center to get a variation or you have to physically move these several ton server racks, put different ones in. That's not efficient. So that so that's the... That's that's where this lays? Is this for efficiency that we're doing this all in simulation and not just putting out robots in real life?
00:23:39 · Steve Xie
[00:23:39] S0: Yeah. Totally. So we actually... We have two ways of generating behavior data, which both are very important. So, basically, there is this data pyramid concept introduced by Ego Drew from UT Austin, and, also, he is, you know, the co lead of GEAR from NVIDIA. So you you could point out this, data pyramid is, you know, that, the real teleoperation data, the the real robotics data is very limited. Right? So it's just the tip of this iceberg, which is a very small part. Maybe 1% or maybe even less than 1% of the data needed to train robotics. And the major data needed to train the robotic is, one, simulation data. Second, egocentric data, so human videos. And the Lightwheel, we really see the behavior as those two things there. One, simulation data. Two, egocentric, real human egocentric videos. And we are biting on those two things together. Right? So for the simulation data, we leverage, you know, the simulation words. Right? We leverage our auto data generation pipeline to generate synthetic data from simulation. Right? And this data is extremely scalable. It's extremely fast. Right? And also, it can be in different robotics forms, right, to provide it to our customers, which is extremely important data source. Right? The second data source is basically, we we have colliders worldwide, you know, by... Wearing, you know, the egocentric devices, for example, the glasses, cameras, and also the wrist cameras of, doing different type of jobs. You know? Like, a lot of time, they are doing jobs. Like, they are doing, you know, the house cleaning work, the housework, related work, chores, in their homes. Right? Others are doing the different type of works in factories, in grocery stores, in coffee shops, things like that. Right? And those data are extremely scalable, diverse too. And also, the other good part is is also from the real world. Right? And we provide those two scalable sources of data to our customers. We we check... We we think are extremely important.
00:25:59 · Jonathan Stephens
[00:25:59] S1: I I think so, especially now, not just to have SimTel ops, because I think if I feel like at least one paper or more a week I'm seeing of researchers finding that if they mix in a large amount of egocentric data and a little bit of real robot data, they're getting fairly good zero shot results. They're getting robots to do things unseen and they're not quite sure the mix, they know that egocentric data and what worse, what you're building is high quality. Right? So, you know, some of these teams are just scraping into, then they get VFX, which is not physics. Correct. Examples are getting cartoons. All these things we got to figure out are either filter out or just, taste the data. What we, what we provide, what Lightwheel provides is this high quality annotated physics plausible because it's it's captured in the right way. We even have, you know, the the the arm state data if you want. So you get this high quality data, then you can use this... The robot tele ops data, so they can get much further down the line. So I think I'm seeing, even know he built a robot in seven months and deployed it. So right now, it's all about how fast you can find the right mix. So us serving that... All that different ways for them to test, I feel like it's critical now. You can't just rely on one type of data.
00:27:19 · Steve Xie, Jonathan Stephens
[00:27:19] S0: I totally agree with you. I totally agree with you. Yeah. The first half of last year, we delivered tens of thousands of data. But the second half of last year, we delivered probably close to 1,000,000 hours of data. And for this year, like, for just, like, this few months, we delivered more than probably 2,000,000 hours data already. Right? So you can see the scale increases exponentially. Right? So I'm really seeing this year probably will deliver more than ten million hours data in total. [00:27:56] S1: I'm seeing more and more projects ingesting that, you know, they say, oh, we have forty four thousand hours of egocentric capture because that's all they could find. [00:28:04] S0: Right? Exactly. [00:28:06] S1: It's not that they don't want more. So if you go read their papers of this research, it'll say in the limitations, a limitation is we only had so much data that we could collect or we could find, or that was high quality. So the thirst for it, because it's, there's like infinite edge cases. We have an infinite complex world you need, You need as much data as possible to have a well informed under, you know, underlying model to do these tasks that we're asking them to do out in the real world. And so that's that's that's what we're doing. For exact structure, scale thing. Right? But not just scale. You're trying to hit quality. Yes. [00:28:42] S0: Because more deeply... Yeah. I I would say the quality and scale are the two most critical and probably the only two most most critical requirements that we need to hit. Yeah. Mhmm.