00:25:59 · Jonathan Stephens
[00:25:59] S1: I I think so, especially now, not just to have SimTel ops, because I think if I feel like at least one paper or more a week I'm seeing of researchers finding that if they mix in a large amount of egocentric data and a little bit of real robot data, they're getting fairly good zero shot results. They're getting robots to do things unseen and they're not quite sure the mix, they know that egocentric data and what worse, what you're building is high quality. Right? So, you know, some of these teams are just scraping into, then they get VFX, which is not physics. Correct. Examples are getting cartoons. All these things we got to figure out are either filter out or just, taste the data. What we, what we provide, what Lightwheel provides is this high quality annotated physics plausible because it's it's captured in the right way. We even have, you know, the the the arm state data if you want. So you get this high quality data, then you can use this... The robot tele ops data, so they can get much further down the line. So I think I'm seeing, even know he built a robot in seven months and deployed it. So right now, it's all about how fast you can find the right mix. So us serving that... All that different ways for them to test, I feel like it's critical now. You can't just rely on one type of data.
00:27:19 · Steve Xie, Jonathan Stephens
[00:27:19] S0: I totally agree with you. I totally agree with you. Yeah. The first half of last year, we delivered tens of thousands of data. But the second half of last year, we delivered probably close to 1,000,000 hours of data. And for this year, like, for just, like, this few months, we delivered more than probably 2,000,000 hours data already. Right? So you can see the scale increases exponentially. Right? So I'm really seeing this year probably will deliver more than ten million hours data in total. [00:27:56] S1: I'm seeing more and more projects ingesting that, you know, they say, oh, we have forty four thousand hours of egocentric capture because that's all they could find. [00:28:04] S0: Right? Exactly. [00:28:06] S1: It's not that they don't want more. So if you go read their papers of this research, it'll say in the limitations, a limitation is we only had so much data that we could collect or we could find, or that was high quality. So the thirst for it, because it's, there's like infinite edge cases. We have an infinite complex world you need, You need as much data as possible to have a well informed under, you know, underlying model to do these tasks that we're asking them to do out in the real world. And so that's that's that's what we're doing. For exact structure, scale thing. Right? But not just scale. You're trying to hit quality. Yes. [00:28:42] S0: Because more deeply... Yeah. I I would say the quality and scale are the two most critical and probably the only two most most critical requirements that we need to hit. Yeah. Mhmm.
00:28:42 · Steve Xie, Jonathan Stephens
[00:28:42] S0: Because more deeply... Yeah. I I would say the quality and scale are the two most critical and probably the only two most most critical requirements that we need to hit. Yeah. Mhmm. [00:28:56] S1: Yeah. No. That makes sense. Okay. So then... So we have that we have... We went with the environment, the world, that's that's just your assets. And then we have... We do behaviors and that could be Ego centric, could be a SimTel ops. And then we need test that these models that we're training work. So, tell me about eval. So that's kind of like the, as I, at the start of this video, I did an intro of talking about the concept of those three. Can you break down then eval and why that is critical? And at this point, I don't see a lot of people. I've seen more people talk about it, but not enough people talking about it.
00:29:32 · Steve Xie
[00:29:32] S0: Totally. So I would say, you know, now we have the world. Now we have the behavior, which is basically the data we generate. Right? We are providing, you know, scalable exponential data growth to our customers. Right? And I would say now the bigger problem the industry faces is where we are. Right? Now we have the enormous data. We can train our models. But can we really understand how well our model is? That's the major problem. So that's the question that Evolve is trying to answer, right, to help the industry, to help the customers, to help our partners to understand where their robotics are. Right? What are their capabilities? What are their boundaries? Things like that. Right? So, you know, one of the typical problem of evolve for the industry, you know, is there are so many great open source evolves provided by academia. You know? There are behavior challenge. There is RoboCasa, and there is. They are amazing. You know? All the papers refer them, and they are actually the, like like, I will say the backbone of this commit... This academia for robotics. Because, you know, you have to take a test from Liberia. You have to take a test from RoboCasa. You have to take test from behavior challenge in order to showcase that your research is something providing insight to this academia. But the problem there is they are good for academia. They are not good for industry. The industry, we are running a much larger cluster of compute. Right? Mhmm. Many of our customers have tens of thousands of GPUs, to train their models. Right? The industry is getting much more data than academia. For example, academia probably have thousands, if at most ten thousand hours data. That... That's it. But for industry, you know, they are they are getting hundreds of thousands, if not million hour of data already. Right? Mhmm. And and the may... Many of them are from us. Then the question is, okay, where can I find the eval? Where can I find the benchmark that can help us to understand the robotics capability at this scale? Right? And for the academia benchmarks, you know, the the frontier robotics foundation models, they can take them on pretty well. You know? They are already re... Reaching, say, 99%, a 100% of post test. Right? So I was looking at a leaderboard
00:32:10 · Jonathan Stephens, Steve Xie
[00:32:10] S1: recently, and it all high nineties. I was like, they're all great. Leaderboard, I don't know which one's truly better. [00:32:18] S0: Exactly. Exactly. They're all similar, right? Now the problem is, okay, can you provide me a really difficult test that can run on large scale? Right? Mhmm. So both the difficulty and also the scalability, they are the two parts which is super important. And I would say that, the real, Evolve cannot provide the scale. Right? Like like like like, it it is not possible to provide 10,000 homes, right, to deploy the robots in 10,000 homes running EVOLVE, maybe, say, 10 times every day. It is not possible to do that. Right? [00:32:56] S1: Right.
00:32:56 · Jonathan Stephens, Steve Xie
[00:32:56] S1: Right. [00:32:56] S0: So we have to do it in simulation. So that's actually the place where we start the RoboFinals because many of our customers, you know, they see we are extremely good with SimReady Assets, Sins, Tasks, and also open source benchmarks. They're like, hey. Can you provide us a cold source benchmark to help us understand where we are... Where we are? But it needs to be very, very accurate. It needs to be very, very high quality. It needs to be diverse. It needs to be scalable. Right? So then, basically, we are working with the industry. We are working with the manufacturing partners. We are working with agriculture fact... Partner. We are working with, you know, the hospital partners. We are working with many, and also we are working in homes. We are collecting all those real things. We are collecting the real tasks because you need to actually help the robotics understand where they are with, and also benchmark them against the real world. So we get the real scene. We get the real task, the real SOPs. Right? And, also, we we we measure the real physics through our scalable physical measurement factory. Right? And then we put all of them into high quality SimReady Assets, scenes, tasks, benchmarks. Together, we provide RoboFinals, which is basically the the industrial grade robotics benchmarks for the frontier models. So it's like everyone has been taking, like, with the with the academic