00:29:32 · Steve Xie
[00:29:32] S0: Totally. So I would say, you know, now we have the world. Now we have the behavior, which is basically the data we generate. Right? We are providing, you know, scalable exponential data growth to our customers. Right? And I would say now the bigger problem the industry faces is where we are. Right? Now we have the enormous data. We can train our models. But can we really understand how well our model is? That's the major problem. So that's the question that Evolve is trying to answer, right, to help the industry, to help the customers, to help our partners to understand where their robotics are. Right? What are their capabilities? What are their boundaries? Things like that. Right? So, you know, one of the typical problem of evolve for the industry, you know, is there are so many great open source evolves provided by academia. You know? There are behavior challenge. There is RoboCasa, and there is. They are amazing. You know? All the papers refer them, and they are actually the, like like, I will say the backbone of this commit... This academia for robotics. Because, you know, you have to take a test from Liberia. You have to take a test from RoboCasa. You have to take test from behavior challenge in order to showcase that your research is something providing insight to this academia. But the problem there is they are good for academia. They are not good for industry. The industry, we are running a much larger cluster of compute. Right? Mhmm. Many of our customers have tens of thousands of GPUs, to train their models. Right? The industry is getting much more data than academia. For example, academia probably have thousands, if at most ten thousand hours data. That... That's it. But for industry, you know, they are they are getting hundreds of thousands, if not million hour of data already. Right? Mhmm. And and the may... Many of them are from us. Then the question is, okay, where can I find the eval? Where can I find the benchmark that can help us to understand the robotics capability at this scale? Right? And for the academia benchmarks, you know, the the frontier robotics foundation models, they can take them on pretty well. You know? They are already re... Reaching, say, 99%, a 100% of post test. Right? So I was looking at a leaderboard
00:32:10 · Jonathan Stephens, Steve Xie
[00:32:10] S1: recently, and it all high nineties. I was like, they're all great. Leaderboard, I don't know which one's truly better. [00:32:18] S0: Exactly. Exactly. They're all similar, right? Now the problem is, okay, can you provide me a really difficult test that can run on large scale? Right? Mhmm. So both the difficulty and also the scalability, they are the two parts which is super important. And I would say that, the real, Evolve cannot provide the scale. Right? Like like like like, it it is not possible to provide 10,000 homes, right, to deploy the robots in 10,000 homes running EVOLVE, maybe, say, 10 times every day. It is not possible to do that. Right? [00:32:56] S1: Right.
00:32:56 · Jonathan Stephens, Steve Xie
[00:32:56] S1: Right. [00:32:56] S0: So we have to do it in simulation. So that's actually the place where we start the RoboFinals because many of our customers, you know, they see we are extremely good with SimReady Assets, Sins, Tasks, and also open source benchmarks. They're like, hey. Can you provide us a cold source benchmark to help us understand where we are... Where we are? But it needs to be very, very accurate. It needs to be very, very high quality. It needs to be diverse. It needs to be scalable. Right? So then, basically, we are working with the industry. We are working with the manufacturing partners. We are working with agriculture fact... Partner. We are working with, you know, the hospital partners. We are working with many, and also we are working in homes. We are collecting all those real things. We are collecting the real tasks because you need to actually help the robotics understand where they are with, and also benchmark them against the real world. So we get the real scene. We get the real task, the real SOPs. Right? And, also, we we we measure the real physics through our scalable physical measurement factory. Right? And then we put all of them into high quality SimReady Assets, scenes, tasks, benchmarks. Together, we provide RoboFinals, which is basically the the industrial grade robotics benchmarks for the frontier models. So it's like everyone has been taking, like, with the with the academic
00:34:27 · Jonathan Stephens
[00:34:27] S1: benchmarks. It's like taking a year end final test in school, but then we're saying, okay. Now we're gonna make you do the full SATs or, you know, taking them like immortalize bb. It's had... It's been like you're trying to get into your master's program. You gotta take a much more difficult... And let's see how well you truly can score on that. I I guess I didn't even realize the scale of how many scenes are in in in RoboFinals as well. So then... So it... Could academia use RoboFinals? Is there a there a use for it there, or is that more really just for, like, large companies? So
00:35:06 · Steve Xie
[00:35:06] S0: I I totally see that academia can use it. And actually, we are providing it as two fold. One, we are providing it as enterprise version to the... So to the top robotics companies, players, and also Frontier model model partners there, you know. And the many of them are already adopting RoboFinals. The other thing there is we want to, provide the infrastructure, I will say for free to them. I mean, we are we are actually selling the benchmark, but we are providing the infrastructure for free because we want to, we want everyone to adopt this standard. If we can have the same standard of use, like, the robotics evaluation infrastructure, then, you know, at least we are speaking the same language. Right? And now, like, let's let's let's talk talk talk about the the more challenging tests. Right? So so we should be focusing on monetizing the test, but we should be, I will say, helping to... Helping the whole industry of Demia to adopt this infrastructure, which we open source. Right? We open source together with NVIDIA, which is called asset lab arena, which is basically the next generation, large scale robotics evaluation benchmark framework. Right? And also, on top of it, we open source, you know, LW-BenchHub, which is... You can see that it's the execution layer of, you know, the the benchmark Mhmm. On top of... As a labyrinth. Right? We want to help everyone to adopt it. So... And also, in... Out of a bunch of hub, we also provided quite a few open source benchmarks to academia, to industry. You know, they can they can run on. You know, there are many things there and many tasks there, which is extremely high quality, SimReady Assets. Right? So it's just a start. This year, you know, we will provide more to this whole industry and also academia that everyone can run on.
00:37:06 · Jonathan Stephens
[00:37:06] S1: So I'll make sure I will I will link the, you know, those... At IsaacLab Arena and, the the, sorry, the LW-BenchHub as well in here because I don't think people are necessarily aware that both of those even exist at this point. Maybe maybe it's IsaacLab-Arena, but I think people get confused because they hear IsaacLab-Arena thinking, oh, I already know IsaacLab. Well, that's... It's different. Or there's IsaacArena in the past. It's kind of a new thing. Yeah. I see that. I like the the fact that we're all working on the same boat, like bench... Benchmark vocabulary, if that's the right term, but saying, okay. If if you understand the benchmark and I've been using the benchmark, there's no... Well, how do I know yours is better than mine if you use a different benchmark system? So it kind of evens the field of where you where you land. So within... Let's say I am a large company. Maybe I I don't know. Maybe I'm Google's labs who've been building the frontier model VLAs and they are using, we'll say they want to use RoboFinals with their benchmarks. So they just keep them internally or is that someone who's just saying, I want to benchmark against what we deployed three months ago or are are they trying to create like a leaderboard so other companies can compete against them? How does that work?