00:34:27 · Jonathan Stephens
[00:34:27] S1: benchmarks. It's like taking a year end final test in school, but then we're saying, okay. Now we're gonna make you do the full SATs or, you know, taking them like immortalize bb. It's had... It's been like you're trying to get into your master's program. You gotta take a much more difficult... And let's see how well you truly can score on that. I I guess I didn't even realize the scale of how many scenes are in in in RoboFinals as well. So then... So it... Could academia use RoboFinals? Is there a there a use for it there, or is that more really just for, like, large companies? So
00:35:06 · Steve Xie
[00:35:06] S0: I I totally see that academia can use it. And actually, we are providing it as two fold. One, we are providing it as enterprise version to the... So to the top robotics companies, players, and also Frontier model model partners there, you know. And the many of them are already adopting RoboFinals. The other thing there is we want to, provide the infrastructure, I will say for free to them. I mean, we are we are actually selling the benchmark, but we are providing the infrastructure for free because we want to, we want everyone to adopt this standard. If we can have the same standard of use, like, the robotics evaluation infrastructure, then, you know, at least we are speaking the same language. Right? And now, like, let's let's let's talk talk talk about the the more challenging tests. Right? So so we should be focusing on monetizing the test, but we should be, I will say, helping to... Helping the whole industry of Demia to adopt this infrastructure, which we open source. Right? We open source together with NVIDIA, which is called asset lab arena, which is basically the next generation, large scale robotics evaluation benchmark framework. Right? And also, on top of it, we open source, you know, LW-BenchHub, which is... You can see that it's the execution layer of, you know, the the benchmark Mhmm. On top of... As a labyrinth. Right? We want to help everyone to adopt it. So... And also, in... Out of a bunch of hub, we also provided quite a few open source benchmarks to academia, to industry. You know, they can they can run on. You know, there are many things there and many tasks there, which is extremely high quality, SimReady Assets. Right? So it's just a start. This year, you know, we will provide more to this whole industry and also academia that everyone can run on.
00:37:06 · Jonathan Stephens
[00:37:06] S1: So I'll make sure I will I will link the, you know, those... At IsaacLab Arena and, the the, sorry, the LW-BenchHub as well in here because I don't think people are necessarily aware that both of those even exist at this point. Maybe maybe it's IsaacLab-Arena, but I think people get confused because they hear IsaacLab-Arena thinking, oh, I already know IsaacLab. Well, that's... It's different. Or there's IsaacArena in the past. It's kind of a new thing. Yeah. I see that. I like the the fact that we're all working on the same boat, like bench... Benchmark vocabulary, if that's the right term, but saying, okay. If if you understand the benchmark and I've been using the benchmark, there's no... Well, how do I know yours is better than mine if you use a different benchmark system? So it kind of evens the field of where you where you land. So within... Let's say I am a large company. Maybe I I don't know. Maybe I'm Google's labs who've been building the frontier model VLAs and they are using, we'll say they want to use RoboFinals with their benchmarks. So they just keep them internally or is that someone who's just saying, I want to benchmark against what we deployed three months ago or are are they trying to create like a leaderboard so other companies can compete against them? How does that work?
00:38:20 · Steve Xie, Jonathan Stephens
[00:38:20] S0: Benchmarks are still kinda new to me. Yeah. Totally. So basically, we are also providing a leaderboard to the whole industry. So, basically, we are working on it right now. So now we have the benchmark. We should be able to help to... Everyone to benchmark where they are. Right? Mhmm. And also to showcase a leaderboard. So it's definitely also coming. It's definitely also coming. Yeah. Okay. So just not a... [00:38:45] S1: Yeah. And that makes sense. But as people adopt it and get more use, then we'll have a good leaderboard because I think that's something people like to know. Like, I have another I have a new frontier model. Where does it land amongst others? There's something beyond that one benchmark to use one or the other, but it it it even it's a playing field of where everyone lines up.
00:38:45 · Jonathan Stephens, Steve Xie
[00:38:45] S1: Yeah. And that makes sense. But as people adopt it and get more use, then we'll have a good leaderboard because I think that's something people like to know. Like, I have another I have a new frontier model. Where does it land amongst others? There's something beyond that one benchmark to use one or the other, but it it it even it's a playing field of where everyone lines up. [00:39:07] S0: Totally. Right. Yeah. It it is extremely important to let everyone understand where they are, not only to help them benchmark against themselves, say, maybe a day ago, a month ago, but also help them to, benchmark against, you know, the industry, especially the top players in the world. Right? So that's, in my opinion, the most important thing. And, you know, it's it's a lot of work. We are working on, I would say, scaling the benchmark because, basically, we are working on a lot... We are working with a lot of industrial partners. Right? And, also, we are working, we are getting a lot of data in order to build the most realistic, diverse, physically accurate, scalable benchmark in the world, but also at the same time, leveraging that benchmark to provide a little bit to help everyone understand where they are.
00:39:07 · Steve Xie, Jonathan Stephens
[00:39:07] S0: Totally. Right. Yeah. It it is extremely important to let everyone understand where they are, not only to help them benchmark against themselves, say, maybe a day ago, a month ago, but also help them to, benchmark against, you know, the industry, especially the top players in the world. Right? So that's, in my opinion, the most important thing. And, you know, it's it's a lot of work. We are working on, I would say, scaling the benchmark because, basically, we are working on a lot... We are working with a lot of industrial partners. Right? And, also, we are working, we are getting a lot of data in order to build the most realistic, diverse, physically accurate, scalable benchmark in the world, but also at the same time, leveraging that benchmark to provide a little bit to help everyone understand where they are. [00:39:59] S1: Yeah. Makes sense. Okay. So... Okay. So we have the the kind of the three different portions of infrastructure. We have a world, we have behavior of the evaluation. You kind of need all three. You can't you can't run simulation without doing all three. Right? You need to build a world, do some actions, train a model and evaluate how well it does. So you know where to improve it, or to create, get more assets, create more assets, try new behaviors. It's just almost like a flywheel or, you know, an ecosystem there. One thing that I've been diving into is world models and how how... Where does that fit in that? Is that something that we're gonna be using for multiple parts of that ecosystem or is that more for just world generation?