00:31:11 · Sachin, Anastasis Germanidis
[00:31:11] S1: need to solve. What's the most compelling use case you've seen your world models used for so far? [00:31:19] S2: The the most interesting use case that we didn't also expect to work that well in the beginning and ended up working kind of beyond our expectations was this idea of as a as a simulator for robotics. So the idea of being able to to test how your robotics model performs inside the world model and getting results that match what you would get in reality if you tested the robot on the same exact task. So we did a lot of work in establishing that correlation, that kind of simulation to reality correlation, where if a robot succeeds or fails inside the simulation, it succeeds or fails in the real world and there is good correlation between the two. Because once you get that correlation, then you can then evaluate your robotics models way faster. You can basically evaluate them, like, thousands or millions of scenarios before you even test them in the real world. [00:32:14] S1: Do you think there'll be application layer companies that sit on top of the world models, or do you think that people just use world models directly for these kinds of tasks?
00:32:14 · Sachin, Anastasis Germanidis, Adam Miller
[00:32:14] S1: Do you think there'll be application layer companies that sit on top of the world models, or do you think that people just use world models directly for these kinds of tasks? [00:32:22] S2: I think it's it's very likely there's gonna be a whole ecosystem around world models similar to what we've seen with, you know, regular image and video models, similar to what we've seen with language models. So there's gonna be more infrastructure companies, more, you know, aggregators. There's gonna be kind of harnesses on top of those models that help kind of combine combine them in different ways, so I think that's definitely going to be the case. [00:32:46] S0: And when you're trying to make simulations in world models, do you think these models, are they starting to understand the principles of physics or are they mimicking the physics of video that it has been trained on and seen before?
00:32:46 · Adam Miller, Anastasis Germanidis, Sachin
[00:32:46] S0: And when you're trying to make simulations in world models, do you think these models, are they starting to understand the principles of physics or are they mimicking the physics of video that it has been trained on and seen before? [00:33:02] S2: I think that there is a you know it's a blurry distinction between understanding physics if you're ever to mimic physics well enough then that to me is an indication that there... That the model has built really powerful representations of the world that let it actually succeed at simulating those physical phenomena. It's something we test a lot. So we have a series of benchmarks where, we're... With every new generation of model that we build, we test how well does it do in solid mechanics or fluid dynamics on on, like, optics. And there is a lot of real world scenarios that you can, you know, test cases that you can be able to understand their performance. And we see that with every... As we increase the scale of pretraining, as we improve post training, it's predictably becomes better. The models predictably fail less in terms of, you know, having objects disappearing in the middle of the video or, like, things like that. So it's an area where I think I I don't see any reason why that improvement is not gonna keep happening. [00:34:03] S1: Do you think there's a risk that you go away from, like, the core creativity that's at the harder runway if there's applications in, all these hard real world problems?
00:34:03 · Sachin, Anastasis Germanidis, Adam Miller
[00:34:03] S1: Do you think there's a risk that you go away from, like, the core creativity that's at the harder runway if there's applications in, all these hard real world problems? [00:34:15] S2: I think there's a way to apply the creative lens on every problem that you solve. Think the storytelling and film is kind of at the core of the company and I think that's never that's not gonna go away. I think there is definitely a lot more that we can do to expand that further and we're working on that. But I think the I think ultimately the mission of simulating the world is a mission that's very close to kind of what the company was founded on and I think that's you know and I think people are quite excited to kind of help to expand like what the use cases and the domains that the models can address. [00:35:07] S0: So one of the visions for World Models is being able to simulate physics of potentially different types of machineries like robotics. So what are the actual limitations of traditional physics simulators that we have now?
00:35:07 · Adam Miller, Anastasis Germanidis
[00:35:07] S0: So one of the visions for World Models is being able to simulate physics of potentially different types of machineries like robotics. So what are the actual limitations of traditional physics simulators that we have now? [00:35:20] S2: So traditional simulators are very effective to teach, to train robots for certain things. So for example, for locomotion for moving around in space, they're not as effective for manipulation, which is performing different tasks like, know, like picking up objects or washing the dishes or doing laundry. And the reason is it's very it's very difficult to simulate object interactions effectively for things like deformable, like if you want to fold clothes, effectively simulating that is can be very, very tricky and and difficult. And it also requires for every new task that you want to simulate, need to basically handcraft the physics rules to basically make that work well. Whereas with a video model you can leverage all the knowledge it already has that he has seen, you know, millions of videos of people performing different tasks and it can leverage that knowledge in order to, you know, get like a good simulation from just the starting kind of, initial image. It doesn't require to build a whole three d environment of the scene. It doesn't require specifying the physics of different objects in the scene. It can just get kind of be able to simulate that from a starting image.
00:35:20 · Anastasis Germanidis, Sachin
[00:35:20] S2: So traditional simulators are very effective to teach, to train robots for certain things. So for example, for locomotion for moving around in space, they're not as effective for manipulation, which is performing different tasks like, know, like picking up objects or washing the dishes or doing laundry. And the reason is it's very it's very difficult to simulate object interactions effectively for things like deformable, like if you want to fold clothes, effectively simulating that is can be very, very tricky and and difficult. And it also requires for every new task that you want to simulate, need to basically handcraft the physics rules to basically make that work well. Whereas with a video model you can leverage all the knowledge it already has that he has seen, you know, millions of videos of people performing different tasks and it can leverage that knowledge in order to, you know, get like a good simulation from just the starting kind of, initial image. It doesn't require to build a whole three d environment of the scene. It doesn't require specifying the physics of different objects in the scene. It can just get kind of be able to simulate that from a starting image. [00:36:39] S1: Where does filmmaking now sit? When I think of Runway and you walk around the office, you see all these wonderful films on the walls and, you know, it feels very much like at the ethos of the company with the filmmaking festival you launched. Where does now with this kind of expanded directions filmmaking sit?