00:08:22 · Amit Jain
And when that came out, then we were like, okay, maybe this is not so impossible, right? I actually remember one of, like, people on our team in in early twenty four, we were talking about is that, no, I don't think the video will ever be generated by... Because, you know, it just seems so impossible from from a systems architecture perspective. It seemed like a pie in the sky problem, but the architecture changed our thinking, really. Then we started to build good tokenizers and like, know, Xiaming, who's our chief scientist today, him, Matthew Tancik, is by the way the author of NERF. Xiaming is the author of DDN, denoising deficient implicit models, which is sort of like the backbone of like, you know, all of these image and video models today. He joined our team. Xiaming was leading image and video models at NVIDIA at that time. Matt was obviously finished his PhD from Berkeley and as the author of NERF and some really other incredible papers had incredible reputation. So when these people joined our team, we really started to think very seriously about it. Okay. We can actually train video models. And we started training our video model, I think, end of twenty four. And then like, you know, we launched it in March '25.
00:09:29 · Finn Meeks
[00:09:29] S1: You said something there, which is that at the time before maybe they joined the team, it it felt like, you know, being able to generate a video, you know, was was something that was totally non obvious. I think a lot of people feel the same way right now when they think about world models. We're just kinda talking about this, you know, in a roundtable before this, and different folks were kind of sharing what their definition of a world model is. That is maybe still up for debate. Maybe just to start, I'm curious, what is your definition of a world model?
00:09:59 · Amit Jain
[00:09:59] S0: Yeah. Don't think it is my definition of world model. My definition of the world model is the right definition of world model. So currently, nobody actually knows what a world model is. Everybody has random ideas about it, honestly. They are literally random. So some people think, hey, a video model is real time, somehow it is a world model. But what really makes it a world model? Just because it is real time? Not really. Because the video model is autoregressive, it's somehow a world model, not really. Because even autoregressive world models today are not really autoregressive real video models, they're just self forced, they're using a technique called self forcing that allows next frame prediction. But they're not really trained on NTP, which is next token prediction as an objective. Some people think because by training three d models, like Gaussian split models or those kind of things, that somehow makes it a world model, not really. The definition of world model is actually really, really simple. So what is the definition of an image model? A model that understands the distribution of pixel space or that understands the distribution of pixels is that it leads to a natural image and that can generate it. So it understands images and it generates images, that's an image model. A language model is one that understands language really well and generates language. What is a world model? A model that has the understanding of the physical world and is able to generate or better said, simulate that, that's pretty much it. So that's the definition of a world model, a model that understands and can simulate the physical world. Because it's real time doesn't make it anything, right? Like it can be slow as hell. You know, you you can you can generate, like, you know, one frame per hour per per per year. I don't care. Right? If it understands the world, it is actually a world, Marty.
00:11:41 · Finn Meeks, Amit Jain
[00:11:41] S1: Is there then some hurdle for useful world models? [00:11:44] S0: No. If you don't understand the world, you can produce like video as fast as you want. It's gonna be still trash. It's gonna be still a trashy video model. Right? The thing you want out of this is just like a language model has deep understanding of language, not in human terms of understanding, but like, you know, great understanding of the distribution of what makes intelligent language to the point where we can use them for coding today. A world model is one that understands or understands, I'm always going to use a scarecrow around that, Laws of physics, causality, time, and understands the human logic behind it, which is language. So the shape of a world model looks like a single tower that is jointly able to model language, audio, video, images, physical contacts, all of these things, but as one single signal stream. That's what a real world model would look like. That's what a brain is. A brain is able to operate in all of these modalities without having separate towers, without having really different systems and all these kind of things. And it understands the physical world. Do we understand the physical world like in equations? No, not really. I don't think like when you're trying to catch a ball, you're actually computing the kinematics of it. Now you intuitively have have now, like, you know, gathered that. I think that's what a world model is.
00:11:44 · Amit Jain, Finn Meeks
[00:11:44] S0: No. If you don't understand the world, you can produce like video as fast as you want. It's gonna be still trash. It's gonna be still a trashy video model. Right? The thing you want out of this is just like a language model has deep understanding of language, not in human terms of understanding, but like, you know, great understanding of the distribution of what makes intelligent language to the point where we can use them for coding today. A world model is one that understands or understands, I'm always going to use a scarecrow around that, Laws of physics, causality, time, and understands the human logic behind it, which is language. So the shape of a world model looks like a single tower that is jointly able to model language, audio, video, images, physical contacts, all of these things, but as one single signal stream. That's what a real world model would look like. That's what a brain is. A brain is able to operate in all of these modalities without having separate towers, without having really different systems and all these kind of things. And it understands the physical world. Do we understand the physical world like in equations? No, not really. I don't think like when you're trying to catch a ball, you're actually computing the kinematics of it. Now you intuitively have have now, like, you know, gathered that. I think that's what a world model is. [00:12:58] S1: So similar to the early days of your work on video models, do you see kind of the green shoots of potential with world models and your latest work with with UniOne?
00:13:08 · Amit Jain
[00:13:08] S0: Again, like, you know, so this is becoming an annual tradition for us at this point, which is like early in 2025, we started to think about like, where do video models go from here? So we had just released Ray two and Ray three was almost done. And now the question was like, all right, what if this was 10 times bigger? What would happen? That's the question you asked, Andre, right? Like you think in logarithms and if Ray four was 10 times larger in computing parameters, whatever have you, would it be substantially better? Or like a categorically different thing? The answer wasn't an obvious yes. And we started to see, okay, well, what are the problems today? Well, the problems are like, it really has no understanding of causality and events and prompts you give it, prompts people have a very bad impression about, but prompts are instructions. Instructions you give it are, it's just so bad at following them, pretty much all video models even today are. It produces something that is plausible looking, but nobody would call it a simulation. And nothing we could do to change it by just scaling that over 10x. So we started to think, okay, what is actually the limitation here? There was a lot of work that happened at Luma on compression. First we thought maybe the problem is that it's not fast enough and hence we can't make it big enough. So if we made it really, really fast, we can scale it to a large parameter size and it would go. Then we thought about MOEs, all these kind of things. But ultimately, we landed on basically this idea that like, you know what the problem is? It just only understands video and that prevents it from actually doing anything useful. It can produce some clips, but that is about it. It can't do end to end work with it. Like, you can't make a movie with it. Why can't you make a movie with it? It's not because it generates five second clips, it's because it has no idea who a character is. That thing just looks like a blob of pixels to it. So how do you give it an idea? Like, well, we know that from language models that, one, you need to scale them, but two, you need to tie them with human understanding. So we started on this track of unified models, which means, again, one single backbone that has language, which means human interpretation of the universe and the universe. So audio, video, images, all of that. Right?