00:11:41 · Finn Meeks, Amit Jain
[00:11:41] S1: Is there then some hurdle for useful world models? [00:11:44] S0: No. If you don't understand the world, you can produce like video as fast as you want. It's gonna be still trash. It's gonna be still a trashy video model. Right? The thing you want out of this is just like a language model has deep understanding of language, not in human terms of understanding, but like, you know, great understanding of the distribution of what makes intelligent language to the point where we can use them for coding today. A world model is one that understands or understands, I'm always going to use a scarecrow around that, Laws of physics, causality, time, and understands the human logic behind it, which is language. So the shape of a world model looks like a single tower that is jointly able to model language, audio, video, images, physical contacts, all of these things, but as one single signal stream. That's what a real world model would look like. That's what a brain is. A brain is able to operate in all of these modalities without having separate towers, without having really different systems and all these kind of things. And it understands the physical world. Do we understand the physical world like in equations? No, not really. I don't think like when you're trying to catch a ball, you're actually computing the kinematics of it. Now you intuitively have have now, like, you know, gathered that. I think that's what a world model is.
00:11:44 · Amit Jain, Finn Meeks
[00:11:44] S0: No. If you don't understand the world, you can produce like video as fast as you want. It's gonna be still trash. It's gonna be still a trashy video model. Right? The thing you want out of this is just like a language model has deep understanding of language, not in human terms of understanding, but like, you know, great understanding of the distribution of what makes intelligent language to the point where we can use them for coding today. A world model is one that understands or understands, I'm always going to use a scarecrow around that, Laws of physics, causality, time, and understands the human logic behind it, which is language. So the shape of a world model looks like a single tower that is jointly able to model language, audio, video, images, physical contacts, all of these things, but as one single signal stream. That's what a real world model would look like. That's what a brain is. A brain is able to operate in all of these modalities without having separate towers, without having really different systems and all these kind of things. And it understands the physical world. Do we understand the physical world like in equations? No, not really. I don't think like when you're trying to catch a ball, you're actually computing the kinematics of it. Now you intuitively have have now, like, you know, gathered that. I think that's what a world model is. [00:12:58] S1: So similar to the early days of your work on video models, do you see kind of the green shoots of potential with world models and your latest work with with UniOne?
00:13:08 · Amit Jain
[00:13:08] S0: Again, like, you know, so this is becoming an annual tradition for us at this point, which is like early in 2025, we started to think about like, where do video models go from here? So we had just released Ray two and Ray three was almost done. And now the question was like, all right, what if this was 10 times bigger? What would happen? That's the question you asked, Andre, right? Like you think in logarithms and if Ray four was 10 times larger in computing parameters, whatever have you, would it be substantially better? Or like a categorically different thing? The answer wasn't an obvious yes. And we started to see, okay, well, what are the problems today? Well, the problems are like, it really has no understanding of causality and events and prompts you give it, prompts people have a very bad impression about, but prompts are instructions. Instructions you give it are, it's just so bad at following them, pretty much all video models even today are. It produces something that is plausible looking, but nobody would call it a simulation. And nothing we could do to change it by just scaling that over 10x. So we started to think, okay, what is actually the limitation here? There was a lot of work that happened at Luma on compression. First we thought maybe the problem is that it's not fast enough and hence we can't make it big enough. So if we made it really, really fast, we can scale it to a large parameter size and it would go. Then we thought about MOEs, all these kind of things. But ultimately, we landed on basically this idea that like, you know what the problem is? It just only understands video and that prevents it from actually doing anything useful. It can produce some clips, but that is about it. It can't do end to end work with it. Like, you can't make a movie with it. Why can't you make a movie with it? It's not because it generates five second clips, it's because it has no idea who a character is. That thing just looks like a blob of pixels to it. So how do you give it an idea? Like, well, we know that from language models that, one, you need to scale them, but two, you need to tie them with human understanding. So we started on this track of unified models, which means, again, one single backbone that has language, which means human interpretation of the universe and the universe. So audio, video, images, all of that. Right?
00:15:16 · Amit Jain, Finn Meeks
Like, you know, video, images, audio are just human measurements of the physical world because, you know, that's how we record them. They're not anything special. They're just signal storage mechanisms. And we sort of think about like, okay, what if, like, you know, there were about 4,000,000,000,000 tokens of both sides in in one single thing, image and models again because they're relatively easier to make, we started with Uni One as an image model, which is like, it's a unified model with language tower, pretty sizable one, and an image tower that is pretty sizable one, all fused into one single thing. It took us a bit of time, but now it is clear as day that this is the path to go forward because it allows image models to do things that could never be done before. I mean, a second example of that is the recent GPT model, which is also a couple of people from Luma who very open AI and then started that effort in December. So I think this is basically the way to think about it. So next video model that we are working on now is like a unified video model. We'll have basically similar characteristics. They're ridiculously hard to train because, you know, language has very different dynamics and then then continuous signals of images and videos. But this is the way to give these models physical understanding and understanding of human interpretation of what that physics actually means. [00:16:31] S1: So were you training on similar types of tokens for your video models? [00:16:35] S0: Sorry. I don't understand.
00:16:35 · Amit Jain, Finn Meeks
[00:16:35] S0: Sorry. I don't understand. [00:16:36] S1: Like, did you have image data, audio data involved in in those as well?
00:16:44 · Amit Jain
[00:16:44] S0: So image data, yes. Audio tower is not there today. Right? So in order to make it a true world model, you you need to have like, you know, a bit more signals. So video is... Video and audio will be the next one basically. And I think once you have video audio language, you have about... You're about 90% of the way there. You have pretty much everything. I mean, some of you who are in robotics here would make the case you don't have grasping information and all these kind of things. Right? And it's true. You don't have a smell. You... That's true. I don't know if robots need smell, a sense of smell. They can just have sensors for some critical things like LPG leak, and those kinds of things, carbon monoxide detection, those kinds of things. And I think it will be sufficient. Maybe not in the fullness of time, but initially. But yeah, you need audio, you need video, and you need language. I think those are really, really important. And then everything else that comes after it is helpful. Because even if you don't have the exact sense of like how much grip strength you should have, you do know from observing videos that like, hey, something like this can be picked up. But if you see like an elephant, oh, let's not pick it up. It doesn't matter how much strength I put it into that. I know my own strength and I kinda understand that. Then you do on policy training in that bold model sense of doing actual pickups, right? Will it crush a lot of cans initially? Yes, but that's okay. You learn very quickly. They're like, oh, this is not the pressure I apply here. This is the pressure... This is how humans learn as well. So human brain generally bootstraps from like all of these kind of things, and then you go forward with by learning the physical world and length of your arm and the strength of your arm and all these kind of things as babies try to grasp. So that's how we think it will go.