00:13:08 · Amit Jain
[00:13:08] S0: Again, like, you know, so this is becoming an annual tradition for us at this point, which is like early in 2025, we started to think about like, where do video models go from here? So we had just released Ray two and Ray three was almost done. And now the question was like, all right, what if this was 10 times bigger? What would happen? That's the question you asked, Andre, right? Like you think in logarithms and if Ray four was 10 times larger in computing parameters, whatever have you, would it be substantially better? Or like a categorically different thing? The answer wasn't an obvious yes. And we started to see, okay, well, what are the problems today? Well, the problems are like, it really has no understanding of causality and events and prompts you give it, prompts people have a very bad impression about, but prompts are instructions. Instructions you give it are, it's just so bad at following them, pretty much all video models even today are. It produces something that is plausible looking, but nobody would call it a simulation. And nothing we could do to change it by just scaling that over 10x. So we started to think, okay, what is actually the limitation here? There was a lot of work that happened at Luma on compression. First we thought maybe the problem is that it's not fast enough and hence we can't make it big enough. So if we made it really, really fast, we can scale it to a large parameter size and it would go. Then we thought about MOEs, all these kind of things. But ultimately, we landed on basically this idea that like, you know what the problem is? It just only understands video and that prevents it from actually doing anything useful. It can produce some clips, but that is about it. It can't do end to end work with it. Like, you can't make a movie with it. Why can't you make a movie with it? It's not because it generates five second clips, it's because it has no idea who a character is. That thing just looks like a blob of pixels to it. So how do you give it an idea? Like, well, we know that from language models that, one, you need to scale them, but two, you need to tie them with human understanding. So we started on this track of unified models, which means, again, one single backbone that has language, which means human interpretation of the universe and the universe. So audio, video, images, all of that. Right?
00:15:16 · Amit Jain, Finn Meeks
Like, you know, video, images, audio are just human measurements of the physical world because, you know, that's how we record them. They're not anything special. They're just signal storage mechanisms. And we sort of think about like, okay, what if, like, you know, there were about 4,000,000,000,000 tokens of both sides in in one single thing, image and models again because they're relatively easier to make, we started with Uni One as an image model, which is like, it's a unified model with language tower, pretty sizable one, and an image tower that is pretty sizable one, all fused into one single thing. It took us a bit of time, but now it is clear as day that this is the path to go forward because it allows image models to do things that could never be done before. I mean, a second example of that is the recent GPT model, which is also a couple of people from Luma who very open AI and then started that effort in December. So I think this is basically the way to think about it. So next video model that we are working on now is like a unified video model. We'll have basically similar characteristics. They're ridiculously hard to train because, you know, language has very different dynamics and then then continuous signals of images and videos. But this is the way to give these models physical understanding and understanding of human interpretation of what that physics actually means. [00:16:31] S1: So were you training on similar types of tokens for your video models? [00:16:35] S0: Sorry. I don't understand.
00:16:35 · Amit Jain, Finn Meeks
[00:16:35] S0: Sorry. I don't understand. [00:16:36] S1: Like, did you have image data, audio data involved in in those as well?
00:16:44 · Amit Jain
[00:16:44] S0: So image data, yes. Audio tower is not there today. Right? So in order to make it a true world model, you you need to have like, you know, a bit more signals. So video is... Video and audio will be the next one basically. And I think once you have video audio language, you have about... You're about 90% of the way there. You have pretty much everything. I mean, some of you who are in robotics here would make the case you don't have grasping information and all these kind of things. Right? And it's true. You don't have a smell. You... That's true. I don't know if robots need smell, a sense of smell. They can just have sensors for some critical things like LPG leak, and those kinds of things, carbon monoxide detection, those kinds of things. And I think it will be sufficient. Maybe not in the fullness of time, but initially. But yeah, you need audio, you need video, and you need language. I think those are really, really important. And then everything else that comes after it is helpful. Because even if you don't have the exact sense of like how much grip strength you should have, you do know from observing videos that like, hey, something like this can be picked up. But if you see like an elephant, oh, let's not pick it up. It doesn't matter how much strength I put it into that. I know my own strength and I kinda understand that. Then you do on policy training in that bold model sense of doing actual pickups, right? Will it crush a lot of cans initially? Yes, but that's okay. You learn very quickly. They're like, oh, this is not the pressure I apply here. This is the pressure... This is how humans learn as well. So human brain generally bootstraps from like all of these kind of things, and then you go forward with by learning the physical world and length of your arm and the strength of your arm and all these kind of things as babies try to grasp. So that's how we think it will go.
00:18:18 · Finn Meeks, Amit Jain
[00:18:18] S1: With this unified model, do you do you think that this eventually replaces language models as well? Yeah, I'm curious, like, is LMs just going to be a thing of the past a few years from now and everything will be structured more as unified models? [00:18:34] S0: It depends on what you mean by that. If you mean that empirically, I think no. If you mean that theoretically, I think yes. So empirically, the considerations are different. When I say empirically, I mean in the field, in deployment, in real usage. In real world usage, you want to strike for efficiency a lot more. So I think LM scaling is now at a point where you just don't see like, oh, okay. So for instance, the recent Qwen 27B model actually is outperforming a Qwen 289B model basically, which is ridiculous at this point. I think we will still continue to use language models quite a bit, but theoretically, a world model which has understanding of language and physics and the world model will strictly be better. It will be more general, it will be able to use in more scenarios, But does that mean you should be using that role model for writing front end code? Maybe not. But if you're making a front end that is really involved and has a lot of visual states and a lot of interactions on it, then yes, you do want to have the world model that can understand the full layout and flow of it and can like, you know, make makes and watch the video of the thing play out and say like, no, no, this is not very good. Like, you know, let me edit on it. So use case.
00:18:34 · Amit Jain, Finn Meeks
[00:18:34] S0: It depends on what you mean by that. If you mean that empirically, I think no. If you mean that theoretically, I think yes. So empirically, the considerations are different. When I say empirically, I mean in the field, in deployment, in real usage. In real world usage, you want to strike for efficiency a lot more. So I think LM scaling is now at a point where you just don't see like, oh, okay. So for instance, the recent Qwen 27B model actually is outperforming a Qwen 289B model basically, which is ridiculous at this point. I think we will still continue to use language models quite a bit, but theoretically, a world model which has understanding of language and physics and the world model will strictly be better. It will be more general, it will be able to use in more scenarios, But does that mean you should be using that role model for writing front end code? Maybe not. But if you're making a front end that is really involved and has a lot of visual states and a lot of interactions on it, then yes, you do want to have the world model that can understand the full layout and flow of it and can like, you know, make makes and watch the video of the thing play out and say like, no, no, this is not very good. Like, you know, let me edit on it. So use case. [00:19:47] S1: So you might see kind of just systems of models, with a unified model guiding guiding them tailored to a specific application.