00:15:16 · Amit Jain, Finn Meeks
Like, you know, video, images, audio are just human measurements of the physical world because, you know, that's how we record them. They're not anything special. They're just signal storage mechanisms. And we sort of think about like, okay, what if, like, you know, there were about 4,000,000,000,000 tokens of both sides in in one single thing, image and models again because they're relatively easier to make, we started with Uni One as an image model, which is like, it's a unified model with language tower, pretty sizable one, and an image tower that is pretty sizable one, all fused into one single thing. It took us a bit of time, but now it is clear as day that this is the path to go forward because it allows image models to do things that could never be done before. I mean, a second example of that is the recent GPT model, which is also a couple of people from Luma who very open AI and then started that effort in December. So I think this is basically the way to think about it. So next video model that we are working on now is like a unified video model. We'll have basically similar characteristics. They're ridiculously hard to train because, you know, language has very different dynamics and then then continuous signals of images and videos. But this is the way to give these models physical understanding and understanding of human interpretation of what that physics actually means. [00:16:31] S1: So were you training on similar types of tokens for your video models? [00:16:35] S0: Sorry. I don't understand.
00:16:35 · Amit Jain, Finn Meeks
[00:16:35] S0: Sorry. I don't understand. [00:16:36] S1: Like, did you have image data, audio data involved in in those as well?
00:16:44 · Amit Jain
[00:16:44] S0: So image data, yes. Audio tower is not there today. Right? So in order to make it a true world model, you you need to have like, you know, a bit more signals. So video is... Video and audio will be the next one basically. And I think once you have video audio language, you have about... You're about 90% of the way there. You have pretty much everything. I mean, some of you who are in robotics here would make the case you don't have grasping information and all these kind of things. Right? And it's true. You don't have a smell. You... That's true. I don't know if robots need smell, a sense of smell. They can just have sensors for some critical things like LPG leak, and those kinds of things, carbon monoxide detection, those kinds of things. And I think it will be sufficient. Maybe not in the fullness of time, but initially. But yeah, you need audio, you need video, and you need language. I think those are really, really important. And then everything else that comes after it is helpful. Because even if you don't have the exact sense of like how much grip strength you should have, you do know from observing videos that like, hey, something like this can be picked up. But if you see like an elephant, oh, let's not pick it up. It doesn't matter how much strength I put it into that. I know my own strength and I kinda understand that. Then you do on policy training in that bold model sense of doing actual pickups, right? Will it crush a lot of cans initially? Yes, but that's okay. You learn very quickly. They're like, oh, this is not the pressure I apply here. This is the pressure... This is how humans learn as well. So human brain generally bootstraps from like all of these kind of things, and then you go forward with by learning the physical world and length of your arm and the strength of your arm and all these kind of things as babies try to grasp. So that's how we think it will go.
00:18:18 · Finn Meeks, Amit Jain
[00:18:18] S1: With this unified model, do you do you think that this eventually replaces language models as well? Yeah, I'm curious, like, is LMs just going to be a thing of the past a few years from now and everything will be structured more as unified models? [00:18:34] S0: It depends on what you mean by that. If you mean that empirically, I think no. If you mean that theoretically, I think yes. So empirically, the considerations are different. When I say empirically, I mean in the field, in deployment, in real usage. In real world usage, you want to strike for efficiency a lot more. So I think LM scaling is now at a point where you just don't see like, oh, okay. So for instance, the recent Qwen 27B model actually is outperforming a Qwen 289B model basically, which is ridiculous at this point. I think we will still continue to use language models quite a bit, but theoretically, a world model which has understanding of language and physics and the world model will strictly be better. It will be more general, it will be able to use in more scenarios, But does that mean you should be using that role model for writing front end code? Maybe not. But if you're making a front end that is really involved and has a lot of visual states and a lot of interactions on it, then yes, you do want to have the world model that can understand the full layout and flow of it and can like, you know, make makes and watch the video of the thing play out and say like, no, no, this is not very good. Like, you know, let me edit on it. So use case.
00:18:34 · Amit Jain, Finn Meeks
[00:18:34] S0: It depends on what you mean by that. If you mean that empirically, I think no. If you mean that theoretically, I think yes. So empirically, the considerations are different. When I say empirically, I mean in the field, in deployment, in real usage. In real world usage, you want to strike for efficiency a lot more. So I think LM scaling is now at a point where you just don't see like, oh, okay. So for instance, the recent Qwen 27B model actually is outperforming a Qwen 289B model basically, which is ridiculous at this point. I think we will still continue to use language models quite a bit, but theoretically, a world model which has understanding of language and physics and the world model will strictly be better. It will be more general, it will be able to use in more scenarios, But does that mean you should be using that role model for writing front end code? Maybe not. But if you're making a front end that is really involved and has a lot of visual states and a lot of interactions on it, then yes, you do want to have the world model that can understand the full layout and flow of it and can like, you know, make makes and watch the video of the thing play out and say like, no, no, this is not very good. Like, you know, let me edit on it. So use case. [00:19:47] S1: So you might see kind of just systems of models, with a unified model guiding guiding them tailored to a specific application.
00:19:47 · Finn Meeks, Amit Jain
[00:19:47] S1: So you might see kind of just systems of models, with a unified model guiding guiding them tailored to a specific application. [00:19:54] S0: Correct. The unified model will replace LLMs as the core, but you might do a tool call to an LLM and ask it like, you know, hey, can you just code this out because you're more efficient than me? [00:20:03] S1: Tell us a little bit more about Luma Agents. This is a recent release of yours, and I I think it kinda goes back to to what you were saying earlier is that there's a lot of value too in just the the orchestration and the ability to actually, like, solve, you know, that end to end problem for our customer beyond just having a good model. And there's more than just a model that goes into it. Can you tell us a little bit more about Luma Agents and just how you actually think about making these useful for customers?