00:06:15 · Amit Jain
[00:06:15] S0: It wasn't really product bet. It was a research... So Luma is a fundamentally research company. We do research, we build products, and we do deploy them. So the guiding principle for the company is one, multimodal AGI, and two, joint end to end optimization. The only way AI systems will work and be able to do meaningful things in the world is if you do top to bottom optimization. So products are really useful for that in addition to being very good for commercialization and revenue and all these things that come out of it. But those were early days and we were still figuring out. So the story of video is something like this. In '24, when h one hundreds came out, Yeah. H100 had a 96 gig of VRAM on the card, but more importantly, H100s had 800 terabits... Oh, sorry, gigabits per second of interconnect between each card. When we started to see that as in the roadmap, it kinda started to become obvious like, all right, and not obvious actually at that time. It was like, this might be enough bandwidth to move not just text and not just small amount of three d visual data. It might be enough bandwidth to move images and videos. Image models at the time were having a moment with stable diffusion and all these kind of things, but image models candy to train, really. Today, we train eight, 12, 14,000,000,000 parameter image model as a test. That's a unit test for us, actually, at this point. But that H100 architecture was like because... So I don't know if you know much about the history of AI compute. Before A100s, we had V100s, and before V100s, we had K80s. K80s were ridiculous, but at that time, they were the biggest deal in in the late late 2020s. Sorry. Yeah. Late twenty eighteen, something like that. A100 was this Frankenstein of an architecture. It was not actually designed for AI. It was never designed for training, the Ampere architecture. But then NVIDIA basically jettisoned all the rendering cores out of it and trans... Basically maximize the number of like, you know, CUDA cores that you can have on this device or or Tensor cores actually. Tensor and CUDA cores are a little bit overlapping here. And that was like the first training chip, first AI chip, but it was extremely inefficient compared to like today's chips. Right? H100 was the hopper, was the first architecture that was designed for AI, for running training, for running inference.
00:08:22 · Amit Jain
And when that came out, then we were like, okay, maybe this is not so impossible, right? I actually remember one of, like, people on our team in in early twenty four, we were talking about is that, no, I don't think the video will ever be generated by... Because, you know, it just seems so impossible from from a systems architecture perspective. It seemed like a pie in the sky problem, but the architecture changed our thinking, really. Then we started to build good tokenizers and like, know, Xiaming, who's our chief scientist today, him, Matthew Tancik, is by the way the author of NERF. Xiaming is the author of DDN, denoising deficient implicit models, which is sort of like the backbone of like, you know, all of these image and video models today. He joined our team. Xiaming was leading image and video models at NVIDIA at that time. Matt was obviously finished his PhD from Berkeley and as the author of NERF and some really other incredible papers had incredible reputation. So when these people joined our team, we really started to think very seriously about it. Okay. We can actually train video models. And we started training our video model, I think, end of twenty four. And then like, you know, we launched it in March '25.
00:09:29 · Finn Meeks
[00:09:29] S1: You said something there, which is that at the time before maybe they joined the team, it it felt like, you know, being able to generate a video, you know, was was something that was totally non obvious. I think a lot of people feel the same way right now when they think about world models. We're just kinda talking about this, you know, in a roundtable before this, and different folks were kind of sharing what their definition of a world model is. That is maybe still up for debate. Maybe just to start, I'm curious, what is your definition of a world model?
00:09:59 · Amit Jain
[00:09:59] S0: Yeah. Don't think it is my definition of world model. My definition of the world model is the right definition of world model. So currently, nobody actually knows what a world model is. Everybody has random ideas about it, honestly. They are literally random. So some people think, hey, a video model is real time, somehow it is a world model. But what really makes it a world model? Just because it is real time? Not really. Because the video model is autoregressive, it's somehow a world model, not really. Because even autoregressive world models today are not really autoregressive real video models, they're just self forced, they're using a technique called self forcing that allows next frame prediction. But they're not really trained on NTP, which is next token prediction as an objective. Some people think because by training three d models, like Gaussian split models or those kind of things, that somehow makes it a world model, not really. The definition of world model is actually really, really simple. So what is the definition of an image model? A model that understands the distribution of pixel space or that understands the distribution of pixels is that it leads to a natural image and that can generate it. So it understands images and it generates images, that's an image model. A language model is one that understands language really well and generates language. What is a world model? A model that has the understanding of the physical world and is able to generate or better said, simulate that, that's pretty much it. So that's the definition of a world model, a model that understands and can simulate the physical world. Because it's real time doesn't make it anything, right? Like it can be slow as hell. You know, you you can you can generate, like, you know, one frame per hour per per per year. I don't care. Right? If it understands the world, it is actually a world, Marty.
00:11:41 · Finn Meeks, Amit Jain
[00:11:41] S1: Is there then some hurdle for useful world models? [00:11:44] S0: No. If you don't understand the world, you can produce like video as fast as you want. It's gonna be still trash. It's gonna be still a trashy video model. Right? The thing you want out of this is just like a language model has deep understanding of language, not in human terms of understanding, but like, you know, great understanding of the distribution of what makes intelligent language to the point where we can use them for coding today. A world model is one that understands or understands, I'm always going to use a scarecrow around that, Laws of physics, causality, time, and understands the human logic behind it, which is language. So the shape of a world model looks like a single tower that is jointly able to model language, audio, video, images, physical contacts, all of these things, but as one single signal stream. That's what a real world model would look like. That's what a brain is. A brain is able to operate in all of these modalities without having separate towers, without having really different systems and all these kind of things. And it understands the physical world. Do we understand the physical world like in equations? No, not really. I don't think like when you're trying to catch a ball, you're actually computing the kinematics of it. Now you intuitively have have now, like, you know, gathered that. I think that's what a world model is.
00:11:44 · Amit Jain, Finn Meeks
[00:11:44] S0: No. If you don't understand the world, you can produce like video as fast as you want. It's gonna be still trash. It's gonna be still a trashy video model. Right? The thing you want out of this is just like a language model has deep understanding of language, not in human terms of understanding, but like, you know, great understanding of the distribution of what makes intelligent language to the point where we can use them for coding today. A world model is one that understands or understands, I'm always going to use a scarecrow around that, Laws of physics, causality, time, and understands the human logic behind it, which is language. So the shape of a world model looks like a single tower that is jointly able to model language, audio, video, images, physical contacts, all of these things, but as one single signal stream. That's what a real world model would look like. That's what a brain is. A brain is able to operate in all of these modalities without having separate towers, without having really different systems and all these kind of things. And it understands the physical world. Do we understand the physical world like in equations? No, not really. I don't think like when you're trying to catch a ball, you're actually computing the kinematics of it. Now you intuitively have have now, like, you know, gathered that. I think that's what a world model is. [00:12:58] S1: So similar to the early days of your work on video models, do you see kind of the green shoots of potential with world models and your latest work with with UniOne?