00:04:20 · Finn Meeks, Amit Jain
[00:04:20] S1: Yeah. And the first the first product that you released was kind of just a three d capture app. Right? Why was that the right place to start? And then obviously you have moved on from that to grander things. We can talk about that too, but I'm curious why that that was the first thing you started with. [00:04:35] S0: In consumer, right, like, you know, you don't think about distribution, you're kind of dead. In ML, if you don't think about data, you're dead. So when we started on the three d path, at the time we believed that, okay, well, that might be sufficient. We didn't think it was possibly sufficient, but we thought it might be sufficient. But then the question is, there is no three d data. Where is three d data? There's no YouTube of three d. There's no Wikipedia of three d. So how do you get that data? Even if you have three d models, but that is not three d world, that is not dynamics, all these kind of things. So the most immediate action was, all right, release something that people would love to use and would be free, and that would give us a large scale of data over which we can bootstrap. We still didn't know what scale of data you really needed. Was it 1,000,000? Was it 1,000,000,000,000? Or somewhere in between? We had no idea. Nobody did because these kind of scaling laws are only now becoming apparent. Even in multimodality, the scaling laws don't exist. So that was the first step and it had commercial value as well. So like, you know, it was it was not a terrible step, but that was the first step.
00:05:41 · Finn Meeks
[00:05:41] S1: Yeah. And then following that, you you know, you worked on a couple of different products, but the one that really I I think put you on the map was was also the release of Dream Machine as you started to kind of move into video. And maybe some of the three d capture was just data collection as well and so this was kind of on your road map from the get go. You know, was was that a pivot? Was that something that you just saw as kind of the next step in a progression towards some future state of the world, world models or or, you know, we can get into that later. Yeah. I'm curious kind of how did that start to emerge from from your research as the next product?
00:06:15 · Amit Jain
[00:06:15] S0: It wasn't really product bet. It was a research... So Luma is a fundamentally research company. We do research, we build products, and we do deploy them. So the guiding principle for the company is one, multimodal AGI, and two, joint end to end optimization. The only way AI systems will work and be able to do meaningful things in the world is if you do top to bottom optimization. So products are really useful for that in addition to being very good for commercialization and revenue and all these things that come out of it. But those were early days and we were still figuring out. So the story of video is something like this. In '24, when h one hundreds came out, Yeah. H100 had a 96 gig of VRAM on the card, but more importantly, H100s had 800 terabits... Oh, sorry, gigabits per second of interconnect between each card. When we started to see that as in the roadmap, it kinda started to become obvious like, all right, and not obvious actually at that time. It was like, this might be enough bandwidth to move not just text and not just small amount of three d visual data. It might be enough bandwidth to move images and videos. Image models at the time were having a moment with stable diffusion and all these kind of things, but image models candy to train, really. Today, we train eight, 12, 14,000,000,000 parameter image model as a test. That's a unit test for us, actually, at this point. But that H100 architecture was like because... So I don't know if you know much about the history of AI compute. Before A100s, we had V100s, and before V100s, we had K80s. K80s were ridiculous, but at that time, they were the biggest deal in in the late late 2020s. Sorry. Yeah. Late twenty eighteen, something like that. A100 was this Frankenstein of an architecture. It was not actually designed for AI. It was never designed for training, the Ampere architecture. But then NVIDIA basically jettisoned all the rendering cores out of it and trans... Basically maximize the number of like, you know, CUDA cores that you can have on this device or or Tensor cores actually. Tensor and CUDA cores are a little bit overlapping here. And that was like the first training chip, first AI chip, but it was extremely inefficient compared to like today's chips. Right? H100 was the hopper, was the first architecture that was designed for AI, for running training, for running inference.
00:08:22 · Amit Jain
And when that came out, then we were like, okay, maybe this is not so impossible, right? I actually remember one of, like, people on our team in in early twenty four, we were talking about is that, no, I don't think the video will ever be generated by... Because, you know, it just seems so impossible from from a systems architecture perspective. It seemed like a pie in the sky problem, but the architecture changed our thinking, really. Then we started to build good tokenizers and like, know, Xiaming, who's our chief scientist today, him, Matthew Tancik, is by the way the author of NERF. Xiaming is the author of DDN, denoising deficient implicit models, which is sort of like the backbone of like, you know, all of these image and video models today. He joined our team. Xiaming was leading image and video models at NVIDIA at that time. Matt was obviously finished his PhD from Berkeley and as the author of NERF and some really other incredible papers had incredible reputation. So when these people joined our team, we really started to think very seriously about it. Okay. We can actually train video models. And we started training our video model, I think, end of twenty four. And then like, you know, we launched it in March '25.
00:09:29 · Finn Meeks
[00:09:29] S1: You said something there, which is that at the time before maybe they joined the team, it it felt like, you know, being able to generate a video, you know, was was something that was totally non obvious. I think a lot of people feel the same way right now when they think about world models. We're just kinda talking about this, you know, in a roundtable before this, and different folks were kind of sharing what their definition of a world model is. That is maybe still up for debate. Maybe just to start, I'm curious, what is your definition of a world model?
00:09:59 · Amit Jain
[00:09:59] S0: Yeah. Don't think it is my definition of world model. My definition of the world model is the right definition of world model. So currently, nobody actually knows what a world model is. Everybody has random ideas about it, honestly. They are literally random. So some people think, hey, a video model is real time, somehow it is a world model. But what really makes it a world model? Just because it is real time? Not really. Because the video model is autoregressive, it's somehow a world model, not really. Because even autoregressive world models today are not really autoregressive real video models, they're just self forced, they're using a technique called self forcing that allows next frame prediction. But they're not really trained on NTP, which is next token prediction as an objective. Some people think because by training three d models, like Gaussian split models or those kind of things, that somehow makes it a world model, not really. The definition of world model is actually really, really simple. So what is the definition of an image model? A model that understands the distribution of pixel space or that understands the distribution of pixels is that it leads to a natural image and that can generate it. So it understands images and it generates images, that's an image model. A language model is one that understands language really well and generates language. What is a world model? A model that has the understanding of the physical world and is able to generate or better said, simulate that, that's pretty much it. So that's the definition of a world model, a model that understands and can simulate the physical world. Because it's real time doesn't make it anything, right? Like it can be slow as hell. You know, you you can you can generate, like, you know, one frame per hour per per per year. I don't care. Right? If it understands the world, it is actually a world, Marty.