00:00:00 · Amit Jain, Finn Meeks
[00:00:00] S0: Promise of AI is not doing a little bit of spot work. Alright? Hey, I can make copy. It's like, alright. Good for you, man. Can you actually make the book for me? In a foundation lab, there is no product or research. The secret to building a great company in this space is to just treat them as one. And foundation labs are probably the blueprint of companies of the future. [00:00:21] S1: What is your definition of a world model? Thank you so much for joining us. I'm Finn, I'm one of the partners here at South Park Commons. For those who don't know, South Park Commons is a community of technologists who are exploring new ideas, technologies, companies at the earliest stages. Very pleased to have Amit Jain, the founder and CEO of Luma Labs here today. Amit, we met probably five years ago or so when you joined South Park Commons, you had just left Apple and you were starting to explore neural radiance fields, kind of new area at the time. Over five years, lot has changed. You are now one of the companies leading the way in visual intelligence, last valued close to $4,000,000,000 and doing really really amazing work with some of the new models that you've released. I would like to start by maybe just going back to the beginning. Tell us a little bit more about your initial exploration in nerfs. What got you interested in that area coming out of Apple and how did that kind of end up shaping some of what Luma has become over the years?
00:01:29 · Amit Jain, Finn Meeks
[00:01:29] S0: Thanks for having me and thanks for coming everyone. So, I mean, at Apple, I joined, we were working on the LiDAR sensor and my job was basically the quantum optics for that sensor and figuring out what the physical world looks like from a beam of light. But the second part of my career there was building personas for VisionPro, which if you don't know, they're the three d photorealistic humans you can have a FaceTime call with. I mean, sorry, the human is not artificial, it could be your mom on the other side of the world and you're sitting here, of you are wearing vision problems. Not [00:02:03] S1: an AI companion.
00:02:04 · Amit Jain
[00:02:04] S0: Exactly. Not an AI companion. That's not in Apple's DNA. That was a really, really difficult project because if you think about it from 2018 to 2021 era, right, Meta ships, if you know the Avatars Without Legs, most famously, if anyone of you are keeping track of that in metaverse, we shipped photorealistic humans with real hair and facial expressions and glasses and everything like that, that you capture by looking at VisionPRO and just swiping this way. So very different world. And so in that process, I got to learn a lot about visual understanding, visual intelligence generation as a problem. We started to work on that in 2019. Actually, the work was going on for a while actually before that as well, but I joined that in 2019. And around that time, so 2020 was when Matt published the NERF paper. Actually, it was in '21, but we had got some access to it early on and things like DALL E came out in, if I'm not wrong, Christmas time of 2020 as well. So you were starting to see that, like, okay, language models were working. Image models are of thing because you can make images with transformers now and you can do three d with transformers now. So the thinking was, okay, well, my background is in, just so you might wanna know because otherwise it would not make too much sense. I learned programming when I was about 12 years old, mostly to do physical simulations. So like, you know, my my goal was to become a physicist and learning physics by running simulations is the most common thing physicists do. So being able to do, like, you know, just simple simulations, being able to do kinematic simulations, being able to do, like, simulations for electromagnetism, all these kinds of things. That's what I learned programming about, and I've been doing that for a very long time, even at that time. So the thinking was, hey, if it's working for language, if it's working for image, if it's working for video, well, not at the time, sorry, but if it's working for all these things, what prevents it from working for literally every signal of the physical world? And so we might be able to train a physical simulator, a thing that can model the world, a thing can actually do that. So that was the initial thinking. And NERF looked like the closest thing that was a practical implementation at that time. So we started exploring that. Right? That's the name of it.
00:04:20 · Finn Meeks, Amit Jain
[00:04:20] S1: Yeah. And the first the first product that you released was kind of just a three d capture app. Right? Why was that the right place to start? And then obviously you have moved on from that to grander things. We can talk about that too, but I'm curious why that that was the first thing you started with. [00:04:35] S0: In consumer, right, like, you know, you don't think about distribution, you're kind of dead. In ML, if you don't think about data, you're dead. So when we started on the three d path, at the time we believed that, okay, well, that might be sufficient. We didn't think it was possibly sufficient, but we thought it might be sufficient. But then the question is, there is no three d data. Where is three d data? There's no YouTube of three d. There's no Wikipedia of three d. So how do you get that data? Even if you have three d models, but that is not three d world, that is not dynamics, all these kind of things. So the most immediate action was, all right, release something that people would love to use and would be free, and that would give us a large scale of data over which we can bootstrap. We still didn't know what scale of data you really needed. Was it 1,000,000? Was it 1,000,000,000,000? Or somewhere in between? We had no idea. Nobody did because these kind of scaling laws are only now becoming apparent. Even in multimodality, the scaling laws don't exist. So that was the first step and it had commercial value as well. So like, you know, it was it was not a terrible step, but that was the first step.
00:05:41 · Finn Meeks
[00:05:41] S1: Yeah. And then following that, you you know, you worked on a couple of different products, but the one that really I I think put you on the map was was also the release of Dream Machine as you started to kind of move into video. And maybe some of the three d capture was just data collection as well and so this was kind of on your road map from the get go. You know, was was that a pivot? Was that something that you just saw as kind of the next step in a progression towards some future state of the world, world models or or, you know, we can get into that later. Yeah. I'm curious kind of how did that start to emerge from from your research as the next product?
00:06:15 · Amit Jain
[00:06:15] S0: It wasn't really product bet. It was a research... So Luma is a fundamentally research company. We do research, we build products, and we do deploy them. So the guiding principle for the company is one, multimodal AGI, and two, joint end to end optimization. The only way AI systems will work and be able to do meaningful things in the world is if you do top to bottom optimization. So products are really useful for that in addition to being very good for commercialization and revenue and all these things that come out of it. But those were early days and we were still figuring out. So the story of video is something like this. In '24, when h one hundreds came out, Yeah. H100 had a 96 gig of VRAM on the card, but more importantly, H100s had 800 terabits... Oh, sorry, gigabits per second of interconnect between each card. When we started to see that as in the roadmap, it kinda started to become obvious like, all right, and not obvious actually at that time. It was like, this might be enough bandwidth to move not just text and not just small amount of three d visual data. It might be enough bandwidth to move images and videos. Image models at the time were having a moment with stable diffusion and all these kind of things, but image models candy to train, really. Today, we train eight, 12, 14,000,000,000 parameter image model as a test. That's a unit test for us, actually, at this point. But that H100 architecture was like because... So I don't know if you know much about the history of AI compute. Before A100s, we had V100s, and before V100s, we had K80s. K80s were ridiculous, but at that time, they were the biggest deal in in the late late 2020s. Sorry. Yeah. Late twenty eighteen, something like that. A100 was this Frankenstein of an architecture. It was not actually designed for AI. It was never designed for training, the Ampere architecture. But then NVIDIA basically jettisoned all the rendering cores out of it and trans... Basically maximize the number of like, you know, CUDA cores that you can have on this device or or Tensor cores actually. Tensor and CUDA cores are a little bit overlapping here. And that was like the first training chip, first AI chip, but it was extremely inefficient compared to like today's chips. Right? H100 was the hopper, was the first architecture that was designed for AI, for running training, for running inference.