The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:15:32 · Harry Stebbings, Erik Bernhardsson
[00:15:32] S1: Is it a market that has software margins? Because we're looking at margins now in the 25 to 40% range. That's kind of traditional players. I don't know if that's yours, but that's kind of what inference providers are generally. Do they go to 70 as you earn more [00:15:45] S0: and more of the stuff? I think basically almost no company today has software margins anymore. Like what you used to see in Silicon Valley ten years ago with this 90% margins plus, I think that's basically gone. And the reason is actually almost topological. It's like you have this like stack of vendors, right? Like today you have applications, you have models, you have infrastructure, you have cloud, you have Nvidia. And so when a customer pays a dollar at the top of the stack, it has to flow through a lot of different every layer, there's going to be, you know, 20%, 30% gross margins. So I think there's a lot of margin opportunity, and there's a lot of opportunity to make a lot of money. I mean, the space is so big, but you're also splitting it across a lot of layers in the stack. So, I mean, you know, like the 90% software margin businesses are gone. I think Modal has an opportunity long term to get to 40%, 50%, 60% margins, but we're never going to get to software margins. [00:16:35] S1: What's a wild prediction you have on the future of the inference market that you think many people would disagree with?
00:16:35 · Harry Stebbings, Erik Bernhardsson
[00:16:35] S1: What's a wild prediction you have on the future of the inference market that you think many people would disagree with? [00:16:40] S0: I one prediction about the inference market is gonna split into a lot of different things because the market got so big in the last... I don't think anyone expected, me included, I don't think anyone expected it to be, you know, so big. It's like 10 x or a 100 x bigger than I ever thought. And that to me means there's gonna be so much room for specialization, low latency, high latency, like various types of fine tuning and post training. I think there's gonna be some wacky new technologies that let you do it with way less resources, new hardware, like all kinds of different things. [00:17:07] S1: I'm really sorry. Does that go against what you just said, though, about the need to own it all and have that breadth of product? If it's so much larger that you need real specialization
00:17:07 · Harry Stebbings, Erik Bernhardsson
[00:17:07] S1: I'm really sorry. Does that go against what you just said, though, about the need to own it all and have that breadth of product? If it's so much larger that you need real specialization [00:17:15] S0: Specialization by product. I think it's gonna fragment into a lot of different products. Now we can build a lot of products and host it all under Modal. And that to me is very exciting because a lot of different, you know, there's gonna be a lot of different products that companies are gonna need. For one product, they're gonna need low latency inference. For one product, they're gonna need batch inference. For one product, they're gonna need sandboxes, which we have. For one product, they're gonna do pre training their own models. You know, maybe it's, you know, certain specialized, you know, audio models or whatever. Like, do a lot of stuff besides LM inference. And so that to me, that creates a lot of opportunity to basically kind of rethink what a cloud looks like in the era of AI, and kind of deconstruct the cloud and rebuild it for this new era of software. [00:17:55] S1: Earlier, you said about building capacity with GPUs. Yep. What's it like doing that today? Is it as crazy as reports suggest? So it is pretty crazy.
00:17:55 · Harry Stebbings, Erik Bernhardsson
[00:17:55] S1: Earlier, you said about building capacity with GPUs. Yep. What's it like doing that today? Is it as crazy as reports suggest? So it is pretty crazy. [00:18:04] S0: I spent some time in March at GTC NVIDIA's big conference in San Jose. I would describe that as like an open air drug market, but GPUs instead of drugs. You're running around, you know, talking to 500 different cloud vendors. You're like, do you have GPUs? Like, please get... And And like, know, in the end, you come out, like you do a lot of deals, and that's something we've had to develop the muscle. I think part of the value prop of Modal is like, our customers don't have to think about that. We do the dirty work of doing all the GPU deals under the hood. We work with 25 different NeoClouds. Or, you know, right now we control like, you know, 0.1% or whatever, I don't even know the number, of like the world's compute. But like, my like crazy, like, you know, how I think about the long term goal is like, as we control more and more of the world's compute, you and aggregate demand on the other side, and manage that capacity and demand extremely efficiently. So that to me is like something we actually think we can do, we can fill a big gap. [00:18:55] S1: What does that allow you to do, being that aggregation layer?
00:18:58 · Erik Bernhardsson
[00:18:58] S0: Well, so we built a lot of our infrastructure ourselves. We don't use Kubernetes and Docker and things like that. We built our own file system, we built our own container runtime, we built our own scheduler, we built our own, basically how we manage the capacity. And that enables us to basically treat like all of the world's compute as one big pool of capacity. And so if you think about the fact that there's... We already have many thousand paying customers, and over time it just grows and grows. Each of those customers have like very sort of variable requirements, like their workloads kind of vary over the day, and they have maybe bursts, and each individually, they're like very sort of unpredictable. But if you pull all of those customers together, you end up with a very sort of smooth overall demand. And that is something we can manage extremely efficiently. And so one of the valid props of Modal besides, like, we we can scale up with you is also, you don't even have to do capacity planning. Just come to us, we'll give you 100 GPUs. I think it's so amazing going back to TypeSay, for instance, like they, we had talked to them, I think a long time ago before they launched, but they launched overnight on Modal and we weren't even aware of it to just like scale up. And we gave them a ton of GPUs and it just happened automatically. So that magic, I think, is something that is people really value with Modal is like we can give them a lot of GPUs on very short notice. And we can handle that sort of variable. Like if you go to us, you don't have to lock up, you know, like a rectangular block of GPUs over three years, you and have to know the exact number. It's pay as you go. You scale up and down. You pay per GPU second.
00:20:19 · Harry Stebbings, Erik Bernhardsson
[00:20:19] S1: How do you do GPU buying in terms of forecasting then? Because you don't wanna have an insane excess of supply just sitting on your books. [00:20:27] S0: Totally. The GPU lead times right now is about three to four months. So we basically do a straight line or exponential line extrapolation, and we look at an aggregate across all our customers. We have a very diversified set of customers. You can actually look at the demand in aggregate pretty well and predict based on monthly growth rates, which have been about 30, 40% month to month for the last twelve months. And you just project that, you know, three to four months into the future, you end up with like two to 3x. You go out and, you know, buy that capacity and make sure to ramp up, and sometimes [00:20:54] S1: But can you just buy that capacity? [00:20:55] S0: You can. Yeah. I mean, I think there's a little bit of, like, a misconception that, we're out of capacity. I mean, NVIDIA is, like, shipping, like, an insane amount of GPUs, like, every month. I don't know how many 100,000 black holes are getting installed. Prices are up for sure. You have to pay more to get the same amount of GPUs. And I don't wanna say it's like an efficient market, but if you're willing to pay more than other people, you can get a lot of GPUs. [00:21:15] S1: Is there a price at which it doesn't make sense for you? Of course there is, but at what point