The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:07:03 · Harry Stebbings, Thomas Sohmers
[00:07:03] S1: So I totally get that in terms of the shift from compute bound to memory bound. Is that what people mean when they say about the memory wall with regards to what you're doing? [00:07:12] S0: Partially. I mean, the memory wall as a phrase has has been around for a long time before, you know, all of the hype around AI. And, really, what it's come down to is if you look at the past fifty, sixty years of computing, we've been able to, you know, have Moore's Law giving us more transistors per, you know, square millimeter of silicon, you know, consistently. And while that has been able to result in, you know, greater raw computes, flops, etcetera, the improvement of the memory technology is not kept up at the same rate. So roughly speaking, you know, between, you know, 2014, just very early innings of the new AI era till 2024, you had about a 120 x improvement in the FLOPs of GPUs. So a single NVIDIA GPU had about a 120 fold improvement in FLOPs, you know, and that's what enabled a whole lot of the the improvements over that decade. The improvement in memory bandwidth is only 17 x. So I would say, like, the the real embodiment of this is we had massive improvements on a per device basis of the flops and then a whole bunch of elements on the periphery of improving the, connectivity, etcetera, etcetera. But just like the ratio of the compute to memory bandwidth had this divergence. And so you had cases where if a problem was memory bound and you couldn't just scale the compute linearly with that, you were getting more and more memory bound as the decade progressed.
00:07:12 · Thomas Sohmers, Harry Stebbings
[00:07:12] S0: Partially. I mean, the memory wall as a phrase has has been around for a long time before, you know, all of the hype around AI. And, really, what it's come down to is if you look at the past fifty, sixty years of computing, we've been able to, you know, have Moore's Law giving us more transistors per, you know, square millimeter of silicon, you know, consistently. And while that has been able to result in, you know, greater raw computes, flops, etcetera, the improvement of the memory technology is not kept up at the same rate. So roughly speaking, you know, between, you know, 2014, just very early innings of the new AI era till 2024, you had about a 120 x improvement in the FLOPs of GPUs. So a single NVIDIA GPU had about a 120 fold improvement in FLOPs, you know, and that's what enabled a whole lot of the the improvements over that decade. The improvement in memory bandwidth is only 17 x. So I would say, like, the the real embodiment of this is we had massive improvements on a per device basis of the flops and then a whole bunch of elements on the periphery of improving the, connectivity, etcetera, etcetera. But just like the ratio of the compute to memory bandwidth had this divergence. And so you had cases where if a problem was memory bound and you couldn't just scale the compute linearly with that, you were getting more and more memory bound as the decade progressed. [00:08:34] S1: Why was there such a misalignment in the progression between the two? One's a 100 x, one's 17 x. Why is that the case?
00:08:42 · Thomas Sohmers
[00:08:42] S0: Yeah. Well, it comes down to a lot of, technical implementation details, like the fact that you if if you look at the lowest level, the type of memory that is used on the silicon itself is called SRAM, static RAM. And SRAM is made out of, you know, six transistors with a bit line and word line, some other control logic around it. But that SRAM cell has not scaled in terms of the the sizing of that with Moore's Law over the past about fifteen years. So they have grown or shrunk, I should say, much slower than just a group of transistors that you'll use for for other purposes. And I would say that there's just been a lot more architectural advancements that could happen on the compute side while an s six TSRAM more or less has not changed in thirty or forty years from a, like, an architectural, you know, primitive perspective. And so, you know, that's on the input side of raw technical capabilities on fabrication, etcetera, have not been able to improve. But I would also say that there wasn't the right motivations for most of that that decade. So with convolutional neural networks, so things that powered, like, AlexNet, which, you know, really launched the deep learning revolution in 2012, and then ResNet and, you know, all of the, I would say, the advancements during the twenty tens was in the realm of machine learning models that were fundamentally compute bound. You could just throw more and more flops at CNNs and get better results, and you didn't really need all that much, be it memory capacity or memory bandwidth. But it was really with the transformer. And even though the attention is all you need paper came out in 2017, would I say it did not really get the attention, you know, pun intended, deserved until, 2020. Well, GPT one and GPT two came out prior to that 2018, 2019, but it was really GPT three showing that, okay, you go from a parameter up to a 175,000,000,000 parameters, and you actually get this massive improvement in capability. And that's really where, I would say, the transformer revolution started, and most people didn't really catch on to that until the end of twenty twenty two when, ChatGPT came out.
00:10:48 · Harry Stebbings, Thomas Sohmers
[00:10:48] S1: When you look at token economics and token efficiency today, what does, like, no one know or talk about that you think should be much more front and center? [00:10:56] S0: I do find it compared to a year or two ago, there are now different prices listed for cash versus uncashed tokens, but I don't think people realize how any providers that charge the same amount, even with a lot of people's cash prices, how high margin that is. It's, like, insane. You you make all of your money on selling cash input and output tokens. [00:11:20] S1: Why is that? Sorry. Just so I understand that. [00:11:22] S0: Oh, because for... Like, when we discussed earlier that both processing a cash token is essentially free. It's it's one one thousandth of the cost, you know, you know, order of magnitude of actually having to recompute and and generate that token. So there there is so much you you can juice out of selling those those cash tokens. And basically, all the providers, they charge you to cash a token. They charge a higher rate than just, you know, the normal processing fee for, like, an input token, and then they charge you a lower rate when you read from that. And it's great when you're paying that lower rate, but they're making obscene margin on that cash read. And there's a reason why, you know, Anthropic is being... Is reported to have, you know, 80 points of gross margin right now on, you know, API business. [00:12:09] S1: Were you surprised by that 80 points?
00:12:09 · Harry Stebbings, Thomas Sohmers
[00:12:09] S1: Were you surprised by that 80 points? [00:12:11] S0: Not really. I'm impressed by 80 points of margin in basically any industry. It's it's difficult to get that margin, and great thing about capitalism is those margins will compress with competition. So I'm I'm confident and happy for that even though those people are, you know, theoretically my customers and my margin is sort of based on their margin. But I care more about, you know, a healthy ecosystem long term. It... It's more surprising to me how many people still today think that these are horribly unprofitable businesses and that the whole market's going to zero. It's it's absurd to me that the meme of OpenAI, Anthropic, etcetera, are just burning cash, and eventually, they'll run out of cash that they can burn. Like, if they stop training, they'd be massively profitable overnight. And there's a ton of other levers that they have without pacing the frontier as as Dario just had in his essay. I I mean, I jokingly think that, you know, a little bit of the pacing the frontier discussion is, oh, this is a great way to to reduce costs, ahead of IPO. But I I I don't think Anthropic or or anyone needs to do that. I think they're they're amazingly profitable businesses with their scaling rates. And [00:13:16] S1: And it would reduce costs, just so I understand, saying, because they would spend less on training because they would be slowing down the speed. Yeah. Right? [00:13:23] S0: Yeah. I don't think that's actually the intention or anything. But, yes, the... That's the bulk of their costs.
00:13:23 · Thomas Sohmers, Harry Stebbings
[00:13:23] S0: Yeah. I don't think that's actually the intention or anything. But, yes, the... That's the bulk of their costs. [00:13:27] S1: How did you analyze the pacing of frontier? You brought it up. How did you analyze it?