The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:04:13 · Thomas Sohmers, Harry Stebbings
[00:04:13] S0: Yeah. Great to be here. Thanks, [00:04:14] S1: Harry. Can we just have a brief description of what positron is and where does it sit in the stack? [00:04:21] S0: Yeah. So Positron is a fabulous semiconductor startup that's building hardware. So really everything from the chips, the software directly running on top of that, all the way up to the full systems and rack scale deployments to power generative AI inference. So effectively, everything in the hardware and sort of the low level direct talking to hardware software stack that powers all of the applications, you know, everyone in the world is excited about right now. So everything from, the likes of ChatJPT, Claude, etcetera. [00:04:50] S1: How does the infrastructure stat required for inference, what you're working on, change compared to training?
00:04:58 · Thomas Sohmers
[00:04:58] S0: So training, I would say, from the underlying compute level, fundamentally, you know, is a compute bound problem. So it's a workload that's the more flops that you have. And if you look at, you know, be it from the regulatory and some of the, you know, export control frameworks are are heavily focused on the just FLOPS required. So how many floating point operations per second can be can be done. And more or less, you know, the the amazing thing that the scaling laws of the past decade have shown is that the more parameters you you add to a network and the more FLOPS you dedicate to that during training, the better that that model is going to become. The big difference with inference and so the the deployment of those models is the fact that for the actual math and the the steps that you're doing is about half of what you're doing during training in terms of the the steps not... Shouldn't be thought of as, like, the actual compute involved. But what it turns out to be is that that forward pass, that inference portion of it is heavily, heavily memory bound due to the fact that basically for every single token that's generated, every little bit of output, that requires going through the the weights, the parameters you could kind of, you know, from a biological sense, think of the neurons. You have to read the values of that for every single individual token. And so the way that that... That's, I I think, very interesting about this... And and I I don't know if it says anything about the value or or kind of actually saying that the the inference is is somehow... I I don't wanna say more important because, of course, you have to train. But fundamentally, when you're training, you already have the corpus. You already have all of the training data. And so with all that data, you can massively parallelize the token inputs, all of the the sequences of words and sentences, paragraphs, etcetera, that are going into that. So that's something you can just crush through with a bunch of compute. But when you're inferring, because that's actually generative, you don't know what the token is five words down the line. And so you have to generate each and every one auto regressively or in order without, you know, foresight. And so that becomes a hugely memory bound problem that can't just be massively paralyzed like training.
00:07:03 · Harry Stebbings, Thomas Sohmers
[00:07:03] S1: So I totally get that in terms of the shift from compute bound to memory bound. Is that what people mean when they say about the memory wall with regards to what you're doing? [00:07:12] S0: Partially. I mean, the memory wall as a phrase has has been around for a long time before, you know, all of the hype around AI. And, really, what it's come down to is if you look at the past fifty, sixty years of computing, we've been able to, you know, have Moore's Law giving us more transistors per, you know, square millimeter of silicon, you know, consistently. And while that has been able to result in, you know, greater raw computes, flops, etcetera, the improvement of the memory technology is not kept up at the same rate. So roughly speaking, you know, between, you know, 2014, just very early innings of the new AI era till 2024, you had about a 120 x improvement in the FLOPs of GPUs. So a single NVIDIA GPU had about a 120 fold improvement in FLOPs, you know, and that's what enabled a whole lot of the the improvements over that decade. The improvement in memory bandwidth is only 17 x. So I would say, like, the the real embodiment of this is we had massive improvements on a per device basis of the flops and then a whole bunch of elements on the periphery of improving the, connectivity, etcetera, etcetera. But just like the ratio of the compute to memory bandwidth had this divergence. And so you had cases where if a problem was memory bound and you couldn't just scale the compute linearly with that, you were getting more and more memory bound as the decade progressed.
00:07:12 · Thomas Sohmers, Harry Stebbings
[00:07:12] S0: Partially. I mean, the memory wall as a phrase has has been around for a long time before, you know, all of the hype around AI. And, really, what it's come down to is if you look at the past fifty, sixty years of computing, we've been able to, you know, have Moore's Law giving us more transistors per, you know, square millimeter of silicon, you know, consistently. And while that has been able to result in, you know, greater raw computes, flops, etcetera, the improvement of the memory technology is not kept up at the same rate. So roughly speaking, you know, between, you know, 2014, just very early innings of the new AI era till 2024, you had about a 120 x improvement in the FLOPs of GPUs. So a single NVIDIA GPU had about a 120 fold improvement in FLOPs, you know, and that's what enabled a whole lot of the the improvements over that decade. The improvement in memory bandwidth is only 17 x. So I would say, like, the the real embodiment of this is we had massive improvements on a per device basis of the flops and then a whole bunch of elements on the periphery of improving the, connectivity, etcetera, etcetera. But just like the ratio of the compute to memory bandwidth had this divergence. And so you had cases where if a problem was memory bound and you couldn't just scale the compute linearly with that, you were getting more and more memory bound as the decade progressed. [00:08:34] S1: Why was there such a misalignment in the progression between the two? One's a 100 x, one's 17 x. Why is that the case?
00:08:42 · Thomas Sohmers
[00:08:42] S0: Yeah. Well, it comes down to a lot of, technical implementation details, like the fact that you if if you look at the lowest level, the type of memory that is used on the silicon itself is called SRAM, static RAM. And SRAM is made out of, you know, six transistors with a bit line and word line, some other control logic around it. But that SRAM cell has not scaled in terms of the the sizing of that with Moore's Law over the past about fifteen years. So they have grown or shrunk, I should say, much slower than just a group of transistors that you'll use for for other purposes. And I would say that there's just been a lot more architectural advancements that could happen on the compute side while an s six TSRAM more or less has not changed in thirty or forty years from a, like, an architectural, you know, primitive perspective. And so, you know, that's on the input side of raw technical capabilities on fabrication, etcetera, have not been able to improve. But I would also say that there wasn't the right motivations for most of that that decade. So with convolutional neural networks, so things that powered, like, AlexNet, which, you know, really launched the deep learning revolution in 2012, and then ResNet and, you know, all of the, I would say, the advancements during the twenty tens was in the realm of machine learning models that were fundamentally compute bound. You could just throw more and more flops at CNNs and get better results, and you didn't really need all that much, be it memory capacity or memory bandwidth. But it was really with the transformer. And even though the attention is all you need paper came out in 2017, would I say it did not really get the attention, you know, pun intended, deserved until, 2020. Well, GPT one and GPT two came out prior to that 2018, 2019, but it was really GPT three showing that, okay, you go from a parameter up to a 175,000,000,000 parameters, and you actually get this massive improvement in capability. And that's really where, I would say, the transformer revolution started, and most people didn't really catch on to that until the end of twenty twenty two when, ChatGPT came out.
00:10:48 · Harry Stebbings, Thomas Sohmers
[00:10:48] S1: When you look at token economics and token efficiency today, what does, like, no one know or talk about that you think should be much more front and center? [00:10:56] S0: I do find it compared to a year or two ago, there are now different prices listed for cash versus uncashed tokens, but I don't think people realize how any providers that charge the same amount, even with a lot of people's cash prices, how high margin that is. It's, like, insane. You you make all of your money on selling cash input and output tokens. [00:11:20] S1: Why is that? Sorry. Just so I understand that. [00:11:22] S0: Oh, because for... Like, when we discussed earlier that both processing a cash token is essentially free. It's it's one one thousandth of the cost, you know, you know, order of magnitude of actually having to recompute and and generate that token. So there there is so much you you can juice out of selling those those cash tokens. And basically, all the providers, they charge you to cash a token. They charge a higher rate than just, you know, the normal processing fee for, like, an input token, and then they charge you a lower rate when you read from that. And it's great when you're paying that lower rate, but they're making obscene margin on that cash read. And there's a reason why, you know, Anthropic is being... Is reported to have, you know, 80 points of gross margin right now on, you know, API business. [00:12:09] S1: Were you surprised by that 80 points?