The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:41:08 · Thomas Sohmers, Harry Stebbings
So it it saves you as an operator money more than anything. At some level, you know, having users' KV caches be persistent will give some level of speed improvements that the user perceives, but it's mostly an economics thing for the service provider where if you can return to them and and use those tokens again and again, that saves you money as an operator massively. [00:41:29] S1: Totally get that. It saves us money because we don't have to use as much compute, but then it's harder from a memory challenge perspective. How do you think about the right logical next step then? If you appreciate the importance of saving on compute, but the challenge of memory with KBCaching, what's the answer then? That we just have bigger and bigger memory stacks on chip? What does that look like?
00:41:50 · Thomas Sohmers
[00:41:50] S0: The most common deployed solution and, you know, the vast majority of inferences out there are taking place on GPUs, you know, followed up by TPUs and a couple other devices. But the most common paradigm today is you've got your GPU accelerator memory that is primarily responsible for holding the weights, and you will keep some number of user sessions on there, the ones that you're actively processing. But the larger group of users has a tiered hierarchy. So you'll have users that were around, say, in the past couple of seconds but haven't returned, don't have an active request. That's residing in host memory. And let's say that's on the order of anywhere four to 10 x more memory on the host than in the accelerators. So you'll be able to store more of those there. And then if someone hasn't been around in a couple minutes, maybe a couple hours, that's going to be stored in even further away memory. So that could be in NVMe. So so flash storage, so a lot slower but a lot larger capacity on that host. It could be in flash storage on a network attached, you know, drive. And eventually, like, I bet the ChatGPT sessions that I had, you know, six months ago, somewhere residing on a, you know, disk, you know, slow SSD or, you know, somewhere in a in a data center. But it would be dumb for them to use expensive memory to store that. So that tiering is the norm, but that introduces a huge amount of complexity of how do you decide when and where you're going to store something for your massive number of users. I would say our solution, kind of how we're trying to go about it, you know, both from our expectation that model sizes are going to drastically increase, the number of users for all these things are going to drastically increase, and the context themselves. Like, two, three years ago, the typical context links were on the order of 8,000 to, like, 64,000 tokens. Then it got up to one twenty eight, 256. You know, a million token context links are the norm now in terms of what the model supports. But a million token context links can only hold, you know, a portion of some of, like, our internal companies', like, largest code repositories. Like, it it will be a fraction of that.
00:43:48 · Thomas Sohmers, Harry Stebbings
And so if you really want a agent that can can take over, you know, the the capabilities of a whole team of programmers, I think the main limiter today isn't, like, the model capabilities itself and and scaling model size. It's on how much context can that that model have of all of the data it needs to make smart decisions. [00:44:06] S1: I just wanna break some of the things you set up there. You said that you think model sizes will increase. I thought we were all moving to owning our own intelligence, every enterprise having their own smaller model with proprietary data. Does that go against what you think in terms of model sizes increasing? Can you help me understand?
00:44:06 · Harry Stebbings, Thomas Sohmers
[00:44:06] S1: I just wanna break some of the things you set up there. You said that you think model sizes will increase. I thought we were all moving to owning our own intelligence, every enterprise having their own smaller model with proprietary data. Does that go against what you think in terms of model sizes increasing? Can you help me understand? [00:44:23] S0: Yeah. I think you can kind of break it into two tiers again. So there's going to be the frontier models and capabilities that are being really at the forefront of the development by OpenAI, Anthropic, maybe Google, you know, SpaceX AI, etcetera. I still think there is a long road to go in terms of getting to pushing the frontier of model capabilities, and those will continue to grow, continue to get get better. And there are a lot of workloads where, let's say, internal, just speaking for how positron uses LLMs, where I don't today care that much about the cost. If I can get 10 times the output value out of model today, I very gladly pay 10 times more, you know, per token. And I really want the frontier to push that. You know, some of it is cost savings. Some of it is just truly owning your proprietary data. There is a push from from enterprises to to have inference on-site, and it's a lot more difficult to provide that for largest models. And most companies, you know, if if they're adapting from open source or developing their own model, don't have the resources to pushing the frontier. And so that is kind of what's gone smaller models. [00:45:27] S1: And do you not buy that reality?
00:45:29 · Thomas Sohmers
[00:45:29] S0: I would say, like, right now, somewhere around 80, 85% of all tokens consumed in Produce are done by just the top four model companies. And I would say the next five or 10% is done by the three or four after them. So I can totally buy... Believe that 5% of all tokens consumed will be done by things on prem, you know, locked into the big guys. But both from Positron's business perspective and just how I see the world evolving, I... I'm going to care more about the the high volume set of things. That being said, I actually think the the small model stuff is actually much more interesting for everything happening on on your phone. And and the amazing thing about that is I think there's this misconception that, oh, if questions could be answered by your phone or or, you know, any prompts can be can be done locally, that actually is meaning that there's less tokens going to be used with the big guys in the cloud. I think it's the opposite. The the reality is that if I have an LLM running on my laptop or phone or in my enterprises, you know, secure on prem cloud, whatever, that is going to be consuming data at such a fast rate of everything coming into it, and it's going to be generating, you know, analysis based on that. And it will decide, okay, what is it that I'm going to actually return to the user, you know, locally, and what is it actually requires more intelligence from a better model that isn't self hosted? I think for, you know, any of the tokens that are being, quote, unquote, saved by running locally, that's actually going to generate more things. Because, I mean, in in some ways, for for a simple naive use case of a personal user of LLMs, they're only going to prompt ChatGBT or Quad every so often. Like, they're they're sort of limited by their thoughts of when to actually ask an LLM something. But if they have a local LLM that is constantly checking their email, their calendar, messages, etcetera, deciding to do these lookups to cloud hosted models frequently, that's now on a per person basis, a massive increase in the no... Number of tokens being consumed and generated by the cloud models even though there was... The naive view is that, oh, there's the shift to this on device, LLM.
00:47:35 · Harry Stebbings, Thomas Sohmers
[00:47:35] S1: So can I just understand? So I completely hear you in terms of maybe 5% of them will be in this smaller model enterprise owned kind of model landscape. Why are you so bullish then on much larger models and the size is increasing? [00:47:49] S0: I'll break it into portions. So there's the increase of the model sizes, which I think of... The thing that would be shared by a lot of people in the AI space is it's kind of a gut feel where based on the fact that we have seen these scaling laws. Like, we we call them scaling laws. The fact that going from a, you know, 100,000,000 to a billion to 10,000,000,000, 100,000,000,000, 1,000,000,000,000 parameter models, we've seen this amazing increase in capabilities with that. We see that, like, still to this day going from a trillion to five to 10,000,000,000,000 at that... The largest end right now. Like, we call it a law because we've observed it, but there's no, like, actual mathematical proof that this will continue. So it's sort of on vibes that, okay, this has continued scale. There's no sign of it slowing down. So is that going to continue to 50,000,000,000,000, 100,000,000,000,000, and and beyond? And I don't see any indication that that's going to stop, so I'll I'll be bullish [00:48:41] S1: about does scaling laws look like at three times what it is now? If AGI has been declared now by Janssen, forgive me, but what is three times this?