The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:38:15 · Thomas Sohmers, Harry Stebbings
And that can be applied to weights, the, you know, actual parameters in the model that could be applied to the KV caches, but there's no such thing as free lunch. You... You'd still have some lobotomy, but, thankfully, it's kept within, like, 1% of a unquantized model. [00:38:31] S1: Is KV caching the hardest element of building that inference infrastructure, or is it, you name it, latency SLOs or load spikes or anything else that we could come up with? Is that the hardest? Like, what do we not see that we should see?
00:38:48 · Thomas Sohmers
[00:38:48] S0: You can run a service and do something without having KV caching at all. You're you're going to economics and performance and everything else wise is gonna be much worse. The dark art and magic with it is the workloads that the industry so far has found the most valuable happen to be very, very highly cacheable. So semi analysis has their agent x benchmark and and, you know, suite of of test data based on taking a whole lot of, quad code sessions and and having dozens to hundreds of turns in in those quad sessions with sub agents and everything else. And what they found is over these massive number of interactions of these, like, real traced code generation agentic coding sessions, about 96% of all the tokens that go through these entire sessions are cached. So if you know your workload is going to have this extremely high caching rate where you're going to be reusing the same tokens again and again, that drastically shifts the importance of how you can retrieve those caches because these things get to be very, very large. Like, we, you know, have gone into trillions of parameters. So if we just take, you know, the GPT four, you know, got leaked as, you know, 1,800,000,000,000 parameter model. Now assuming that that is in four quantized and and rounding down a little bit, you know, that's 900 gigabytes of data size for the model weights. If we take, like, the high expectations of, like, Claude Fable, you know, that's a 10,000,000,000,000 parameter model, around five terabytes of model weights. But the crazy thing is at these long context links for these size models, you have the individual user sessions being in the, let's say, in the 100 gigabyte range. So with just 50 users on your service, the user context, they're just those individual sessions end up being greater than the model weights that you're trying to store. So that's... You know, Quad and OpenAI have have a whole lot more than 50 users. And so it becomes a really interesting trade off of, okay, how much of the accelerator memory do you wanna dedicate to weights, which you need to process every single token generated and you want that to be as fast as possible because that sets your SLO. That sets the token latency. But if you don't have their KV caches persistent, you're actually losing a huge amount of efficiency because that was work that you didn't have to actually repeat.
00:41:08 · Thomas Sohmers, Harry Stebbings
So it it saves you as an operator money more than anything. At some level, you know, having users' KV caches be persistent will give some level of speed improvements that the user perceives, but it's mostly an economics thing for the service provider where if you can return to them and and use those tokens again and again, that saves you money as an operator massively. [00:41:29] S1: Totally get that. It saves us money because we don't have to use as much compute, but then it's harder from a memory challenge perspective. How do you think about the right logical next step then? If you appreciate the importance of saving on compute, but the challenge of memory with KBCaching, what's the answer then? That we just have bigger and bigger memory stacks on chip? What does that look like?
00:41:50 · Thomas Sohmers
[00:41:50] S0: The most common deployed solution and, you know, the vast majority of inferences out there are taking place on GPUs, you know, followed up by TPUs and a couple other devices. But the most common paradigm today is you've got your GPU accelerator memory that is primarily responsible for holding the weights, and you will keep some number of user sessions on there, the ones that you're actively processing. But the larger group of users has a tiered hierarchy. So you'll have users that were around, say, in the past couple of seconds but haven't returned, don't have an active request. That's residing in host memory. And let's say that's on the order of anywhere four to 10 x more memory on the host than in the accelerators. So you'll be able to store more of those there. And then if someone hasn't been around in a couple minutes, maybe a couple hours, that's going to be stored in even further away memory. So that could be in NVMe. So so flash storage, so a lot slower but a lot larger capacity on that host. It could be in flash storage on a network attached, you know, drive. And eventually, like, I bet the ChatGPT sessions that I had, you know, six months ago, somewhere residing on a, you know, disk, you know, slow SSD or, you know, somewhere in a in a data center. But it would be dumb for them to use expensive memory to store that. So that tiering is the norm, but that introduces a huge amount of complexity of how do you decide when and where you're going to store something for your massive number of users. I would say our solution, kind of how we're trying to go about it, you know, both from our expectation that model sizes are going to drastically increase, the number of users for all these things are going to drastically increase, and the context themselves. Like, two, three years ago, the typical context links were on the order of 8,000 to, like, 64,000 tokens. Then it got up to one twenty eight, 256. You know, a million token context links are the norm now in terms of what the model supports. But a million token context links can only hold, you know, a portion of some of, like, our internal companies', like, largest code repositories. Like, it it will be a fraction of that.
00:43:48 · Thomas Sohmers, Harry Stebbings
And so if you really want a agent that can can take over, you know, the the capabilities of a whole team of programmers, I think the main limiter today isn't, like, the model capabilities itself and and scaling model size. It's on how much context can that that model have of all of the data it needs to make smart decisions. [00:44:06] S1: I just wanna break some of the things you set up there. You said that you think model sizes will increase. I thought we were all moving to owning our own intelligence, every enterprise having their own smaller model with proprietary data. Does that go against what you think in terms of model sizes increasing? Can you help me understand?
00:44:06 · Harry Stebbings, Thomas Sohmers
[00:44:06] S1: I just wanna break some of the things you set up there. You said that you think model sizes will increase. I thought we were all moving to owning our own intelligence, every enterprise having their own smaller model with proprietary data. Does that go against what you think in terms of model sizes increasing? Can you help me understand? [00:44:23] S0: Yeah. I think you can kind of break it into two tiers again. So there's going to be the frontier models and capabilities that are being really at the forefront of the development by OpenAI, Anthropic, maybe Google, you know, SpaceX AI, etcetera. I still think there is a long road to go in terms of getting to pushing the frontier of model capabilities, and those will continue to grow, continue to get get better. And there are a lot of workloads where, let's say, internal, just speaking for how positron uses LLMs, where I don't today care that much about the cost. If I can get 10 times the output value out of model today, I very gladly pay 10 times more, you know, per token. And I really want the frontier to push that. You know, some of it is cost savings. Some of it is just truly owning your proprietary data. There is a push from from enterprises to to have inference on-site, and it's a lot more difficult to provide that for largest models. And most companies, you know, if if they're adapting from open source or developing their own model, don't have the resources to pushing the frontier. And so that is kind of what's gone smaller models. [00:45:27] S1: And do you not buy that reality?