The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:34:19 · Thomas Sohmers
And, you know, clever, I would say, kind of obvious based on, you know, all of the developments in the past of of computer history, but wasn't done initially, was that, well, you don't actually have to redo the compute of the things that you've already, you know, had as input and what you've already generated in this turn. And so the KV cache was born where within the model, there's these two matrices called k and v, keys and values, and those matrices are are fully based on the prompt and whatever is generated during a turn. And so by actually storing those two matrices, you can avoid having to redo computation at the cost of now having to store this thing in memory. And that's, you know, the simple example, you know, very, very small hundreds of kilobytes of data. But the the thing is that these things grow with with the sequence length. But the the interesting thing is that for the attention mechanism, the compute per token grows quadratically with the sequence length. So you're having to spend more and more compute quadratically. So that's, you know, an exponential curve as sequence length grows. But when you store that as just your k and v, that's just a linear growth. What KV caching does is means that you don't have to do that compute, which gets very expensive very quickly at the expense of needing to store these things. Storing that is a complexity in itself because that's a unique KV cache for every single user that you're serving. And then it comes questions of how long do you wanna keep that for and how do you manage all of that in a large system.
00:35:49 · Harry Stebbings
[00:35:49] S1: What does it mean then when we hear about compression and uncompression of KV caching and potential entropy within the system?
00:35:57 · Thomas Sohmers
[00:35:57] S0: So there's two different forms of compression, a couple more than that, but the two main ones. So one is quantization. So if you've got each of your values, be it your weights, your k v caches, activations stored in a particular data type. So before the machine learning revolution, you know, most of the world's computation was done in FP 32. So you have 32 bits to represent a floating point number, and that's broken into mantissa and exponents. Well, it's pretty quickly realized that having 32 bits of precision was super overkill for the things that you're wanting to represent, and, you know, it's both cost more from a storage and computational perspective than lower precision. So we went to f p 16. You know, Google developed b f 16, you know, a little rejiggering of those bits. Went to f p eight. Now we're at f p four, you know, in popular systems. And so we've been reducing the precision quite quickly, but that does lead to, you know, for lack of better words, some brain loss when when these models run just because you are now trying to encode the the same information into fewer bits. And so there's been a lot of interesting schemes to say, okay. I'm going to take this group of f p 16 values, b f 16 values, and I'm going to quantize those. So I'm going to, you know, use a truncate and rounding that that down to, let's say, into four values. So now you you actually saved 75% of your total size of that group of values. You shrunk that down from 16 bits to four bits. But just doing that naively will mean that on a lot of benchmark scores, you'll have them get 20, 30% worse. So you you get that 75% savings in in space, but you you kind of lobotomize the model. But, you know, advanced quantization techniques actually say, okay. These 16 values, I'm able to have a, you know, shared bias and a multiplier for it. Let's say for those 16 now int four or f p four values, you store one new f p 16 value that gets applied to all of those at compute time. So you get a 75% compression on all those values at the cost of now adding to that one new FP 16. And basically, the state of the art here is you're able to get things compressed from, you know, FP sixteen sixteen bits per per value down to, like, four and a half bits per value.
00:38:15 · Thomas Sohmers, Harry Stebbings
And that can be applied to weights, the, you know, actual parameters in the model that could be applied to the KV caches, but there's no such thing as free lunch. You... You'd still have some lobotomy, but, thankfully, it's kept within, like, 1% of a unquantized model. [00:38:31] S1: Is KV caching the hardest element of building that inference infrastructure, or is it, you name it, latency SLOs or load spikes or anything else that we could come up with? Is that the hardest? Like, what do we not see that we should see?
00:38:48 · Thomas Sohmers
[00:38:48] S0: You can run a service and do something without having KV caching at all. You're you're going to economics and performance and everything else wise is gonna be much worse. The dark art and magic with it is the workloads that the industry so far has found the most valuable happen to be very, very highly cacheable. So semi analysis has their agent x benchmark and and, you know, suite of of test data based on taking a whole lot of, quad code sessions and and having dozens to hundreds of turns in in those quad sessions with sub agents and everything else. And what they found is over these massive number of interactions of these, like, real traced code generation agentic coding sessions, about 96% of all the tokens that go through these entire sessions are cached. So if you know your workload is going to have this extremely high caching rate where you're going to be reusing the same tokens again and again, that drastically shifts the importance of how you can retrieve those caches because these things get to be very, very large. Like, we, you know, have gone into trillions of parameters. So if we just take, you know, the GPT four, you know, got leaked as, you know, 1,800,000,000,000 parameter model. Now assuming that that is in four quantized and and rounding down a little bit, you know, that's 900 gigabytes of data size for the model weights. If we take, like, the high expectations of, like, Claude Fable, you know, that's a 10,000,000,000,000 parameter model, around five terabytes of model weights. But the crazy thing is at these long context links for these size models, you have the individual user sessions being in the, let's say, in the 100 gigabyte range. So with just 50 users on your service, the user context, they're just those individual sessions end up being greater than the model weights that you're trying to store. So that's... You know, Quad and OpenAI have have a whole lot more than 50 users. And so it becomes a really interesting trade off of, okay, how much of the accelerator memory do you wanna dedicate to weights, which you need to process every single token generated and you want that to be as fast as possible because that sets your SLO. That sets the token latency. But if you don't have their KV caches persistent, you're actually losing a huge amount of efficiency because that was work that you didn't have to actually repeat.
00:41:08 · Thomas Sohmers, Harry Stebbings
So it it saves you as an operator money more than anything. At some level, you know, having users' KV caches be persistent will give some level of speed improvements that the user perceives, but it's mostly an economics thing for the service provider where if you can return to them and and use those tokens again and again, that saves you money as an operator massively. [00:41:29] S1: Totally get that. It saves us money because we don't have to use as much compute, but then it's harder from a memory challenge perspective. How do you think about the right logical next step then? If you appreciate the importance of saving on compute, but the challenge of memory with KBCaching, what's the answer then? That we just have bigger and bigger memory stacks on chip? What does that look like?