Invest Like the Best with Patrick O'Shaughnessy
00:28:56 · Patrick O'Shaughnessy, Neil Movva
[00:28:56] S0: So there you go. You have a path to a thousand tokens per second. So what is your prediction for, like, that segment of the market? [00:29:02] S2: What happens to them is some hybrid, we had to pair the Cerberus chip where it's very strong, it's very, very good at fast access to memory with something that has more capacity for memory. It's true that you can take a 1,000,000,000,000 parameter model like Kimi and fit it on a large number of Cerberus Diode wafers. But you can't do something about the KBChash very easily. The KBChash is something that grows as people use the model more and that is always dynamic. You don't even know how much KBChash you're gonna need. It depends on how many users you have and how many users you want to serve. Can you explain KBChash? KBChash, when you ever use a language model, every token you send through the language model stays in the context window of the language model for as long as you're having a conversation. We talk for a 100,000 tokens, the 100,001 token is still in the conversation behind us and the model is referencing all the past conversation history in order to make better predictions about what the next thing we're gonna say is. That KvCache, it's a bunch of memory. You have to store a representation for every token that we send through the language model and it frequently gets to be larger than the weights of the model themselves. You have this crystallized knowledge in the model weights and you have the dynamic knowledge of the exact conversation we're having in the KvCache is the way I like to think about it.
00:29:02 · Neil Movva, Patrick O'Shaughnessy
[00:29:02] S2: What happens to them is some hybrid, we had to pair the Cerberus chip where it's very strong, it's very, very good at fast access to memory with something that has more capacity for memory. It's true that you can take a 1,000,000,000,000 parameter model like Kimi and fit it on a large number of Cerberus Diode wafers. But you can't do something about the KBChash very easily. The KBChash is something that grows as people use the model more and that is always dynamic. You don't even know how much KBChash you're gonna need. It depends on how many users you have and how many users you want to serve. Can you explain KBChash? KBChash, when you ever use a language model, every token you send through the language model stays in the context window of the language model for as long as you're having a conversation. We talk for a 100,000 tokens, the 100,001 token is still in the conversation behind us and the model is referencing all the past conversation history in order to make better predictions about what the next thing we're gonna say is. That KvCache, it's a bunch of memory. You have to store a representation for every token that we send through the language model and it frequently gets to be larger than the weights of the model themselves. You have this crystallized knowledge in the model weights and you have the dynamic knowledge of the exact conversation we're having in the KvCache is the way I like to think about it. [00:30:16] S0: And this is why sometimes people would observe deep in a conversation things start to degrade because there's some sort
00:30:16 · Patrick O'Shaughnessy, Neil Movva
[00:30:16] S0: And this is why sometimes people would observe deep in a conversation things start to degrade because there's some sort [00:30:21] S2: of technical problem? So the KB cache is quite interesting in that regard. The KB cache is an exact representation of everything that came before. We store all the information that we've seen in the conversation. However, during training, the model did not get trained primarily on very long context conversations. It got trained primarily on, let's say, 8,000 token conversations or 16,000 token conversations. So if you take the model to 200,000 tokens, there was some training that happened at that context length, but it's not the models like core strength. And so there's always been a challenge for the frontier labs to figure out how do we make the model exactly as intelligent at 10,000 tokens as we expect it to be at 200,000 tokens. And it's gonna be a perennial battle for us. We've had 1,000,000 context windows as a concept for years now. Anthropic was, I think, the first to hit that 1,000,000 context window length length. I still, you know, use slash compact in my cloud code well before 1,000,000 context length. I don't think it's actually great to hit the full length. [00:31:12] S0: These extremely fast, extremely low latency approaches ultimately are limited by this factor. [00:31:18] S2: Yes. You can do whatever you want for the weights. It's very possible to have much unbeatable performance on weight storage. However, the KBChash is gonna be a big thorn in
00:31:18 · Neil Movva, Patrick O'Shaughnessy
[00:31:18] S2: Yes. You can do whatever you want for the weights. It's very possible to have much unbeatable performance on weight storage. However, the KBChash is gonna be a big thorn in [00:31:26] S0: your side. So three years from now, five years from now, what role do you think these kinds of chips play? Like, what sort of market share do they have in the heterogeneous chip market like...
00:31:35 · Neil Movva
[00:31:35] S2: Cerberus and Grok and maybe a couple others, you should think of them as accelerators. What they are really good at is being used in conjunction with a more traditional GPU like device that critically has this off chip memory built in. You want off chip memory for capacity and on chip memory for speed. We wanna hybridize these two things. So if you take transformers in the limit, you take a transformer to a million context length. What ends up happening is you have this compute bound stage, which is the actual matrix multiplies for the what we call the MLP, which is where most of the model's knowledge, world knowledge is encoded. And then you have the attention layer, which is where we're dynamically adapting to the current conversation. Attention in the limit is usually memory bound. The MLP in the limit is compute bound at large enough batch size. I would say the original sin of transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a compute bound layer. It is very difficult to have a single chip that is good at both compute operations and memory operations. The GPU is quite balanced in this regard but you had to choose one or the other. Cerebras has a very fast memory access for something like a matrix will apply and it's really good to host the MLP, the weights essentially on the Cerebras chip. But the GPU has the capacity to scale to really long context lengths. You would like to put the attention possibly on the GPU and the MLP on the Cerebras chip. And I believe this is what's happening with NVIDIA and Grok.
00:32:57 · Patrick O'Shaughnessy
[00:32:57] S0: Can you riff for minute just on transformers? For people that, again, aren't deeply familiar with what this innovation was in 2017, what its strengths and weaknesses are and whether or not you think it will remain the dominant architecture or a dominant architecture for the future of AI?