Invest Like the Best with Patrick O'Shaughnessy
00:29:02 · Neil Movva, Patrick O'Shaughnessy
[00:29:02] S2: What happens to them is some hybrid, we had to pair the Cerberus chip where it's very strong, it's very, very good at fast access to memory with something that has more capacity for memory. It's true that you can take a 1,000,000,000,000 parameter model like Kimi and fit it on a large number of Cerberus Diode wafers. But you can't do something about the KBChash very easily. The KBChash is something that grows as people use the model more and that is always dynamic. You don't even know how much KBChash you're gonna need. It depends on how many users you have and how many users you want to serve. Can you explain KBChash? KBChash, when you ever use a language model, every token you send through the language model stays in the context window of the language model for as long as you're having a conversation. We talk for a 100,000 tokens, the 100,001 token is still in the conversation behind us and the model is referencing all the past conversation history in order to make better predictions about what the next thing we're gonna say is. That KvCache, it's a bunch of memory. You have to store a representation for every token that we send through the language model and it frequently gets to be larger than the weights of the model themselves. You have this crystallized knowledge in the model weights and you have the dynamic knowledge of the exact conversation we're having in the KvCache is the way I like to think about it. [00:30:16] S0: And this is why sometimes people would observe deep in a conversation things start to degrade because there's some sort
00:30:16 · Patrick O'Shaughnessy, Neil Movva
[00:30:16] S0: And this is why sometimes people would observe deep in a conversation things start to degrade because there's some sort [00:30:21] S2: of technical problem? So the KB cache is quite interesting in that regard. The KB cache is an exact representation of everything that came before. We store all the information that we've seen in the conversation. However, during training, the model did not get trained primarily on very long context conversations. It got trained primarily on, let's say, 8,000 token conversations or 16,000 token conversations. So if you take the model to 200,000 tokens, there was some training that happened at that context length, but it's not the models like core strength. And so there's always been a challenge for the frontier labs to figure out how do we make the model exactly as intelligent at 10,000 tokens as we expect it to be at 200,000 tokens. And it's gonna be a perennial battle for us. We've had 1,000,000 context windows as a concept for years now. Anthropic was, I think, the first to hit that 1,000,000 context window length length. I still, you know, use slash compact in my cloud code well before 1,000,000 context length. I don't think it's actually great to hit the full length. [00:31:12] S0: These extremely fast, extremely low latency approaches ultimately are limited by this factor. [00:31:18] S2: Yes. You can do whatever you want for the weights. It's very possible to have much unbeatable performance on weight storage. However, the KBChash is gonna be a big thorn in
00:31:18 · Neil Movva, Patrick O'Shaughnessy
[00:31:18] S2: Yes. You can do whatever you want for the weights. It's very possible to have much unbeatable performance on weight storage. However, the KBChash is gonna be a big thorn in [00:31:26] S0: your side. So three years from now, five years from now, what role do you think these kinds of chips play? Like, what sort of market share do they have in the heterogeneous chip market like...
00:31:35 · Neil Movva
[00:31:35] S2: Cerberus and Grok and maybe a couple others, you should think of them as accelerators. What they are really good at is being used in conjunction with a more traditional GPU like device that critically has this off chip memory built in. You want off chip memory for capacity and on chip memory for speed. We wanna hybridize these two things. So if you take transformers in the limit, you take a transformer to a million context length. What ends up happening is you have this compute bound stage, which is the actual matrix multiplies for the what we call the MLP, which is where most of the model's knowledge, world knowledge is encoded. And then you have the attention layer, which is where we're dynamically adapting to the current conversation. Attention in the limit is usually memory bound. The MLP in the limit is compute bound at large enough batch size. I would say the original sin of transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a compute bound layer. It is very difficult to have a single chip that is good at both compute operations and memory operations. The GPU is quite balanced in this regard but you had to choose one or the other. Cerebras has a very fast memory access for something like a matrix will apply and it's really good to host the MLP, the weights essentially on the Cerebras chip. But the GPU has the capacity to scale to really long context lengths. You would like to put the attention possibly on the GPU and the MLP on the Cerebras chip. And I believe this is what's happening with NVIDIA and Grok.
00:32:57 · Patrick O'Shaughnessy
[00:32:57] S0: Can you riff for minute just on transformers? For people that, again, aren't deeply familiar with what this innovation was in 2017, what its strengths and weaknesses are and whether or not you think it will remain the dominant architecture or a dominant architecture for the future of AI?
00:33:12 · Neil Movva
[00:33:12] S2: What it did, it allowed us to learn unsupervised data really effectively. Because transformers, what they're all about at the end of the day is taking any sequence, any arbitrary sequence of data and trying to find patterns in that data. And they critically, the attention operation, which is the headline component of transformers, it allows the model to dynamically adapt to what it thinks is the most relevant component of the sequence. Every step you take through a transformer, you are essentially like reweighting the input that you looked at before and figuring out which is most relevant for your next prediction. It's extremely amenable to learning arbitrary sequence data. And the most interesting sequences of data that we produce on a regular basis is language. And that's how we got to dominance in the language regime. To zoom out even further, think what transformers really did well is that they scaled. Transformers make no such human prior. Transformers just say, well, there's gonna be a pattern in the sequence of data and if there is a pattern, I'm gonna find I'm going to throw more and more parameters at this problem until it works. Transformers benefit from a lot of the computer vision work too. For example, one of the challenges in computer vision was we had a hard time going from hundreds of thousands of parameters, which you get for linear models like support vector machines or other legacy machine learning models. Those had thousands of parameters. Then we got to deep learning and got to tens of millions of parameters with computer vision. The biggest models were a 150,000,000 parameters was a huge model for computer vision. And now we routinely talk about trillions of parameters. And transformers are the link to go from millions to trillions of parameters.