Invest Like the Best with Patrick O'Shaughnessy
00:31:35 · Neil Movva
[00:31:35] S2: Cerberus and Grok and maybe a couple others, you should think of them as accelerators. What they are really good at is being used in conjunction with a more traditional GPU like device that critically has this off chip memory built in. You want off chip memory for capacity and on chip memory for speed. We wanna hybridize these two things. So if you take transformers in the limit, you take a transformer to a million context length. What ends up happening is you have this compute bound stage, which is the actual matrix multiplies for the what we call the MLP, which is where most of the model's knowledge, world knowledge is encoded. And then you have the attention layer, which is where we're dynamically adapting to the current conversation. Attention in the limit is usually memory bound. The MLP in the limit is compute bound at large enough batch size. I would say the original sin of transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a compute bound layer. It is very difficult to have a single chip that is good at both compute operations and memory operations. The GPU is quite balanced in this regard but you had to choose one or the other. Cerebras has a very fast memory access for something like a matrix will apply and it's really good to host the MLP, the weights essentially on the Cerebras chip. But the GPU has the capacity to scale to really long context lengths. You would like to put the attention possibly on the GPU and the MLP on the Cerebras chip. And I believe this is what's happening with NVIDIA and Grok.
00:32:57 · Patrick O'Shaughnessy
[00:32:57] S0: Can you riff for minute just on transformers? For people that, again, aren't deeply familiar with what this innovation was in 2017, what its strengths and weaknesses are and whether or not you think it will remain the dominant architecture or a dominant architecture for the future of AI?
00:33:12 · Neil Movva
[00:33:12] S2: What it did, it allowed us to learn unsupervised data really effectively. Because transformers, what they're all about at the end of the day is taking any sequence, any arbitrary sequence of data and trying to find patterns in that data. And they critically, the attention operation, which is the headline component of transformers, it allows the model to dynamically adapt to what it thinks is the most relevant component of the sequence. Every step you take through a transformer, you are essentially like reweighting the input that you looked at before and figuring out which is most relevant for your next prediction. It's extremely amenable to learning arbitrary sequence data. And the most interesting sequences of data that we produce on a regular basis is language. And that's how we got to dominance in the language regime. To zoom out even further, think what transformers really did well is that they scaled. Transformers make no such human prior. Transformers just say, well, there's gonna be a pattern in the sequence of data and if there is a pattern, I'm gonna find I'm going to throw more and more parameters at this problem until it works. Transformers benefit from a lot of the computer vision work too. For example, one of the challenges in computer vision was we had a hard time going from hundreds of thousands of parameters, which you get for linear models like support vector machines or other legacy machine learning models. Those had thousands of parameters. Then we got to deep learning and got to tens of millions of parameters with computer vision. The biggest models were a 150,000,000 parameters was a huge model for computer vision. And now we routinely talk about trillions of parameters. And transformers are the link to go from millions to trillions of parameters.
00:34:41 · Patrick O'Shaughnessy
[00:34:41] S0: So if I think about the important units of scaling being data and compute, does it stand to reason that you think transformers will just stick around because that's the thing that we're good at getting more of, those two things?
00:34:52 · Neil Movva
[00:34:52] S2: Transformers are such great sponges. You increase the compute available to transformer by 10 x and you'll get some log improvement somewhere. And so far, the scaling laws really work. They're really quite beautiful. And to the point about what do transformers do really well, they extend to almost any data set you can throw at them. They're extremely powerful general learners. And I think what's especially useful about transformers over other techniques that we've tried to replace attention is transformers represent any pairwise relationship that you want. Any token in the sequence can attend to any other token in the sequence. So if there's any relationship that's in the sequence at all, you're gonna find it with transformer. Now, may be the case that you don't need all to all modeling. You don't need every token to look at every other token. But if you need to, transformers give you that option. And until we know a better way to prune that space down, a better way to kind of have information modeling be more selective, Attention is a very, very good operation. This is, another trick that we learned in the computer vision days. One of the old Karpathy sayings is that if you have a new dataset that you want to train a model for, your first goal should be to overparameterize the model and try to overfit the data that you have to prove that there is a relationship that you can model or memorize, that your learning algorithm works, that you can instill knowledge into the model. Once you can overfit, then you can compress. And the compression is how you get generalization. You don't want to actually memorize the data that you have in front of you, you want to generalize and therefore, once you overfit the dataset, then you can kind of work backwards and try to find the general patterns that fit into the smallest parameter count possible.
00:36:22 · Patrick O'Shaughnessy
[00:36:22] S0: What's your prediction for the future of data and riff on the importance of data in this whole story?