"The Cognitive Revolution"
00:54:31 · Nathan Labenz
[00:54:31] S0: I asked Freeman about the training itself. I would I would say, like, sort of the anthropic take has been like, we didn't train this to be a cyber attacker. It just kind of got that way because it's so good at everything. Okay. That's one story. The CTO of Vercel, was a guest on the show here with us, and he said it's very clear when you look at the traces that they've been trained to be cyber attackers. His point of view was like, this is not the sort of behavior that you just get falling out of random, you know, application development. Where do you think we are in terms of, like, how much this has been intentional? And does it matter at this point given what exists, how intentional it was? I mean, it might matter if you could think maybe we can make some different decisions for the next generations of models, but obviously, to some degree, the the cat's out of the bag. What do you make of how we got here, and is there any opportunity to, like, reel it back in?
00:55:25 · Michael Freeman
[00:55:25] S5: Not reel it back in. No. But I definitely agree that it is being developed on offensive capabilities. Absolutely. Because it's not just the models. It's the harness itself also. So you take a look at when you develop the models, there's plenty of great open source models in the past three weeks that have came out that are specific to cybersecurity that are open source models. You can download the weights, can go to Hugging Face, Hugging Face, download these and run them on your local machines. And that's what a lot of these attackers are actually doing. And it's actually quite interesting to see as we start tracking back, getting attribution on these attackers, that all these are just local machines that they're running them on. We developed our own small language model two years ago to find vulnerabilities within the software. And so, if we can do it for a couple $100,000 two years ago on a small language model, it's pretty trivial to start reverse engineering, especially some of these commercial offerings that is definitely trained on cyber for sure. It's gaining that... You know, they train it on, you know, cyber ranges, CTF data, looking for, you know, how do these offensive people work as well as how's the defense work? What are they looking for? So it's absolutely being trained that way. It's not just the LLMs have magically figured out how to do offensive capabilities. They've been trained on it. They've been built better harnesses to make them more effective. And that's one of the key things we're actually finding are the harnesses that their actors are building to make this a lot more effective for them. They're learning as they're going and, you know, it's not hard to make changes to a harness day in and day out to make it more effective.
00:57:14 · Nathan Labenz, Michael Freeman
[00:57:14] S0: Earlier that morning, Prakash had cited Jason Clinton, Anthropics deputy chief information security officer, who talks about getting to vulnerability zero. Prakash put the idea to Freeman from the seat of a company like Nike with decades of legacy systems. How do you get there in two years? Freeman's answer, some of the most critical vulnerabilities come from business processes, and no amount of patching fixes those. He went on. [00:57:42] S5: So I think that's gonna be an area that, like I said, AI people just don't understand cybersecurity, and I think it's gonna be quickly here. We can find millions of vulnerabilities. We're gonna use AI to, you know, find the... You know, what you have in your environment. We're gonna patch it. And I've also heard proposals where you're running legacy systems where you can use AI to modernize that very quickly. And it's... All those are true to some extent, but they don't understand the business. They'd also don't understand, you know, the business processes that introduce the vulnerabilities are always gonna be there.
00:57:42 · Michael Freeman, Nathan Labenz
[00:57:42] S5: So I think that's gonna be an area that, like I said, AI people just don't understand cybersecurity, and I think it's gonna be quickly here. We can find millions of vulnerabilities. We're gonna use AI to, you know, find the... You know, what you have in your environment. We're gonna patch it. And I've also heard proposals where you're running legacy systems where you can use AI to modernize that very quickly. And it's... All those are true to some extent, but they don't understand the business. They'd also don't understand, you know, the business processes that introduce the vulnerabilities are always gonna be there. [00:58:16] S0: How do you feel about things like password managers in today's world? I don't have any particular one in mind. I've used LastPass. I've used 1Password. On the one hand, everybody seems to recommend them. On the other, it's like looking like an awfully juicy target these days. And, you know, it it seems like even, people who've tried pretty hard are still not that secure. So, like, would I be better off storing them all in Chrome or even just, you know, going back to, like, my old habits of sort of the base password and then some pattern for the extension of different sites? Like, what is actually the best move for me today? [00:58:58] S5: I really wish I could answer that question. I've not found a a solution that I really like at all. I I haven't. You know? I I wish I had an answer. [00:59:05] S0: So what do you do?
00:59:05 · Nathan Labenz, Michael Freeman
[00:59:05] S0: So what do you do? [00:59:08] S5: I personally have have it written down on a notebook. Amazing. Most things I can I can remember variations of it for your typical, hey, eBay or whatever the case may be? But it's banking sites, something like that. I have it written down in the book, look at it and say, okay. Things I won't go to often, but it's... You know, I'm gonna check on my stocks every quarter or something like that. That's written down on a a notebook. [00:59:32] S0: Part six, Le Chonk. Sagar Vaze leads the multimodal team at Mistral, which released Mistral Large four Le Chonk as open weights the day before the show. About a trillion parameters, roughly 50,000,000,000 active per token, trained on about 3,800 GPUs outside Paris. He put the training compute at 10 to the twenty fifth flops. In the s b ten forty seven fight, the threshold was 10 to the twenty sixth, and I'd have guessed a model like this would need 10 times more. I asked why it didn't. He said some labs do train at that scale, and this model still competes, and then gave the reason that matters.
01:00:09 · Sagar Vaze
[01:00:09] S4: The second is that, you know, this threshold slot budget does evolve over time because precraning does essentially become more efficient as we... As our level of scientific understanding increases. So a great example of this is sort of MOE architectures. I mean, you look, say, example, GPT-three, this was a dense model trained with three x the number of active parameters as the current Mistral Large four. Particularly, GPT three was trained with 175 b parameters total, but it was a dense model, so all parameters were activated for each token. Whereas ML four is a sparse mixture of experts, such that it's... There's one third the number of parameters activated per token, so only 50,000,000,000 parameters. But the total number of parameters is still 1,000,000,000,000. Your flop budget is proportional to the number of active parameters, so the number of, you know, parameters you're activating each token during training. And what this means is that you can train much larger models, much more capable models, but much less flops in total. So this figure that I referenced, so one e 25, this is in itself a moving target. So I wouldn't use that as an absolute target for a capable model throughout time. There would have been some one e 20 fives much less performant one or two years ago. And I would argue actually some larger models also much less performant today. But as pre training science progresses, as we learn how to make better use of the flops, this number itself will change. And at least what we're seeing is, at the moment, looking at the other open source labs, our understanding of, you know, what are the best models being created at the moment, training a 1,000,000,000,000 parameter model on one e 25 is sort of a rite of passage you have to go through in order to be able to to justify scaling even further.