"The Cognitive Revolution"
00:55:25 · Michael Freeman
[00:55:25] S5: Not reel it back in. No. But I definitely agree that it is being developed on offensive capabilities. Absolutely. Because it's not just the models. It's the harness itself also. So you take a look at when you develop the models, there's plenty of great open source models in the past three weeks that have came out that are specific to cybersecurity that are open source models. You can download the weights, can go to Hugging Face, Hugging Face, download these and run them on your local machines. And that's what a lot of these attackers are actually doing. And it's actually quite interesting to see as we start tracking back, getting attribution on these attackers, that all these are just local machines that they're running them on. We developed our own small language model two years ago to find vulnerabilities within the software. And so, if we can do it for a couple $100,000 two years ago on a small language model, it's pretty trivial to start reverse engineering, especially some of these commercial offerings that is definitely trained on cyber for sure. It's gaining that... You know, they train it on, you know, cyber ranges, CTF data, looking for, you know, how do these offensive people work as well as how's the defense work? What are they looking for? So it's absolutely being trained that way. It's not just the LLMs have magically figured out how to do offensive capabilities. They've been trained on it. They've been built better harnesses to make them more effective. And that's one of the key things we're actually finding are the harnesses that their actors are building to make this a lot more effective for them. They're learning as they're going and, you know, it's not hard to make changes to a harness day in and day out to make it more effective.
00:57:14 · Nathan Labenz, Michael Freeman
[00:57:14] S0: Earlier that morning, Prakash had cited Jason Clinton, Anthropics deputy chief information security officer, who talks about getting to vulnerability zero. Prakash put the idea to Freeman from the seat of a company like Nike with decades of legacy systems. How do you get there in two years? Freeman's answer, some of the most critical vulnerabilities come from business processes, and no amount of patching fixes those. He went on. [00:57:42] S5: So I think that's gonna be an area that, like I said, AI people just don't understand cybersecurity, and I think it's gonna be quickly here. We can find millions of vulnerabilities. We're gonna use AI to, you know, find the... You know, what you have in your environment. We're gonna patch it. And I've also heard proposals where you're running legacy systems where you can use AI to modernize that very quickly. And it's... All those are true to some extent, but they don't understand the business. They'd also don't understand, you know, the business processes that introduce the vulnerabilities are always gonna be there.
00:57:42 · Michael Freeman, Nathan Labenz
[00:57:42] S5: So I think that's gonna be an area that, like I said, AI people just don't understand cybersecurity, and I think it's gonna be quickly here. We can find millions of vulnerabilities. We're gonna use AI to, you know, find the... You know, what you have in your environment. We're gonna patch it. And I've also heard proposals where you're running legacy systems where you can use AI to modernize that very quickly. And it's... All those are true to some extent, but they don't understand the business. They'd also don't understand, you know, the business processes that introduce the vulnerabilities are always gonna be there. [00:58:16] S0: How do you feel about things like password managers in today's world? I don't have any particular one in mind. I've used LastPass. I've used 1Password. On the one hand, everybody seems to recommend them. On the other, it's like looking like an awfully juicy target these days. And, you know, it it seems like even, people who've tried pretty hard are still not that secure. So, like, would I be better off storing them all in Chrome or even just, you know, going back to, like, my old habits of sort of the base password and then some pattern for the extension of different sites? Like, what is actually the best move for me today? [00:58:58] S5: I really wish I could answer that question. I've not found a a solution that I really like at all. I I haven't. You know? I I wish I had an answer. [00:59:05] S0: So what do you do?
00:59:05 · Nathan Labenz, Michael Freeman
[00:59:05] S0: So what do you do? [00:59:08] S5: I personally have have it written down on a notebook. Amazing. Most things I can I can remember variations of it for your typical, hey, eBay or whatever the case may be? But it's banking sites, something like that. I have it written down in the book, look at it and say, okay. Things I won't go to often, but it's... You know, I'm gonna check on my stocks every quarter or something like that. That's written down on a a notebook. [00:59:32] S0: Part six, Le Chonk. Sagar Vaze leads the multimodal team at Mistral, which released Mistral Large four Le Chonk as open weights the day before the show. About a trillion parameters, roughly 50,000,000,000 active per token, trained on about 3,800 GPUs outside Paris. He put the training compute at 10 to the twenty fifth flops. In the s b ten forty seven fight, the threshold was 10 to the twenty sixth, and I'd have guessed a model like this would need 10 times more. I asked why it didn't. He said some labs do train at that scale, and this model still competes, and then gave the reason that matters.
01:00:09 · Sagar Vaze
[01:00:09] S4: The second is that, you know, this threshold slot budget does evolve over time because precraning does essentially become more efficient as we... As our level of scientific understanding increases. So a great example of this is sort of MOE architectures. I mean, you look, say, example, GPT-three, this was a dense model trained with three x the number of active parameters as the current Mistral Large four. Particularly, GPT three was trained with 175 b parameters total, but it was a dense model, so all parameters were activated for each token. Whereas ML four is a sparse mixture of experts, such that it's... There's one third the number of parameters activated per token, so only 50,000,000,000 parameters. But the total number of parameters is still 1,000,000,000,000. Your flop budget is proportional to the number of active parameters, so the number of, you know, parameters you're activating each token during training. And what this means is that you can train much larger models, much more capable models, but much less flops in total. So this figure that I referenced, so one e 25, this is in itself a moving target. So I wouldn't use that as an absolute target for a capable model throughout time. There would have been some one e 20 fives much less performant one or two years ago. And I would argue actually some larger models also much less performant today. But as pre training science progresses, as we learn how to make better use of the flops, this number itself will change. And at least what we're seeing is, at the moment, looking at the other open source labs, our understanding of, you know, what are the best models being created at the moment, training a 1,000,000,000,000 parameter model on one e 25 is sort of a rite of passage you have to go through in order to be able to to justify scaling even further.
01:01:56 · Nathan Labenz, Sagar Vaze
[01:01:56] S0: I asked how much uncertainty is left once a run that size is launched and whether they had ever aborted one. Halfway through this run, he said, they found something wrong with a big share of their compute already spent. [01:02:12] S4: Your other question was about, okay. So you do all this prep, and then, you know, what happened when you actually launched the model? And again, I think that every precraner will have, you know, the same stories to tell you, which is that precraning is kind of like you... You're you're doing all these scaling laws, you're sort sort of trying to align the tanker, and then you sort of let it go and you have to let it crane for however long you budgeted for it. And it's a huge amount of compute and capital expenditure in that run. And sometimes sometimes things were great, and the the loss goes down exactly as you expect it to. And sometimes there are surprises along the way and you have to be creative and find innovative solutions. So we had to figure out, okay, how can we fix it? And it was sort of all hands on deck, bunch of ablations, bunch of analysis of the model. And then we we found a solution. We found a way to be... We called it a... We found a way to patch it. We called it patch trial internally. And this it worked even better than we expected, and the the result is the model you see today. And, yeah, part of part of scaling compute is gonna be building the bones of how to iterate on this process more reliably.