"The Cognitive Revolution"
00:53:08 · Prakash Narayanan, Michael Freeman
[00:53:08] S6: And what would be the impact of these? I mean, Are these banks? Are these airlines? Are these manufacturers? What is the point of... Are people making money off of this? Is it just random mayhem? [00:53:21] S5: I would say it's all of the above right now. We're seeing a lot of it is just they're capable of compromising the environment. Some frameworks we've been able to get our hands on that threat actors are using, build certain capabilities to look for certain things, you know, IP, things like that. That has been a fork of a couple of different systems we've seen looking for that. A lot of it is just to see what it can do. Can it gain access to this environment? Can it persist? And so that's the new capability that we're seeing being developed, identifying ways that can take advantage of different configurations within active directory to build persistent mechanisms in place. So that's where we're seeing the evolution happening now. We have analyzed a fork of a really well known offensive AI framework last night where it was developing better capabilities of malware, more so ransomware based, using cryptography that's post quantum cryptography developed instead of the typical AES that you would typically see.
00:54:31 · Nathan Labenz
[00:54:31] S0: I asked Freeman about the training itself. I would I would say, like, sort of the anthropic take has been like, we didn't train this to be a cyber attacker. It just kind of got that way because it's so good at everything. Okay. That's one story. The CTO of Vercel, was a guest on the show here with us, and he said it's very clear when you look at the traces that they've been trained to be cyber attackers. His point of view was like, this is not the sort of behavior that you just get falling out of random, you know, application development. Where do you think we are in terms of, like, how much this has been intentional? And does it matter at this point given what exists, how intentional it was? I mean, it might matter if you could think maybe we can make some different decisions for the next generations of models, but obviously, to some degree, the the cat's out of the bag. What do you make of how we got here, and is there any opportunity to, like, reel it back in?
00:55:25 · Michael Freeman
[00:55:25] S5: Not reel it back in. No. But I definitely agree that it is being developed on offensive capabilities. Absolutely. Because it's not just the models. It's the harness itself also. So you take a look at when you develop the models, there's plenty of great open source models in the past three weeks that have came out that are specific to cybersecurity that are open source models. You can download the weights, can go to Hugging Face, Hugging Face, download these and run them on your local machines. And that's what a lot of these attackers are actually doing. And it's actually quite interesting to see as we start tracking back, getting attribution on these attackers, that all these are just local machines that they're running them on. We developed our own small language model two years ago to find vulnerabilities within the software. And so, if we can do it for a couple $100,000 two years ago on a small language model, it's pretty trivial to start reverse engineering, especially some of these commercial offerings that is definitely trained on cyber for sure. It's gaining that... You know, they train it on, you know, cyber ranges, CTF data, looking for, you know, how do these offensive people work as well as how's the defense work? What are they looking for? So it's absolutely being trained that way. It's not just the LLMs have magically figured out how to do offensive capabilities. They've been trained on it. They've been built better harnesses to make them more effective. And that's one of the key things we're actually finding are the harnesses that their actors are building to make this a lot more effective for them. They're learning as they're going and, you know, it's not hard to make changes to a harness day in and day out to make it more effective.
00:57:14 · Nathan Labenz, Michael Freeman
[00:57:14] S0: Earlier that morning, Prakash had cited Jason Clinton, Anthropics deputy chief information security officer, who talks about getting to vulnerability zero. Prakash put the idea to Freeman from the seat of a company like Nike with decades of legacy systems. How do you get there in two years? Freeman's answer, some of the most critical vulnerabilities come from business processes, and no amount of patching fixes those. He went on. [00:57:42] S5: So I think that's gonna be an area that, like I said, AI people just don't understand cybersecurity, and I think it's gonna be quickly here. We can find millions of vulnerabilities. We're gonna use AI to, you know, find the... You know, what you have in your environment. We're gonna patch it. And I've also heard proposals where you're running legacy systems where you can use AI to modernize that very quickly. And it's... All those are true to some extent, but they don't understand the business. They'd also don't understand, you know, the business processes that introduce the vulnerabilities are always gonna be there.
00:57:42 · Michael Freeman, Nathan Labenz
[00:57:42] S5: So I think that's gonna be an area that, like I said, AI people just don't understand cybersecurity, and I think it's gonna be quickly here. We can find millions of vulnerabilities. We're gonna use AI to, you know, find the... You know, what you have in your environment. We're gonna patch it. And I've also heard proposals where you're running legacy systems where you can use AI to modernize that very quickly. And it's... All those are true to some extent, but they don't understand the business. They'd also don't understand, you know, the business processes that introduce the vulnerabilities are always gonna be there. [00:58:16] S0: How do you feel about things like password managers in today's world? I don't have any particular one in mind. I've used LastPass. I've used 1Password. On the one hand, everybody seems to recommend them. On the other, it's like looking like an awfully juicy target these days. And, you know, it it seems like even, people who've tried pretty hard are still not that secure. So, like, would I be better off storing them all in Chrome or even just, you know, going back to, like, my old habits of sort of the base password and then some pattern for the extension of different sites? Like, what is actually the best move for me today? [00:58:58] S5: I really wish I could answer that question. I've not found a a solution that I really like at all. I I haven't. You know? I I wish I had an answer. [00:59:05] S0: So what do you do?
00:59:05 · Nathan Labenz, Michael Freeman
[00:59:05] S0: So what do you do? [00:59:08] S5: I personally have have it written down on a notebook. Amazing. Most things I can I can remember variations of it for your typical, hey, eBay or whatever the case may be? But it's banking sites, something like that. I have it written down in the book, look at it and say, okay. Things I won't go to often, but it's... You know, I'm gonna check on my stocks every quarter or something like that. That's written down on a a notebook. [00:59:32] S0: Part six, Le Chonk. Sagar Vaze leads the multimodal team at Mistral, which released Mistral Large four Le Chonk as open weights the day before the show. About a trillion parameters, roughly 50,000,000,000 active per token, trained on about 3,800 GPUs outside Paris. He put the training compute at 10 to the twenty fifth flops. In the s b ten forty seven fight, the threshold was 10 to the twenty sixth, and I'd have guessed a model like this would need 10 times more. I asked why it didn't. He said some labs do train at that scale, and this model still competes, and then gave the reason that matters.