"The Cognitive Revolution"
00:50:06 · Nathan Labenz
[00:50:06] S0: It it sounds more dystopian than utopian to think that, like, the bulk of human activity will go into status games, I have to say. I'm like a bear on employment, bullish on people's ability to find meaning in absence of employment, I think, well, certainly relative to most. I really think way too many people in the AI policy space forget or maybe never even realized that, like, a huge portion of people are working to live and not really tying up their identity or their sense of self worth with their work in the way that, you know, the the academic or, you know, intellectual class tends to. So I'm I'm more bullish that, like, people will find things to do that will make them happy. Part five, 5 to 10¢. Michael Freeman is senior director of threat research at ServiceNow. His team watches AI agents attack companies, and he has spent his career finding and exploiting vulnerabilities. I asked what he makes of the current moment.
00:51:16 · Michael Freeman
[00:51:16] S5: It's been a very fast, rapid development. Back in July or end of July, I was speaking to reporters about watching AI agents attack companies. And at the time, all other companies were telling these reporters, it looks just like script kiddies right now. It's just a bunch of noise. And I said, Well, it's evolving fast. Give it a couple of months and we're gonna see some very sophisticated attacks coming from these agents. And know, look in the past two weeks, that's exactly what we're seeing. We're seeing AI development for offensive capabilities that perform at a scale I would say of a very competent APT teams right now. It is, you know, still noisy. I won't say it's very sophisticated and covering its tracks, but I'll definitely say when you take a look at the way they work, they are very competent of adapting and learning the environment and the technologies very well When we take a look at what's happening, especially in Southeast Asia, with the agentic attacks that are happening there in the past two weeks, we were able to identify it costs nowhere between 5 to 10¢ to compromise a company now. So we're actually seeing AI agents more so different frameworks being developed rapidly in FORT and new capabilities being added very quickly. It looks like it's AI actually adding these new capabilities into these frameworks. So it's going from very targeted, smart web based attacks to compromising the environment, pivoting, looking for credentials and ticketing systems, things like that, once it identifies that, then does active directory attacks. So we have seen 3,000 to 4,000 on average attacks successfully in the past two weeks targeting a lot of Southeast Asian companies.
00:53:08 · Prakash Narayanan, Michael Freeman
[00:53:08] S6: And what would be the impact of these? I mean, Are these banks? Are these airlines? Are these manufacturers? What is the point of... Are people making money off of this? Is it just random mayhem? [00:53:21] S5: I would say it's all of the above right now. We're seeing a lot of it is just they're capable of compromising the environment. Some frameworks we've been able to get our hands on that threat actors are using, build certain capabilities to look for certain things, you know, IP, things like that. That has been a fork of a couple of different systems we've seen looking for that. A lot of it is just to see what it can do. Can it gain access to this environment? Can it persist? And so that's the new capability that we're seeing being developed, identifying ways that can take advantage of different configurations within active directory to build persistent mechanisms in place. So that's where we're seeing the evolution happening now. We have analyzed a fork of a really well known offensive AI framework last night where it was developing better capabilities of malware, more so ransomware based, using cryptography that's post quantum cryptography developed instead of the typical AES that you would typically see.
00:54:31 · Nathan Labenz
[00:54:31] S0: I asked Freeman about the training itself. I would I would say, like, sort of the anthropic take has been like, we didn't train this to be a cyber attacker. It just kind of got that way because it's so good at everything. Okay. That's one story. The CTO of Vercel, was a guest on the show here with us, and he said it's very clear when you look at the traces that they've been trained to be cyber attackers. His point of view was like, this is not the sort of behavior that you just get falling out of random, you know, application development. Where do you think we are in terms of, like, how much this has been intentional? And does it matter at this point given what exists, how intentional it was? I mean, it might matter if you could think maybe we can make some different decisions for the next generations of models, but obviously, to some degree, the the cat's out of the bag. What do you make of how we got here, and is there any opportunity to, like, reel it back in?
00:55:25 · Michael Freeman
[00:55:25] S5: Not reel it back in. No. But I definitely agree that it is being developed on offensive capabilities. Absolutely. Because it's not just the models. It's the harness itself also. So you take a look at when you develop the models, there's plenty of great open source models in the past three weeks that have came out that are specific to cybersecurity that are open source models. You can download the weights, can go to Hugging Face, Hugging Face, download these and run them on your local machines. And that's what a lot of these attackers are actually doing. And it's actually quite interesting to see as we start tracking back, getting attribution on these attackers, that all these are just local machines that they're running them on. We developed our own small language model two years ago to find vulnerabilities within the software. And so, if we can do it for a couple $100,000 two years ago on a small language model, it's pretty trivial to start reverse engineering, especially some of these commercial offerings that is definitely trained on cyber for sure. It's gaining that... You know, they train it on, you know, cyber ranges, CTF data, looking for, you know, how do these offensive people work as well as how's the defense work? What are they looking for? So it's absolutely being trained that way. It's not just the LLMs have magically figured out how to do offensive capabilities. They've been trained on it. They've been built better harnesses to make them more effective. And that's one of the key things we're actually finding are the harnesses that their actors are building to make this a lot more effective for them. They're learning as they're going and, you know, it's not hard to make changes to a harness day in and day out to make it more effective.
00:57:14 · Nathan Labenz, Michael Freeman
[00:57:14] S0: Earlier that morning, Prakash had cited Jason Clinton, Anthropics deputy chief information security officer, who talks about getting to vulnerability zero. Prakash put the idea to Freeman from the seat of a company like Nike with decades of legacy systems. How do you get there in two years? Freeman's answer, some of the most critical vulnerabilities come from business processes, and no amount of patching fixes those. He went on. [00:57:42] S5: So I think that's gonna be an area that, like I said, AI people just don't understand cybersecurity, and I think it's gonna be quickly here. We can find millions of vulnerabilities. We're gonna use AI to, you know, find the... You know, what you have in your environment. We're gonna patch it. And I've also heard proposals where you're running legacy systems where you can use AI to modernize that very quickly. And it's... All those are true to some extent, but they don't understand the business. They'd also don't understand, you know, the business processes that introduce the vulnerabilities are always gonna be there.