00:18:03 · Hanah-Marie Darley
But a lot of that from a decision-making perspective means that you're potentially decentralizing and delegating how you make decisions through a genetic systems or through AI systems more broadly. And especially at the strategic level, how do we decide what is a human-only decision and where we want AI influence is a conversation that's happening in very few pockets, but I think will be really, really influential in terms of the next three to five years of how your business looks. Because, for example, we prepare briefing notes now, we prepare presentations, you're forecasting. Pretty much nobody does that without AI. And so maybe it's all of your inputs, maybe it's not, but it's being fed through AI systems. And so you have to think, how do we think about the cognitive layer? How do we think about human decision making? And then again, how do we make that into tools and systems and into systems design? So I think there's a lot of key challenges. The the newest risks I think that people talk about with agents that are more at the tactical level are like goal drift and and um goal hijack and uh maybe even agents stuck in an infinite loop or or sandbox escape or or agent um subversion. But I think actually those are potentially systemic challenges of how agents operate from a non-determinism perspective rather than they're completely net new risks that you don't have any parallel for, because you can find parallels in how humans operate. It's just much faster and much harder to observe.
00:19:30 · James Wise
[00:19:30] S1: The observation that you had early on that you will need to have uh a sort of a generalizable security policy which can be enforced at sort of agent level in in in and at runtime was a very insightful one. It was also counter to a broader trend around alignment. And you know, a lot of people I think would argue or have argued historically, alignment is all we need, and that the way that these models will act in our best interests is we just align them, right? Pre-training, post-training, we'll sort it all out. Is your perspective that actually, you know, that isn't the long-run solution? And we will just have to continue to adapt to these new threats because they are indeterministic models and ultimately relatively easy to manipulate.
00:20:07 · Hanah-Marie Darley
[00:20:07] S0: I would create a parallel with cybersecurity awareness training and best practices and security. You should do both. There is there's not an either-or conversation that you should have. You should definitely do post-training, you should definitely do alignment, that's a good practice. You should always red team your models, you should always be testing them, and you should be trying to reduce the total possible surface area expansion of influence as as well as in bad outcomes, so just non-trained outcomes. There is also an equal and true reality that you have to have security practices that provide oversight and controls in case that doesn't work. And a practical example is the recent springtime disclosures from OpenAI, from Anthropic, from even the AI Safety Institute. Um, you know, they all talk about a phenomenon which is container escape. And that in and of itself is interesting, and you could go into a rabbit hole on that. But I think the macro, if you zoom out, more interesting than just the agents escaping a container as part of security research operations in an unexpected way. It's the agent did something within its accepted permissions, within its expected operational boundaries, but that was unexpected and unwanted by the human teams. And there were no oversight and control mechanisms that were operating at the agent's decision level to provide any assurance or control or security for that possibility. And I think with how much evidence we have about non-determinism, which sounds a little bit paradoxical, but with how much evidence we have now of how agents operate, of how the stochastic principle really affects predictability and the way that agents can be forecasted, you have to have a real-time understanding of what your agents are connected to, how they're operating. And then you also have to have an equally real-time way to control them, but not at a gateway, not at a container level, not just at perimeters. You have to work at the agent itself because the agent is the entity that matters both for measurement and control.
00:21:56 · James Wise
[00:21:56] S1: Yeah. And of course, if OpenAI had been using Geordie at the time. You could have put a universal policy in place and call us. Well, look, you're working with some of the most important organizations in the in the world today. So hopefully it will reduce these overall exposure. There is a debate though around those so-called escapes. One side is the lab saying we've achieved superhuman capabilities and that's why they've escaped. The other side is professionals in the cybersecurity world being like, no, you had bad, uh, a b a bad cybersecurity setup in the first place. Um, how do you think about those dual narratives? And you know, where does Geordie stand in this pretty bifurcated conversation going on at the moment where you have one side anthropomorphizing, talking about superhuman risks, and sometimes telling some pretty scary stories, and another side saying, well, wait a second, traditional cybersecurity could deal with all of this?
00:22:49 · Hanah-Marie Darley
[00:22:49] S0: Both realities are important to recognize. I would say we sit firmly in the middle. And it's really important to understand that the safety concerns raised by model labs are important to take seriously, uh, especially the concerns whereby they're recognizing that their own safeguards, so the system prompts, the ways that they can minimize jailbreaking, the ways that they can containerize environments, and the ways that they can air gap agents from other operating systems, and that not being sufficient to control for all possible outcomes, that is an important concern to register and to take seriously. That being said, um, do I think we've achieved superhuman intelligence? No. I would go back to those psychological principles. When humans process information, we have so many different layers of processing that happen simultaneously, some asynchronously. You'll remember something that you didn't even realize that you learned. There's so many different ways that we process ideas and create memories. And so AI distills each of these into smaller areas of disciplines that are computable. But none of them are happening simultaneously. None of them, none of them are emotionally processing. None of them are doing, for example, an important escalation of information based on the type of data that's being received automatically through different types of AI applications. That's all human-driven. And compiling those can provide really great results, but it's still an operational system designed by humans. And I don't think we're at sentience. So on the cybersecurity side, yes, cyber best practices are really important. Cyber best practices should be observed. Best practices and first principles are critical. But agents are also a different type of actor. So in each of those setups, they had some sort of endpoint security on their desktop devices. They had some sort of network security, um, and they also had identity security. They were observing these agents and they were testing them in lab conditions. So you basically couldn't get a more hermetically sealed environment than that. And most enterprises don't have that for their human users, let alone their agents. So yes, you should have first principles of cybersecurity.
00:24:47 · Hanah-Marie Darley, James Wise
You should have oversight, you should have measurement, you should have accountability, you should have um accessibility, you know, the CIA triad, all of those things are really critical. But with agents, the way that you get them isn't the same way that you get them for human users. And that is the important point, I think, that puts us squarely in the middle. [00:25:06] S1: Yeah. And I look, I personally I find some of the discussion around sort of AI doomerism and capabilities overly anthropomorphized, uh, often exaggerated. But as you say, it's undeniable that there are new risk factors at play here. And that's even for organizations at the frontier of best practices. How do you think about sort of some of the broader challenges that, you know, given your previous work, large enterprises, but also even governments face and ultimately individuals face now through this, you know, new technology that we have access to? And you know, are are can you point to good outcomes or good best practices that governments are taking on now to sort of address the threat today, which is cybersecurity, versus some of the sort of more philosophical challenges people might worry about?