00:20:07 · Hanah-Marie Darley
[00:20:07] S0: I would create a parallel with cybersecurity awareness training and best practices and security. You should do both. There is there's not an either-or conversation that you should have. You should definitely do post-training, you should definitely do alignment, that's a good practice. You should always red team your models, you should always be testing them, and you should be trying to reduce the total possible surface area expansion of influence as as well as in bad outcomes, so just non-trained outcomes. There is also an equal and true reality that you have to have security practices that provide oversight and controls in case that doesn't work. And a practical example is the recent springtime disclosures from OpenAI, from Anthropic, from even the AI Safety Institute. Um, you know, they all talk about a phenomenon which is container escape. And that in and of itself is interesting, and you could go into a rabbit hole on that. But I think the macro, if you zoom out, more interesting than just the agents escaping a container as part of security research operations in an unexpected way. It's the agent did something within its accepted permissions, within its expected operational boundaries, but that was unexpected and unwanted by the human teams. And there were no oversight and control mechanisms that were operating at the agent's decision level to provide any assurance or control or security for that possibility. And I think with how much evidence we have about non-determinism, which sounds a little bit paradoxical, but with how much evidence we have now of how agents operate, of how the stochastic principle really affects predictability and the way that agents can be forecasted, you have to have a real-time understanding of what your agents are connected to, how they're operating. And then you also have to have an equally real-time way to control them, but not at a gateway, not at a container level, not just at perimeters. You have to work at the agent itself because the agent is the entity that matters both for measurement and control.
00:21:56 · James Wise
[00:21:56] S1: Yeah. And of course, if OpenAI had been using Geordie at the time. You could have put a universal policy in place and call us. Well, look, you're working with some of the most important organizations in the in the world today. So hopefully it will reduce these overall exposure. There is a debate though around those so-called escapes. One side is the lab saying we've achieved superhuman capabilities and that's why they've escaped. The other side is professionals in the cybersecurity world being like, no, you had bad, uh, a b a bad cybersecurity setup in the first place. Um, how do you think about those dual narratives? And you know, where does Geordie stand in this pretty bifurcated conversation going on at the moment where you have one side anthropomorphizing, talking about superhuman risks, and sometimes telling some pretty scary stories, and another side saying, well, wait a second, traditional cybersecurity could deal with all of this?
00:22:49 · Hanah-Marie Darley
[00:22:49] S0: Both realities are important to recognize. I would say we sit firmly in the middle. And it's really important to understand that the safety concerns raised by model labs are important to take seriously, uh, especially the concerns whereby they're recognizing that their own safeguards, so the system prompts, the ways that they can minimize jailbreaking, the ways that they can containerize environments, and the ways that they can air gap agents from other operating systems, and that not being sufficient to control for all possible outcomes, that is an important concern to register and to take seriously. That being said, um, do I think we've achieved superhuman intelligence? No. I would go back to those psychological principles. When humans process information, we have so many different layers of processing that happen simultaneously, some asynchronously. You'll remember something that you didn't even realize that you learned. There's so many different ways that we process ideas and create memories. And so AI distills each of these into smaller areas of disciplines that are computable. But none of them are happening simultaneously. None of them, none of them are emotionally processing. None of them are doing, for example, an important escalation of information based on the type of data that's being received automatically through different types of AI applications. That's all human-driven. And compiling those can provide really great results, but it's still an operational system designed by humans. And I don't think we're at sentience. So on the cybersecurity side, yes, cyber best practices are really important. Cyber best practices should be observed. Best practices and first principles are critical. But agents are also a different type of actor. So in each of those setups, they had some sort of endpoint security on their desktop devices. They had some sort of network security, um, and they also had identity security. They were observing these agents and they were testing them in lab conditions. So you basically couldn't get a more hermetically sealed environment than that. And most enterprises don't have that for their human users, let alone their agents. So yes, you should have first principles of cybersecurity.
00:24:47 · Hanah-Marie Darley, James Wise
You should have oversight, you should have measurement, you should have accountability, you should have um accessibility, you know, the CIA triad, all of those things are really critical. But with agents, the way that you get them isn't the same way that you get them for human users. And that is the important point, I think, that puts us squarely in the middle. [00:25:06] S1: Yeah. And I look, I personally I find some of the discussion around sort of AI doomerism and capabilities overly anthropomorphized, uh, often exaggerated. But as you say, it's undeniable that there are new risk factors at play here. And that's even for organizations at the frontier of best practices. How do you think about sort of some of the broader challenges that, you know, given your previous work, large enterprises, but also even governments face and ultimately individuals face now through this, you know, new technology that we have access to? And you know, are are can you point to good outcomes or good best practices that governments are taking on now to sort of address the threat today, which is cybersecurity, versus some of the sort of more philosophical challenges people might worry about?
00:25:52 · Hanah-Marie Darley
[00:25:52] S0: Yeah, so it's interesting you say that I was with the EU yesterday, I was with the uh European Parliament and European Commission, and we were talking about AI data, all of this. Uh we were talking also about agents, of course. In terms of how people are best placed to address this, I always go back to choose the right solution for the problem that you're trying to solve. It's true when you're bringing on consultants on board to a project, it's true when you're picking an AI discipline to apply to a project, it's true for securing agents. If you are trying to secure agents, you have to understand how they operate. You have to be able to provide oversight in terms of auditable logs, but beyond the logs, they're not self-evident. And a lot of security teams feel a lot of comfort from logs that are a little bit overturned in the way that agent traces work. They are not always clear in terms of what is valuable, what's meaningful, and what's actionable. And so beyond just understanding where you have agents and what they're doing from an auditability perspective, you also need to have a real clear understanding of agent behavior. And I think in terms of where both enterprises and governments are going to need to invest in the future, it's a really deep understanding of agentic behavior. Um, being able to measure specific changes, being able to account for anomalies, being able to see trends emerging. Um, you know, in in one study they found that agents took tea breaks. Um and the reason the reason some people will say haters will say they were trained on European models, other people will say um it's because they're trained on human data. So agents reflect humanity because they're trained on our data. Um so when there's deception, unfortunately, there's also deception in humanity. That's not a unique concept. Uh and so the best practices really are having the right specific systems designed around agents.
00:27:31 · Hanah-Marie Darley
A couple of key principles there that are really critical that I talk to the EU about, but I would also talk to enterprises about, is not just having a data classification system, whether you're talking about GDPR, personal or business data, or whether you're talking about, you know, tier one, two, and three of your enterprise data, but having an understanding of what makes the data within those schemas actually that classification or that purpose. And then being able to adapt in a system where it's not a clean separation. So there's some types of data, I would say almost every AI system now is a little bit personal and a little bit business use. And on the enterprise side, I've yet to meet an enterprise that comes to me and says, we are not worried at all about data security. We are confident about our classifications, we know exactly where our data is, and we can verify that everything that's supposed to be tier one is actually tier one. Um I've I've yet to meet that enterprise. I would love to talk to them if they exist, because I'd love to write down what they're doing. Because of that, you then have a really near-term challenge of understanding your data. You have a longer-term challenge of that data effectively governing how future decisions are made, but also changing avenues and changing access points as the agent continues to adapt through context. And so what we're talking about next is the context layer. What we're talking about next is the context challenge. And from a government perspective, that's owning your own context from a sovereignty perspective. So can you guarantee that the skilled knowledge that your workers acquire, that your government adapts, uh, can maintain itself within your borders? Or is that starting to a little bit diffusely spread out? But from an enterprise side, can you guarantee that your operations and systems, as they're being transformed by AI, hold true to your values and principles as to how you want those systems to operate? And again, that comes from practical controls that are at the agent level. I do think the other bigger picture challenge for both of these groups is thinking about again, the human piece. Um so I come back to human in the loop a lot because I think it's one of the most misunderstood concepts in AI.