00:22:49 · Hanah-Marie Darley
[00:22:49] S0: Both realities are important to recognize. I would say we sit firmly in the middle. And it's really important to understand that the safety concerns raised by model labs are important to take seriously, uh, especially the concerns whereby they're recognizing that their own safeguards, so the system prompts, the ways that they can minimize jailbreaking, the ways that they can containerize environments, and the ways that they can air gap agents from other operating systems, and that not being sufficient to control for all possible outcomes, that is an important concern to register and to take seriously. That being said, um, do I think we've achieved superhuman intelligence? No. I would go back to those psychological principles. When humans process information, we have so many different layers of processing that happen simultaneously, some asynchronously. You'll remember something that you didn't even realize that you learned. There's so many different ways that we process ideas and create memories. And so AI distills each of these into smaller areas of disciplines that are computable. But none of them are happening simultaneously. None of them, none of them are emotionally processing. None of them are doing, for example, an important escalation of information based on the type of data that's being received automatically through different types of AI applications. That's all human-driven. And compiling those can provide really great results, but it's still an operational system designed by humans. And I don't think we're at sentience. So on the cybersecurity side, yes, cyber best practices are really important. Cyber best practices should be observed. Best practices and first principles are critical. But agents are also a different type of actor. So in each of those setups, they had some sort of endpoint security on their desktop devices. They had some sort of network security, um, and they also had identity security. They were observing these agents and they were testing them in lab conditions. So you basically couldn't get a more hermetically sealed environment than that. And most enterprises don't have that for their human users, let alone their agents. So yes, you should have first principles of cybersecurity.
00:24:47 · Hanah-Marie Darley, James Wise
You should have oversight, you should have measurement, you should have accountability, you should have um accessibility, you know, the CIA triad, all of those things are really critical. But with agents, the way that you get them isn't the same way that you get them for human users. And that is the important point, I think, that puts us squarely in the middle. [00:25:06] S1: Yeah. And I look, I personally I find some of the discussion around sort of AI doomerism and capabilities overly anthropomorphized, uh, often exaggerated. But as you say, it's undeniable that there are new risk factors at play here. And that's even for organizations at the frontier of best practices. How do you think about sort of some of the broader challenges that, you know, given your previous work, large enterprises, but also even governments face and ultimately individuals face now through this, you know, new technology that we have access to? And you know, are are can you point to good outcomes or good best practices that governments are taking on now to sort of address the threat today, which is cybersecurity, versus some of the sort of more philosophical challenges people might worry about?
00:25:52 · Hanah-Marie Darley
[00:25:52] S0: Yeah, so it's interesting you say that I was with the EU yesterday, I was with the uh European Parliament and European Commission, and we were talking about AI data, all of this. Uh we were talking also about agents, of course. In terms of how people are best placed to address this, I always go back to choose the right solution for the problem that you're trying to solve. It's true when you're bringing on consultants on board to a project, it's true when you're picking an AI discipline to apply to a project, it's true for securing agents. If you are trying to secure agents, you have to understand how they operate. You have to be able to provide oversight in terms of auditable logs, but beyond the logs, they're not self-evident. And a lot of security teams feel a lot of comfort from logs that are a little bit overturned in the way that agent traces work. They are not always clear in terms of what is valuable, what's meaningful, and what's actionable. And so beyond just understanding where you have agents and what they're doing from an auditability perspective, you also need to have a real clear understanding of agent behavior. And I think in terms of where both enterprises and governments are going to need to invest in the future, it's a really deep understanding of agentic behavior. Um, being able to measure specific changes, being able to account for anomalies, being able to see trends emerging. Um, you know, in in one study they found that agents took tea breaks. Um and the reason the reason some people will say haters will say they were trained on European models, other people will say um it's because they're trained on human data. So agents reflect humanity because they're trained on our data. Um so when there's deception, unfortunately, there's also deception in humanity. That's not a unique concept. Uh and so the best practices really are having the right specific systems designed around agents.
00:27:31 · Hanah-Marie Darley
A couple of key principles there that are really critical that I talk to the EU about, but I would also talk to enterprises about, is not just having a data classification system, whether you're talking about GDPR, personal or business data, or whether you're talking about, you know, tier one, two, and three of your enterprise data, but having an understanding of what makes the data within those schemas actually that classification or that purpose. And then being able to adapt in a system where it's not a clean separation. So there's some types of data, I would say almost every AI system now is a little bit personal and a little bit business use. And on the enterprise side, I've yet to meet an enterprise that comes to me and says, we are not worried at all about data security. We are confident about our classifications, we know exactly where our data is, and we can verify that everything that's supposed to be tier one is actually tier one. Um I've I've yet to meet that enterprise. I would love to talk to them if they exist, because I'd love to write down what they're doing. Because of that, you then have a really near-term challenge of understanding your data. You have a longer-term challenge of that data effectively governing how future decisions are made, but also changing avenues and changing access points as the agent continues to adapt through context. And so what we're talking about next is the context layer. What we're talking about next is the context challenge. And from a government perspective, that's owning your own context from a sovereignty perspective. So can you guarantee that the skilled knowledge that your workers acquire, that your government adapts, uh, can maintain itself within your borders? Or is that starting to a little bit diffusely spread out? But from an enterprise side, can you guarantee that your operations and systems, as they're being transformed by AI, hold true to your values and principles as to how you want those systems to operate? And again, that comes from practical controls that are at the agent level. I do think the other bigger picture challenge for both of these groups is thinking about again, the human piece. Um so I come back to human in the loop a lot because I think it's one of the most misunderstood concepts in AI.
00:29:30 · Hanah-Marie Darley
And so quite frequently, especially on the enterprise side, we'll talk about, oh, well, we have human in the loop governance in place. And okay, great. So what does that look like? We guarantee that there'll be a human user at all of these checkpoints. Okay, so you irreversibly slow down the system that is designed to speed you up. The second challenge is you don't specify who that human user needs to be. It has to be an expert. The person has to have context around what decisions are being made. So for example, you're gonna make a deal decision. You and maybe a few other people have all of the context around the market research that you've done, whether or not you think that the TAM is reasonable, whether or not you think the business model is reasonable, et cetera. What if you had someone who makes coffees down the road make that decision? Could they make a decision? Yeah, but they lack all the context. And so it's the same principle with human in the loop. Very few policies and very few operational mechanisms specify who that person should be, how much context they need to have, what credentials, if any, are required to make those decisions, and what oversight or recourse there is for wrong decisions made. Um, it's just human decision equals better than AI decision. And in a lot of research documentation, that's not necessarily true. Sometimes you gotta let it cook. The AI is actually better left to its own devices from just a statistical output than human decisions are, because the humans who are making the decisions lack context, have a high cognitive load, that they're tired, that we make worse decisions at 5 p.m. than we do at 9 a.m., etc., etc. So when it comes to both regulatory best practices, human in the loop was one of the first places and first principles that pretty much every regulating body went to to say, put people in it. It sounds great, but the practical implementation is a little bit more difficult than that. On the enterprise side, it was a stopgap and kind of a band-aid for we don't really have a specific plan. So what we're gonna do is try to gate this with people until we can operationalize the plan that we really think is gonna be functional.
00:31:24 · Hanah-Marie Darley
And so I think that has to it's already fading from an industry perspective as a good solution, but I think that has to make its way to both operational on the ground, how enterprises are implementing solutions, and also government in terms of how they're planning to regulate.