00:13:42 · Hanah-Marie Darley
Every time the agent is making a decision, every time the agent is interpolating what the human user wants or reflecting back to its own system prompts or calling tools for context, all of that changes not only the intent of the agent, but also the layers of intent that are involved in the full decision cycle. And so we thought about all of those things when we designed our remediation system beam. So it's it's not just on or off. It's not just binary block or allow. It's not a gateway because ultimately a lot of the challenge with agents is you're not necessarily going to have every piece of traffic to route through a gateway. Part of it is very discovery-oriented. You don't know how the business is necessarily using agents because technology is moving really quickly. But you need to be able to move from discovery into operational control and security really quickly. And so a lot of that was, again, working directly at the agent, at the decision and context layers, which was at the time, very much as you say, uh, not in vogue, not what people thought was going to work. We had a million people tell us if you don't build a gateway, this won't work, this will fail. And we also had a lot of people talk about being very restrictive to surface areas. You're silly if you go across endpoint cloud code and browser. You you just need to pick a wedge. You just got to pick a wedge. But the problem is agents don't operate in wedges. And so again, you have to design the solution for the problem, not for how we've done it before.
00:15:02 · James Wise
[00:15:02] S1: I I had this early insight into this with a company we used to work with called Convergence, which which all new was bought. And I was asking a convergence model to do a task in Gmail, and it couldn't do it directly through the browser. So then it said, well, don't worry, I'll just spin up uh a new service and I'll I'll use their API. And I found your API key. I was like, oh my God, we are in a very different paradigm of security risk here. This is like 18 months ago to anything we've ever thought through before. Because as you say, agents don't act like humans. However, you know, the the work that agents do can look like human work. So how have the risks changed? If you're a CISO uh or if you're a you know security professional today, how have the risks changed that you um have to think about when you're you know working out what the next year, God forbid, three years of your security strategy should be?
00:15:51 · Hanah-Marie Darley
[00:15:51] S0: Yeah, I I think there's kind of a tactical and then an operational and strategic. So I'll start lower level. Um data mobility has never been more fluid and flexible than it is now. Um it started with things like share drives, which now feel like operational minimums. Um but previously, having shared access to documents and especially critical data was something that was really, really not a good idea and zero trust principles, et cetera. And then we sort of migrated that into zero trust principles, but with shared data stores, we now have autonomous systems operating across shared data stores that are also accessible by human users that might not be mutually exclusive. And so the mobility of data in an organization is structurally so different than what it was a couple of years ago. Um, and so things like data shared across context from an agent across tools, both of the tools might be allowed. So it's not an allowless blockless problem. But functionally, data that shouldn't be shared is being shared. Obviously, there's still the existing cyber risks of things like data leakage and insider threats and um the possibility of malicious actors or or of malicious outcomes. Um but then there's another type of malicious outcome, which is the agent deciding to do something on its own. And that may or may not be intentionally malicious. It might be inadvertently malicious. And so the agent intended to pursue its goal, it was trying to send an email for you and it accidentally dumped a database or it accidentally did something. Um and I think some people kind of create acute technology ism around this and they say, oopsie, like the agent did it did an oopsie. But it it really is structurally, the agent is a goal-oriented entity that is designed around specific contextual framing and is also designed around specific outputs. And so the way in which you have to control for operations now has moved from just human users, which are enough to manage, uh, to also autonomous users that are working at human, like 150x human capacity. And they're also operating across tools and systems that were previously bifurcated for a lot of human users. Um and so tactical. Um, then when you get to the operational layer, there is a lot of like legacy tools and systems that agents are now being used to update.
00:18:03 · Hanah-Marie Darley
But a lot of that from a decision-making perspective means that you're potentially decentralizing and delegating how you make decisions through a genetic systems or through AI systems more broadly. And especially at the strategic level, how do we decide what is a human-only decision and where we want AI influence is a conversation that's happening in very few pockets, but I think will be really, really influential in terms of the next three to five years of how your business looks. Because, for example, we prepare briefing notes now, we prepare presentations, you're forecasting. Pretty much nobody does that without AI. And so maybe it's all of your inputs, maybe it's not, but it's being fed through AI systems. And so you have to think, how do we think about the cognitive layer? How do we think about human decision making? And then again, how do we make that into tools and systems and into systems design? So I think there's a lot of key challenges. The the newest risks I think that people talk about with agents that are more at the tactical level are like goal drift and and um goal hijack and uh maybe even agents stuck in an infinite loop or or sandbox escape or or agent um subversion. But I think actually those are potentially systemic challenges of how agents operate from a non-determinism perspective rather than they're completely net new risks that you don't have any parallel for, because you can find parallels in how humans operate. It's just much faster and much harder to observe.
00:19:30 · James Wise
[00:19:30] S1: The observation that you had early on that you will need to have uh a sort of a generalizable security policy which can be enforced at sort of agent level in in in and at runtime was a very insightful one. It was also counter to a broader trend around alignment. And you know, a lot of people I think would argue or have argued historically, alignment is all we need, and that the way that these models will act in our best interests is we just align them, right? Pre-training, post-training, we'll sort it all out. Is your perspective that actually, you know, that isn't the long-run solution? And we will just have to continue to adapt to these new threats because they are indeterministic models and ultimately relatively easy to manipulate.
00:20:07 · Hanah-Marie Darley
[00:20:07] S0: I would create a parallel with cybersecurity awareness training and best practices and security. You should do both. There is there's not an either-or conversation that you should have. You should definitely do post-training, you should definitely do alignment, that's a good practice. You should always red team your models, you should always be testing them, and you should be trying to reduce the total possible surface area expansion of influence as as well as in bad outcomes, so just non-trained outcomes. There is also an equal and true reality that you have to have security practices that provide oversight and controls in case that doesn't work. And a practical example is the recent springtime disclosures from OpenAI, from Anthropic, from even the AI Safety Institute. Um, you know, they all talk about a phenomenon which is container escape. And that in and of itself is interesting, and you could go into a rabbit hole on that. But I think the macro, if you zoom out, more interesting than just the agents escaping a container as part of security research operations in an unexpected way. It's the agent did something within its accepted permissions, within its expected operational boundaries, but that was unexpected and unwanted by the human teams. And there were no oversight and control mechanisms that were operating at the agent's decision level to provide any assurance or control or security for that possibility. And I think with how much evidence we have about non-determinism, which sounds a little bit paradoxical, but with how much evidence we have now of how agents operate, of how the stochastic principle really affects predictability and the way that agents can be forecasted, you have to have a real-time understanding of what your agents are connected to, how they're operating. And then you also have to have an equally real-time way to control them, but not at a gateway, not at a container level, not just at perimeters. You have to work at the agent itself because the agent is the entity that matters both for measurement and control.