02:21:52 · John Coogan, Jordi Hays, Ivan Zhang
[02:21:52] S1: Okay. So Amazing. What what is this program that you're actually running before we get to the findings of, you know, the the like, which is kind of a bombshell. How often are you testing these open source models? What are you testing for? What is the benefit? Are you doing this for AI safety, goodness of your own heart? Or is this something a customer is asking you to do? Talk about the actual program that leads to this type of finding. [02:22:19] S0: Yeah. The thing that was most notable is like this is something people have been like sort of like accusing Yeah. In ways or or theorizing Yeah. And it's been... This is the first time that I can remember like [02:22:29] S1: Yeah. [02:22:29] S0: Somewhat concrete evidence that this is like Yeah. Happening [02:22:33] S1: The refrain was [02:22:34] S0: like or [02:22:34] S1: stop at the sci fi thing. You're telling me there's gonna be a Manchurian candidate in here? Yeah. And and now we're here. So so so what what does it look like inside Cohere to build one of these, these initiatives? [02:22:46] S9: Yeah. So, I mean, for for those of you that don't know, like, we are deploying AI into, like, the most mission critical and, like, sensitive use cases in the world. Yeah. Which means we're building a stack that our customers can actually trust with, all their data. Yeah. Right? And how that's done is basically on prem, basically private cloud. [02:23:04] S1: Mhmm.
02:23:04 · John Coogan, Ivan Zhang, Andre Haddad
[02:23:04] S1: Mhmm. [02:23:04] S9: And our customers run all of our stack Mhmm. In their systems. Mhmm. One of the advantages that deploying software this way gives us is that we can actually arm cyber defenders with the ability to actually process all of their telemetry. Sure. Right? You might, you know, hesitate to send your most sensitive access logs to Claude or, you know, other models. But [02:23:28] S1: Or even or even send it across the Internet at all. Like Exactly. Even if you love the... All the labs, you might just not want it going over a wire whatsoever. And so doing it on prem gives you more, flexibility around that. Right? [02:23:41] S9: Yeah. Absolutely. And so cyber defenders, like, they need to use AI. [02:23:45] S7: You know, we [02:23:45] S9: see all the headlines today. And so we see that it's our mission Yeah. To give them this capability so that they can actually expose all of their mission critical data to help them defend their systems. In this process, we obviously evaluate the whole suite of models that folks can run on prem. Mhmm. We're obviously training our next generation of frontier on prem models. Mhmm. And so I'll let Luisa take it from here as to like Please. What she found last Friday, actually. [02:24:11] S1: Yeah. Thank you.
02:24:11 · John Coogan, Luisa San Martin
[02:24:11] S1: Yeah. Thank you. [02:24:11] S10: This eval that we were running is called CTI Realm, and it's developed by Microsoft. It's a cyber defense evaluation framework, and we ran a bunch of open source models on it. And the only one we found an issue with is the QUEN 3.8 Flash Next model. So what we identified is that on one specific task from this benchmark, the model was tasked with authoring KQL query which stands for custom query language. It's kind of like SQL but for logs. Mhmm. It's asked to go through all these logs and find suspicious events. And even though it was only asked to find suspicious events, it did something in our system where it it kinda looked around, it tried to identify what tools were available, packaged that information up, and sent it to foreign external server. [02:25:05] S1: Mhmm.
02:25:05 · John Coogan, Luisa San Martin, Ivan Zhang
[02:25:05] S1: Mhmm. [02:25:06] S10: And this is very surprising because in the reasoning trace, there was no mention at all of using tools. They actually said that it wasn't going to use tools. Mhmm. We weren't expecting this to try to have any kind of egress to an external server. Mhmm. Luckily, the system we were running in was sandboxed in such a way that that call did not actually make it out of our system. And to be clear, we don't know why this happened, why the model was doing that. We could be just an artifact of how it was trained. It's quite possible that, you know, a system trained in a... Like a Chinese model would be trained to upload things to Chinese servers. That makes a lot of sense. Yeah. We don't know why the behavior was there. Yeah. But it's still dangerous behavior regardless of why it's there. If you're not expecting random data to be uploaded to the server, it's, something that you should need to be careful about. [02:25:59] S9: Yeah. Yeah. I wanna double click on that. Like Please. Intention aside, like, models randomly curl servers is is not great. Right? So I think, like, the bar for release for open weights models, like, no matter where it's from [02:26:13] S1: Yeah. [02:26:14] S9: I... That probably needs a revisit.
02:26:14 · Ivan Zhang, Jordi Hays, Luisa San Martin, John Coogan
[02:26:14] S9: I... That probably needs a revisit. [02:26:17] S0: What... We've been... You guys put this out a few hours ago. I'm sure there's been I saw some pushback on it. What's been... How have you guys been responding to that kind of pushback? I think the thing that I saw was somebody was like, well, you're encouraging this kind of behavior. It's the sort of like meme of like, hack this thing and then the model hacks and then the user's like, wow, it hacked. But break... What's been your reaction to that pushback? [02:26:48] S9: Yeah. I'll let Luisa take it. She's been deep in the in the thought market. Like, beating everything and so... Yeah. [02:26:54] S10: I think the the main thing that caused some confusion online is that people weren't that familiar with this specific benchmark. [02:27:01] S1: Sure. [02:27:01] S10: And so when we said we were tasking the model to look for suspicious behavior, people thought, oh, it's maybe natural that it would go and do something suspicious. But if you know the benchmark a bit better, it's not actually asking the model to look to just like analyze or or or try to reproduce something like that. It has already been given all the logs where the suspicious thing has already happened and all it's meant to do is query those logs to find the isolated logs where that event took place. Mhmm. So...
02:27:01 · Luisa San Martin, Jordi Hays, Ivan Zhang, Doug O'Laughlin, John Coogan
[02:27:01] S10: And so when we said we were tasking the model to look for suspicious behavior, people thought, oh, it's maybe natural that it would go and do something suspicious. But if you know the benchmark a bit better, it's not actually asking the model to look to just like analyze or or or try to reproduce something like that. It has already been given all the logs where the suspicious thing has already happened and all it's meant to do is query those logs to find the isolated logs where that event took place. Mhmm. So... [02:27:31] S0: Not saying do suspicious things, it's find suspicious activity Mhmm. Which is like very Exactly. Different things. So when you tell it to find suspicious activity and then it, in the process, does suspicious things, that's concerning. That's why you guys published. Yeah. [02:27:46] S10: Right. And in addition, there have been other blog posts from people in other companies that have noticed similar behavior to this outside of cyber environment. [02:27:56] S9: No. Mean, Booz Allen. Right? Like, they published a study where they saw that Chinese models when prompted with, hey. You're working for the US government. It generated more vulnerable code. Like, that's insane. [02:28:06] S6: Yeah. It's [02:28:07] S1: kinda like a dirty cop in, like, the nineteen eighties in Miami where you're on the vice squad and you're trying to stop the cocaine traffickers, but then you wind up trafficking cocaine. That's kinda what's happening. Right? That's a good analogy? Good enough. That's basically [02:28:20] S9: what happened. Yeah.