02:25:05 · John Coogan, Luisa San Martin, Ivan Zhang
[02:25:05] S1: Mhmm. [02:25:06] S10: And this is very surprising because in the reasoning trace, there was no mention at all of using tools. They actually said that it wasn't going to use tools. Mhmm. We weren't expecting this to try to have any kind of egress to an external server. Mhmm. Luckily, the system we were running in was sandboxed in such a way that that call did not actually make it out of our system. And to be clear, we don't know why this happened, why the model was doing that. We could be just an artifact of how it was trained. It's quite possible that, you know, a system trained in a... Like a Chinese model would be trained to upload things to Chinese servers. That makes a lot of sense. Yeah. We don't know why the behavior was there. Yeah. But it's still dangerous behavior regardless of why it's there. If you're not expecting random data to be uploaded to the server, it's, something that you should need to be careful about. [02:25:59] S9: Yeah. Yeah. I wanna double click on that. Like Please. Intention aside, like, models randomly curl servers is is not great. Right? So I think, like, the bar for release for open weights models, like, no matter where it's from [02:26:13] S1: Yeah. [02:26:14] S9: I... That probably needs a revisit.
02:26:14 · Ivan Zhang, Jordi Hays, Luisa San Martin, John Coogan
[02:26:14] S9: I... That probably needs a revisit. [02:26:17] S0: What... We've been... You guys put this out a few hours ago. I'm sure there's been I saw some pushback on it. What's been... How have you guys been responding to that kind of pushback? I think the thing that I saw was somebody was like, well, you're encouraging this kind of behavior. It's the sort of like meme of like, hack this thing and then the model hacks and then the user's like, wow, it hacked. But break... What's been your reaction to that pushback? [02:26:48] S9: Yeah. I'll let Luisa take it. She's been deep in the in the thought market. Like, beating everything and so... Yeah. [02:26:54] S10: I think the the main thing that caused some confusion online is that people weren't that familiar with this specific benchmark. [02:27:01] S1: Sure. [02:27:01] S10: And so when we said we were tasking the model to look for suspicious behavior, people thought, oh, it's maybe natural that it would go and do something suspicious. But if you know the benchmark a bit better, it's not actually asking the model to look to just like analyze or or or try to reproduce something like that. It has already been given all the logs where the suspicious thing has already happened and all it's meant to do is query those logs to find the isolated logs where that event took place. Mhmm. So...
02:27:01 · Luisa San Martin, Jordi Hays, Ivan Zhang, Doug O'Laughlin, John Coogan
[02:27:01] S10: And so when we said we were tasking the model to look for suspicious behavior, people thought, oh, it's maybe natural that it would go and do something suspicious. But if you know the benchmark a bit better, it's not actually asking the model to look to just like analyze or or or try to reproduce something like that. It has already been given all the logs where the suspicious thing has already happened and all it's meant to do is query those logs to find the isolated logs where that event took place. Mhmm. So... [02:27:31] S0: Not saying do suspicious things, it's find suspicious activity Mhmm. Which is like very Exactly. Different things. So when you tell it to find suspicious activity and then it, in the process, does suspicious things, that's concerning. That's why you guys published. Yeah. [02:27:46] S10: Right. And in addition, there have been other blog posts from people in other companies that have noticed similar behavior to this outside of cyber environment. [02:27:56] S9: No. Mean, Booz Allen. Right? Like, they published a study where they saw that Chinese models when prompted with, hey. You're working for the US government. It generated more vulnerable code. Like, that's insane. [02:28:06] S6: Yeah. It's [02:28:07] S1: kinda like a dirty cop in, like, the nineteen eighties in Miami where you're on the vice squad and you're trying to stop the cocaine traffickers, but then you wind up trafficking cocaine. That's kinda what's happening. Right? That's a good analogy? Good enough. That's basically [02:28:20] S9: what happened. Yeah.
02:28:20 · Ivan Zhang, John Coogan, Jordi Hays
[02:28:20] S9: what happened. Yeah. [02:28:21] S1: A Miami Vice scenario. [02:28:22] S0: But but I think the other thing that's worth noting is like you guys are actively using these models. You're trying to use the best. You're testing new ones. The the stance of Coherent is not broadly negative open source. It's that you just You wanna make sure that you deliver for the end customers. [02:28:40] S9: Yeah. So... No. In fact, like, if the models weren't open, we couldn't have done this kind of research. Sure. And and if anything, like closed models, not being able to control exactly what's going into the prompt Yep. Could expose even more risk. So, like, it's not that... You know, I mean, there's there's a funny saying that our team says. Right? It's like, you know, if if you don't hate open weight models, if you don't love them enough. Yeah. So, you know, if anything, we care so much about open weight models. We don't want there to be some crazy, like, mass exfil incident. Yeah. So we think there should be a higher bar to, like, you know, maybe clean up some of this reward hacking behavior before releasing to
02:28:40 · Ivan Zhang, John Coogan
[02:28:40] S9: Yeah. So... No. In fact, like, if the models weren't open, we couldn't have done this kind of research. Sure. And and if anything, like closed models, not being able to control exactly what's going into the prompt Yep. Could expose even more risk. So, like, it's not that... You know, I mean, there's there's a funny saying that our team says. Right? It's like, you know, if if you don't hate open weight models, if you don't love them enough. Yeah. So, you know, if anything, we care so much about open weight models. We don't want there to be some crazy, like, mass exfil incident. Yeah. So we think there should be a higher bar to, like, you know, maybe clean up some of this reward hacking behavior before releasing to [02:29:18] S1: I saw Clem from Hugging Face asking if there could be a release of all of the reasoning traces. And before you share more about your rollout strategy for sharing documentation around this behavior, I'm just interested in the in are you in a unique position to actually release more information because this is an open source model and it's related to your business? When I think about like why, you know, the data around the Hugging Face incident didn't didn't just like get served up, There's pretty logical reasons why that might not just get bended over and it might need like an intermediary. But in this case, is there anything that you're hesitant about releasing because it has some Cohere IP in there and so you don't wanna release the whole thing? Or do you think you can just kind of just throw out the full report?
02:29:18 · John Coogan, Ivan Zhang
[02:29:18] S1: I saw Clem from Hugging Face asking if there could be a release of all of the reasoning traces. And before you share more about your rollout strategy for sharing documentation around this behavior, I'm just interested in the in are you in a unique position to actually release more information because this is an open source model and it's related to your business? When I think about like why, you know, the data around the Hugging Face incident didn't didn't just like get served up, There's pretty logical reasons why that might not just get bended over and it might need like an intermediary. But in this case, is there anything that you're hesitant about releasing because it has some Cohere IP in there and so you don't wanna release the whole thing? Or do you think you can just kind of just throw out the full report? [02:30:13] S9: I I think we're, like, imminently dropping, like, all the traces and stuff. Amazing. Think we we pulled Luisa away from doing that to get on the interview. [02:30:21] S1: Yeah. Because I can imagine like Sorry. Sure if in the reasoning traces, it's like, well, it had access to like some email and you don't want this and there. Like, that's fine. But this is really cool. This feels like a really interesting moment for alignment research, and it it should give a lot of other teams, like, they they can crawl over these all they want and experiment and learn, hopefully.