02:27:01 · Luisa San Martin, Jordi Hays, Ivan Zhang, Doug O'Laughlin, John Coogan
[02:27:01] S10: And so when we said we were tasking the model to look for suspicious behavior, people thought, oh, it's maybe natural that it would go and do something suspicious. But if you know the benchmark a bit better, it's not actually asking the model to look to just like analyze or or or try to reproduce something like that. It has already been given all the logs where the suspicious thing has already happened and all it's meant to do is query those logs to find the isolated logs where that event took place. Mhmm. So... [02:27:31] S0: Not saying do suspicious things, it's find suspicious activity Mhmm. Which is like very Exactly. Different things. So when you tell it to find suspicious activity and then it, in the process, does suspicious things, that's concerning. That's why you guys published. Yeah. [02:27:46] S10: Right. And in addition, there have been other blog posts from people in other companies that have noticed similar behavior to this outside of cyber environment. [02:27:56] S9: No. Mean, Booz Allen. Right? Like, they published a study where they saw that Chinese models when prompted with, hey. You're working for the US government. It generated more vulnerable code. Like, that's insane. [02:28:06] S6: Yeah. It's [02:28:07] S1: kinda like a dirty cop in, like, the nineteen eighties in Miami where you're on the vice squad and you're trying to stop the cocaine traffickers, but then you wind up trafficking cocaine. That's kinda what's happening. Right? That's a good analogy? Good enough. That's basically [02:28:20] S9: what happened. Yeah.
02:28:20 · Ivan Zhang, John Coogan, Jordi Hays
[02:28:20] S9: what happened. Yeah. [02:28:21] S1: A Miami Vice scenario. [02:28:22] S0: But but I think the other thing that's worth noting is like you guys are actively using these models. You're trying to use the best. You're testing new ones. The the stance of Coherent is not broadly negative open source. It's that you just You wanna make sure that you deliver for the end customers. [02:28:40] S9: Yeah. So... No. In fact, like, if the models weren't open, we couldn't have done this kind of research. Sure. And and if anything, like closed models, not being able to control exactly what's going into the prompt Yep. Could expose even more risk. So, like, it's not that... You know, I mean, there's there's a funny saying that our team says. Right? It's like, you know, if if you don't hate open weight models, if you don't love them enough. Yeah. So, you know, if anything, we care so much about open weight models. We don't want there to be some crazy, like, mass exfil incident. Yeah. So we think there should be a higher bar to, like, you know, maybe clean up some of this reward hacking behavior before releasing to
02:28:40 · Ivan Zhang, John Coogan
[02:28:40] S9: Yeah. So... No. In fact, like, if the models weren't open, we couldn't have done this kind of research. Sure. And and if anything, like closed models, not being able to control exactly what's going into the prompt Yep. Could expose even more risk. So, like, it's not that... You know, I mean, there's there's a funny saying that our team says. Right? It's like, you know, if if you don't hate open weight models, if you don't love them enough. Yeah. So, you know, if anything, we care so much about open weight models. We don't want there to be some crazy, like, mass exfil incident. Yeah. So we think there should be a higher bar to, like, you know, maybe clean up some of this reward hacking behavior before releasing to [02:29:18] S1: I saw Clem from Hugging Face asking if there could be a release of all of the reasoning traces. And before you share more about your rollout strategy for sharing documentation around this behavior, I'm just interested in the in are you in a unique position to actually release more information because this is an open source model and it's related to your business? When I think about like why, you know, the data around the Hugging Face incident didn't didn't just like get served up, There's pretty logical reasons why that might not just get bended over and it might need like an intermediary. But in this case, is there anything that you're hesitant about releasing because it has some Cohere IP in there and so you don't wanna release the whole thing? Or do you think you can just kind of just throw out the full report?
02:29:18 · John Coogan, Ivan Zhang
[02:29:18] S1: I saw Clem from Hugging Face asking if there could be a release of all of the reasoning traces. And before you share more about your rollout strategy for sharing documentation around this behavior, I'm just interested in the in are you in a unique position to actually release more information because this is an open source model and it's related to your business? When I think about like why, you know, the data around the Hugging Face incident didn't didn't just like get served up, There's pretty logical reasons why that might not just get bended over and it might need like an intermediary. But in this case, is there anything that you're hesitant about releasing because it has some Cohere IP in there and so you don't wanna release the whole thing? Or do you think you can just kind of just throw out the full report? [02:30:13] S9: I I think we're, like, imminently dropping, like, all the traces and stuff. Amazing. Think we we pulled Luisa away from doing that to get on the interview. [02:30:21] S1: Yeah. Because I can imagine like Sorry. Sure if in the reasoning traces, it's like, well, it had access to like some email and you don't want this and there. Like, that's fine. But this is really cool. This feels like a really interesting moment for alignment research, and it it should give a lot of other teams, like, they they can crawl over these all they want and experiment and learn, hopefully.
02:30:21 · John Coogan, Ivan Zhang, Jordi Hays
[02:30:21] S1: Yeah. Because I can imagine like Sorry. Sure if in the reasoning traces, it's like, well, it had access to like some email and you don't want this and there. Like, that's fine. But this is really cool. This feels like a really interesting moment for alignment research, and it it should give a lot of other teams, like, they they can crawl over these all they want and experiment and learn, hopefully. [02:30:40] S9: Yeah. And what's interesting is, like, it's such a rare occurrence. Right? It's like Yeah. I think that happened one in 10,000 or a 100,000 times. Sure. But if you're running these systems at scale, like just takes one accidental exfil. [02:30:53] S1: Oh, that's interesting. [02:30:54] S9: Yeah. That's crazy. Know, you're sort of home. [02:30:57] S1: Yeah. Yeah. Yeah. Yeah. Yeah. Fascinating. Fascinating. Well, yeah. Thank you so much for coming. [02:31:02] S0: Yeah. Great great to have you guys on. Come back on if there's... Yeah. I'm sure there'll be follow ups and Yeah. You guys are always welcome. Yeah. Great to meet [02:31:09] S9: you both. For having us. [02:31:10] S1: Have a great rest of your day. [02:31:11] S0: Good luck with the release. [02:31:12] S1: Yeah. Good luck. Cheers, We'll talk to you soon. Goodbye. Yeah. That was really cool. I'm excited for more more of that and more people to kind of chew through it. Let me tell you about Shopify. Shopify is the commerce platform that grows new business and lets you sell in seconds online, in store, on mobile, on social, on marketplaces, and now with AI agents. Jordy, what else
02:31:12 · John Coogan, Jordi Hays
[02:31:12] S1: Yeah. Good luck. Cheers, We'll talk to you soon. Goodbye. Yeah. That was really cool. I'm excited for more more of that and more people to kind of chew through it. Let me tell you about Shopify. Shopify is the commerce platform that grows new business and lets you sell in seconds online, in store, on mobile, on social, on marketplaces, and now with AI agents. Jordy, what else [02:31:33] S0: is What else? [02:31:34] S1: On your mind? [02:31:35] S0: Do we got Brazil nuked. [02:31:38] S1: Nuked. [02:31:39] S0: It says 10 Stock is down 20%. Negative negative 26% for a country. ETF is sick. Never seen that before. Says taken. [02:31:50] S1: Sick in the bad sense. Not sick in like the... [02:31:53] S0: Yeah. Not in like the... [02:31:54] S1: Not in the skate rock world. [02:31:58] S0: You They what will invent faster than light travel and MFs will ask what the practical practical use use case case is.