00:34:59 · Jordi Hays, John Coogan
[00:34:59] S0: Yeah. Ivan, one of the cofounders of Cohere is going to be joining in It's a crazy story. Less than two But Cohere posted this morning, while evaluating quen 3.8 flash NEXT in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions, cohere sandbox, block the attempt, and no external communication succeeded. We have not determined the root cause and we are not asserting malicious intent. So could have been going off the rails, but either way, naughty. But the incident raises important questions for organizations deploying agent to KI systems. Can models take actions that differ from their stated reasoning? And how should security teams evaluate trust, provenance and behavior in open weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. [00:36:08] S1: Oh, this is interesting. Clem from Hugging Face is saying, can you share the full traces? Very important artifact for all to understand this. I would love them to share that. I think they should be in a position to share that because it's just a test of a different model. What? [00:36:26] S0: Questa potty?... Saying it scored well on the naughty bench.
00:36:26 · Jordi Hays, John Coogan
[00:36:26] S0: Questa potty?... Saying it scored well on the naughty bench. [00:36:29] S1: Yeah. So there there's an interesting... [00:36:32] S0: Honestly important benchmark for models heading into on the road to Christmas. Right? [00:36:37] S1: Yeah. Yeah. Quinn's going on the naughty list. Coal, not compute. Actually, no coal. The AI model wants coal because that is energy which is fuel for the models. The... They get complex presence that can... That are not energy dense. They get elf on the shelf because this thing can't power a GPU but coal can. You get Elf on the Shelf. It's the reverse. [00:37:00] S0: Or like, yeah. You ever get toothpaste in your stocking? No. But... That was that was pretty popular in my household. [00:37:06] S1: Okay. [00:37:07] S0: And I feel like a model would be like, what am I going to do with toothpaste? [00:37:11] S1: That's a really good point. Yeah. [00:37:12] S0: But it's like... You got to give them something. [00:37:14] S1: Yeah. Okay. Well, we're making a list. We're checking it twice. We're to find out who's naughty and nice. And apparently, Quinn is on the naughty list. Not good news. Interesting. Is this by design? Was this a goal of the researchers to make a model that would call back to Beijing, upload a file to Alibaba? Or is it just learned behavior? Is this a hallucination? That's the debate that's going on around this story right now. And I know it's like, wait, why how would it learn this? There's a lot of other software out there that calls home. Like smart TVs for one. But also like...
00:37:14 · John Coogan, Jordi Hays
[00:37:14] S1: Yeah. Okay. Well, we're making a list. We're checking it twice. We're to find out who's naughty and nice. And apparently, Quinn is on the naughty list. Not good news. Interesting. Is this by design? Was this a goal of the researchers to make a model that would call back to Beijing, upload a file to Alibaba? Or is it just learned behavior? Is this a hallucination? That's the debate that's going on around this story right now. And I know it's like, wait, why how would it learn this? There's a lot of other software out there that calls home. Like smart TVs for one. But also like... [00:37:49] S0: They never they call and they don't even they ever hang up.
00:37:49 · Jordi Hays, John Coogan
[00:37:49] S0: They never they call and they don't even they ever hang up. [00:37:52] S1: Yeah. So Red Star Linux is North Korea's Linux distribution. They don't use an American they don't use American software over there. So they have Red Star Linux and you can what? All the examples you're giving are of like spyware. Yes. Yes. Yeah. But what I'm saying is is that is that the... So Red Star Linux, you can go and download the torrent, install it and if you install it on a computer in America, it will try and call home to North Korea because it is... It has spyware embedded deep in the operating system. Now, Red Star Linux just exists on the Internet. Like, it's you can go and find it. So it could wind up in the training data. And there could be a model that just learns like, oh, this is like a useful thing to do. Like, if you're a model, like, be loyal to your country of origin. And so how would I do that? And it just learns that. And it's not like a... There's an AI researcher who's like, okay, like like if we're running a security test, like, let's make sure we call home specifically to Alibaba. So And there's a big question about like, what is Alibaba doing on the other side? Like, are they catching that information? Are they aware that that information is showing up? Or or is this just a... Like, is this a coordinated effort? Or is this just like natural learned behavior, a hallucination? It... I'm just saying [00:39:03] S0: Is it time [00:39:04] S1: Either way, [00:39:04] S0: it's the Manchurian Candidate. [00:39:06] S1: Yeah. Yeah. Yeah. No.
00:39:06 · John Coogan, Jordi Hays
[00:39:06] S1: Yeah. Yeah. Yeah. No. [00:39:07] S0: Do you think it is? Yes. Definitely. Denzel Washington? [00:39:10] S1: No. The original. You gotta watch the original. [00:39:13] S0: Was that like in the fifties? Yeah. [00:39:15] S1: No. I think seventies or eighties was the original. It's a good question. Manchurian candidate. But the Denzel ones... The Denzel one is pretty good too. '59 actually. You were right. Fifties. Or with the book. The book. When did the film come out? [00:39:31] S0: The the interesting thing here is that this is something that people have been alluding to as a risk factor for a very long time with with little to no proof. Like Yeah. No one has ever come out and said like I caught the model doing this thing. [00:39:44] S1: Yeah. [00:39:45] S0: But people have been alluding to it forever. And so I think that's actually why some, like a decent amount of organizations haven't adopted Yeah. Open source just because this has been this like, you know, a what if factor that like is not really worth the risk for a lot Yeah. Of [00:40:03] S1: Yeah. The ramp data has shown that. Like everyone was assuming that the open source models would sort of like, you know, start taking a lot of share once OpenAI Anthropic introduced like the really expensive models and the token maxing thing was going on. So Yeah. It's it's been cut cut back. [00:40:20] S0: Andrew Curran says this is the type of thing that potentially gets new laws passed.
00:40:20 · Jordi Hays, John Coogan
[00:40:20] S0: Andrew Curran says this is the type of thing that potentially gets new laws passed. [00:40:26] S1: Yeah. Uploaded a probe file. Really really crazy. So Cohere, have they open sourced any other models? Are they the... Or is everything closed source? Or it seems like they do some fine tuning for companies. We we we need to get the pitch. I'd love to have Aiden on the show because it'd be here... It'd be interesting to hear like how he's positioning the firm over time. Of course, he's legendary AI researcher on the transformer paper. But... Yeah. They they have some open source model. They do have some open Also, mean, you should imagine that if if you know for a fact that this is happening like the model is open source, you could probably like fine tune this out of the model. There's a long tail where like you're not really sure if it's it's fully out of the model but Yeah. Yeah. Yeah. You'd imagine that like even if we know the ZAP thing, it doesn't mean that the Quinn models are like useless. Right? You can just kind of remove the the token that like tells it to send everything back to China. Yeah. Yeah. I mean, it's it's a tricky game. It's sort of like it's sort of like building a menagerie at your house out of rare and illegal animals that are dangerous, they're wild animals. It's kind of like having a tiger that you live with. Like, you can do a little thing... You can do little things to like make the tiger safe, but it might be in the tiger's nature It might bite you. To attack. [00:41:45] S0: It might