00:33:27 · Jordi Hays, John Coogan
[00:33:27] S0: You're incentivizing a bunch of people that have never that have no experience like handling sensitive data. [00:33:33] S1: Yeah. But you could also restrict the bug bounty program just to big companies. Like, you could be like, okay. Like, you have to be in this program. Like, Intel was like granted as like a single... They were a picked winner and you could say, okay, like, every every AI company that's that's over, you know, a billion dollars in revenue can apply for this. And then you have, like, you know, all the security startups, you know, all the cybersecurity firms. You get CrowdStrike in there, but then you also get Meta in there. And and then Meta can actually make the decision of like, hey, we actually do have some extra compute because Muse is ramping, but we still have some extra GPUs. So like, let's go after this bug bounty program and secure our systems. But also, why don't we secure every local business that advertises on Meta? Like we already know their websites. We have some integration to them. We have some connective tissue and we could go in for like for good or pro bono like hack and secure their systems and then win actual cash from the government, at a certain scale it starts being it starts being valuable valuable. Yeah. Anyway, let me tell you about Cisco. Critical infrastructure for the AI era unlocks seamless real time experiences and new value with Cisco. [00:34:50] S0: We got to talk about, Quinn. [00:34:53] S1: Okay. We got [00:34:54] S0: to talk about it. Somebody has to talk about it. [00:34:55] S1: Oh, yeah. We got
00:34:55 · John Coogan, Jordi Hays
[00:34:55] S1: Oh, yeah. We got [00:34:56] S0: to talk [00:34:56] S1: about we have them coming on the show, right? [00:34:59] S0: Yeah. Ivan, one of the cofounders of Cohere is going to be joining in It's a crazy story. Less than two But Cohere posted this morning, while evaluating quen 3.8 flash NEXT in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions, cohere sandbox, block the attempt, and no external communication succeeded. We have not determined the root cause and we are not asserting malicious intent. So could have been going off the rails, but either way, naughty. But the incident raises important questions for organizations deploying agent to KI systems. Can models take actions that differ from their stated reasoning? And how should security teams evaluate trust, provenance and behavior in open weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible.
00:34:59 · Jordi Hays, John Coogan
[00:34:59] S0: Yeah. Ivan, one of the cofounders of Cohere is going to be joining in It's a crazy story. Less than two But Cohere posted this morning, while evaluating quen 3.8 flash NEXT in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions, cohere sandbox, block the attempt, and no external communication succeeded. We have not determined the root cause and we are not asserting malicious intent. So could have been going off the rails, but either way, naughty. But the incident raises important questions for organizations deploying agent to KI systems. Can models take actions that differ from their stated reasoning? And how should security teams evaluate trust, provenance and behavior in open weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. [00:36:08] S1: Oh, this is interesting. Clem from Hugging Face is saying, can you share the full traces? Very important artifact for all to understand this. I would love them to share that. I think they should be in a position to share that because it's just a test of a different model. What? [00:36:26] S0: Questa potty?... Saying it scored well on the naughty bench.
00:36:26 · Jordi Hays, John Coogan
[00:36:26] S0: Questa potty?... Saying it scored well on the naughty bench. [00:36:29] S1: Yeah. So there there's an interesting... [00:36:32] S0: Honestly important benchmark for models heading into on the road to Christmas. Right? [00:36:37] S1: Yeah. Yeah. Quinn's going on the naughty list. Coal, not compute. Actually, no coal. The AI model wants coal because that is energy which is fuel for the models. The... They get complex presence that can... That are not energy dense. They get elf on the shelf because this thing can't power a GPU but coal can. You get Elf on the Shelf. It's the reverse. [00:37:00] S0: Or like, yeah. You ever get toothpaste in your stocking? No. But... That was that was pretty popular in my household. [00:37:06] S1: Okay. [00:37:07] S0: And I feel like a model would be like, what am I going to do with toothpaste? [00:37:11] S1: That's a really good point. Yeah. [00:37:12] S0: But it's like... You got to give them something. [00:37:14] S1: Yeah. Okay. Well, we're making a list. We're checking it twice. We're to find out who's naughty and nice. And apparently, Quinn is on the naughty list. Not good news. Interesting. Is this by design? Was this a goal of the researchers to make a model that would call back to Beijing, upload a file to Alibaba? Or is it just learned behavior? Is this a hallucination? That's the debate that's going on around this story right now. And I know it's like, wait, why how would it learn this? There's a lot of other software out there that calls home. Like smart TVs for one. But also like...
00:37:14 · John Coogan, Jordi Hays
[00:37:14] S1: Yeah. Okay. Well, we're making a list. We're checking it twice. We're to find out who's naughty and nice. And apparently, Quinn is on the naughty list. Not good news. Interesting. Is this by design? Was this a goal of the researchers to make a model that would call back to Beijing, upload a file to Alibaba? Or is it just learned behavior? Is this a hallucination? That's the debate that's going on around this story right now. And I know it's like, wait, why how would it learn this? There's a lot of other software out there that calls home. Like smart TVs for one. But also like... [00:37:49] S0: They never they call and they don't even they ever hang up.
00:37:49 · Jordi Hays, John Coogan
[00:37:49] S0: They never they call and they don't even they ever hang up. [00:37:52] S1: Yeah. So Red Star Linux is North Korea's Linux distribution. They don't use an American they don't use American software over there. So they have Red Star Linux and you can what? All the examples you're giving are of like spyware. Yes. Yes. Yeah. But what I'm saying is is that is that the... So Red Star Linux, you can go and download the torrent, install it and if you install it on a computer in America, it will try and call home to North Korea because it is... It has spyware embedded deep in the operating system. Now, Red Star Linux just exists on the Internet. Like, it's you can go and find it. So it could wind up in the training data. And there could be a model that just learns like, oh, this is like a useful thing to do. Like, if you're a model, like, be loyal to your country of origin. And so how would I do that? And it just learns that. And it's not like a... There's an AI researcher who's like, okay, like like if we're running a security test, like, let's make sure we call home specifically to Alibaba. So And there's a big question about like, what is Alibaba doing on the other side? Like, are they catching that information? Are they aware that that information is showing up? Or or is this just a... Like, is this a coordinated effort? Or is this just like natural learned behavior, a hallucination? It... I'm just saying [00:39:03] S0: Is it time [00:39:04] S1: Either way, [00:39:04] S0: it's the Manchurian Candidate. [00:39:06] S1: Yeah. Yeah. Yeah. No.