00:29:38 · Irene Solaiman, Rebecca Balan
[00:29:38] S2: I mean, the conversations have gotten to a point where a lot of folks are... We we are responding to our community. We are just seeing so much interest in obliterated models. This is... It's can of worms, really dicey topic, and they they do instill fear in a lot of folks. [00:29:55] S1: Can you just briefly explain to our listeners what an obliterated model is? [00:29:58] S2: Yeah. It sounds like such a funny word. Obliterated models are models that have had all of their guardrails removed. And already, some of these open frontier models are pretty permissive in their in their refusals and guardrails, but you... It can pretty easily just kinda fine tune that out and take those off and reupload them on Hugging Face. This is not a violation of our terms of service. And part of why we are keeping them up is we have so much feedback from cyber defenders and researchers on how this is their only option for either research or defense.
00:30:33 · Rebecca Balan
[00:30:33] S1: Yes. Yeah. I've I've also come around on obliterated models a little bit after talking to cyber people because, you know, I wrote about this. There's a company called Obliteration dot ai, I think it's what they're called. And and they basically just like host or will provide you with the ability to obliterate any model. So you just... There's no guardrails at all. And you can just imagine how horrible this is. Right? Like, you don't wanna make it easier for attackers. But at the same time, this makes it easier for defenders. And I guess, the argument is that the attackers are gonna obliterate... They're gonna do this anyway. It's important that the defenders have... Know how they're going to attack them, right, essentially, and are able to... It's really good for red teaming as well. You know, really helpful for, you know, when you're trying to do the most insane version of a hack that you're trying to test your systems on to have an obliterated model. Yeah, it's tough. But then if open model infrastructure becomes, you know, more costly and risky to maintain, who who, I wonder, would pay for the security? Like, would that potentially give large, well funded platforms an advantage over the smaller open source projects?
00:31:40 · Irene Solaiman, Rebecca Balan
[00:31:40] S2: I mean, there there's going to be just an advantage in the level of product support that you get from a closed weight lab. Most of these outside of the the really well resourced, like the Alibaba's, you know, we're still 200 people while we're undergoing this acquisition. So we're not gonna be able to provide that level of handholding. But for what folks need for for cyber defense at a time where there's a lot of... I'm I'm quite nervous about the state of the Internet. We we need to have some level of... This is a trade off. We need to have access to the folks who know what they're doing, who are getting refusals from the most frontier models. There's a dichotomy, unfortunately, between cybersecurity and AI safety folks, and both have well and good intentions. But a lot of security folks love open source because of the privacy, because of those on prem considerations, because of what they can do with it, and really have this ideological preference for open weeks in a way that is creating a new kind of open versus closed conversation. But the urgency from our perspective and from where I'm sitting at Hugging Face is that we just need to quit Defenders yesterday. And if that means giving folks a more permissive model, it seems that's the balance that we'll have to make. [00:33:05] S1: Is there anything that could convince you otherwise? Like, is there any type of attack that might happen that would make you think, actually, like, I don't want peep... Like, attackers to have access to, you know, powerful obliterated open models.
00:33:05 · Rebecca Balan, Irene Solaiman
[00:33:05] S1: Is there anything that could convince you otherwise? Like, is there any type of attack that might happen that would make you think, actually, like, I don't want peep... Like, attackers to have access to, you know, powerful obliterated open models. [00:33:20] S2: I mean, this is the visibility of who's doing the attacks and who's doing the defense. I wish that we had better ways. This is what we've asked for policy wise is better incident reporting and standards. We're so glad that we were able to be as explicit as possible, but it was quite difficult to figure out how to report it and to whom. So as folks both on the the frontier side but also who are doing defense, who are victims of attacks, are undergoing this, I would ask that they be explicit as possible in sharing openly what they're undergoing, if they have an understanding of what model was being used. To your explicit question of what would change my mind, I think it is a clear clear data that the risks of open weights being leveraged in attacks outweighs the possibility for defense. And a lot of this is compute constrained. I don't think there is enough compute available actor to do the level of large scale at this point, which could change in the near future. But in our level of urgency, the need for defense, for even if you've seen these European grocery stores, it might not be the highest priority for critical infrastructure, but for businesses to defend hundreds of thousands, if not millions, in damages seems important.
00:33:20 · Irene Solaiman, Rebecca Balan
[00:33:20] S2: I mean, this is the visibility of who's doing the attacks and who's doing the defense. I wish that we had better ways. This is what we've asked for policy wise is better incident reporting and standards. We're so glad that we were able to be as explicit as possible, but it was quite difficult to figure out how to report it and to whom. So as folks both on the the frontier side but also who are doing defense, who are victims of attacks, are undergoing this, I would ask that they be explicit as possible in sharing openly what they're undergoing, if they have an understanding of what model was being used. To your explicit question of what would change my mind, I think it is a clear clear data that the risks of open weights being leveraged in attacks outweighs the possibility for defense. And a lot of this is compute constrained. I don't think there is enough compute available actor to do the level of large scale at this point, which could change in the near future. But in our level of urgency, the need for defense, for even if you've seen these European grocery stores, it might not be the highest priority for critical infrastructure, but for businesses to defend hundreds of thousands, if not millions, in damages seems important. [00:34:39] S1: Would you support some kind of a a policy that restricted... I mean, if not restricting access to models, restricting access to compute? Compute?
00:34:39 · Rebecca Balan, Irene Solaiman
[00:34:39] S1: Would you support some kind of a a policy that restricted... I mean, if not restricting access to models, restricting access to compute? Compute? [00:34:45] S2: I mean, I think this is a... The kind of... We're playing catch up here. Token efficiency and compute efficiency have made leaps and bounds in the last few years, particularly since DeepSeek R1. I don't think that's the right metric. The general guidance that we need is around what is the appropriate level of access control, because I fear that we're gonna start putting some level of access control on open weight models, and then nobody has access to anything. [00:35:12] S1: Yeah. Okay. You know, a lot of startups are building on open models, open source infra. We touched on this a bit last week when we had Wiz on. A decent amount of them don't have dedicated security teams. I just wanna get your quick take on this before we close out. How can small startups protect themselves now? Like, what are they underestimating perhaps about the current threat landscape?