"The Cognitive Revolution"
01:06:46 · Nathan Labenz
[01:06:46] S0: Prakash's conclusion was that you can't carve a capability out of the latent space. My reply? I'm I'm kinda more bullish on some of these, like, not necessarily unlearning, but, you know, I I just recorded a podcast earlier today where I was talking a bit about how feasible it is. I mean, there's there's multiple levels of control, of course, but, like, there are efforts underway to basically lock up certain classes of biodata that are being newly generated now because the worry is that they'll create dangerous capabilities and models if they're they're used in training, and not necessarily to shut that out, you know, forever from everyone, but at least to have it on kind of a know your customer basis or whatever. It does seem that those... What what few experiments have been done on, like... And, you know, I'm thinking here of ESM and also some work, the EVO series of models that came out of ARC Institute and Stanford and whatever big collaborations, of course. It seemed like they showed that if you do withhold certain classes of data, you can get generally capable models that that still kind of don't know about these areas of, censorship. So I'm I'm not so sure that that can't work. Took that question to Sagar when he came on. First though, Mistral built Large four with cyber defense as a focus, and Prakash asked how that changed the training.
01:08:22 · Sagar Vaze
[01:08:22] S4: So so the cyber capabilities and agent decoding capabilities in general are something which are typically given to the model in the in the post training stage. So most of the stuff we've discussed today so far has been about pre training. So Right. Where a large percentage of the flops go at at the start. Right. And then in post training is where you say, okay, well, what are people going to really use this model for? And that's when, you know, you start giving it these these new capabilities of finance, law, defense, etcetera. Earlier in the year, there were there were some events where some very capable models had restricted access to both certain geographies as well as to certain enterprises. And at the same time, we were seeing that these very capable models had these incredible cyber capabilities, which often were being jailbroken by malicious actors in order to to infiltrate or expose vulnerabilities in in, you know, legitimate defenders. And an imbalance which we saw here is that these defenders were then trying to, you know, use these same models to patch their own vulnerabilities, but were finding that their requests were refused. Because in practice, for very good reasons, these closed labs want to make sure that the models are not misused. They've got classifiers under the hood to identify malicious cyber requests from a standard request. But unfortunately, in practice, if you're a defender and you're trying to shore up your infrastructure, your defensive request looks quite a lot like a malicious request because you have to try and find vulnerabilities in your own code base to patch them. And so we thought it was imperative for there to be an open weight cyber model of similar capabilities to these closed models such that, you know, enterprises could scalably defend themselves.
01:09:59 · Nathan Labenz
[01:09:59] S0: Prakash noted that people had been taking open weight models and obliterating them, stripping out their refusals to run cyberattacks. He asked whether Mistral Large Four ships with guardrails or comes pre obliterated.
01:10:15 · Sagar Vaze
[01:10:15] S4: Yes. So something... This is something we discussed in our blog post. So, you know, we believe in open ways because we think that it's a net benefit to society. We think that it's a net benefit to the ecosystem. And so we're always dynamic about exactly what... How to balance that net benefit. In practice, what's happened here is that, you know, we do understand that these models can be misused. So we do have a version of Mistral Large four, which has been released, which has been now assessed, which refuses the most dangerous cyber requests. And then we're working with trusted third parties... Sorry, trusted partners to give them access to the unobliterated model unobliterated model so that they can defend themselves. And, you know, these are people we vet and make sure that make sure that they're using it in ways we find responsible. But coming back to the the framing of the question, your framing of question was that, okay, there's this open source model, say, I mean, it could be Mistral's. It could be anyone else's. And there are some guardrails, and in principle, these guardrails can be overcome by, you know, malicious actors so that these capabilities can be unlocked. I think that's the sort of sentiment. But what I would highlight is that, you know, that anytime you have access to the model, this is in principle possible, right? Even if it's behind a closed source API, because you can jailbreak a model regardless of whether you have access to the weights or not. And so here I'd consider the counterfactual, right? Imagine if we were in a world in which there was only closed source models. Right? And these are... This jailbreaking still existed and there was still a way in which you could use them to to perpetrate malicious attacks. Where would we be? We really wouldn't... We would have no recourse because these same these same same these defensive requests are being blocked by these same companies. And if there was no open waste model, we would really have no scalable way to defend our infrastructure as these models become more and more capable, which we're seeing in each iteration. So we we understand the risks. We have a responsible perspective every time that we we open source a model. It's something that we consider very carefully.
01:12:22 · Sagar Vaze, Nathan Labenz
But we fundamentally believe in the open wage principle, we think that it applies in this case too, as hopefully illustrated by the examples I just gave. [01:12:33] S0: Then I asked about the approach I'd raised with Prakash. Localize the dangerous know how in a few of a mixture of experts, models, experts, put those behind know your customer checks, and release everything else. I asked whether he saw any promise in it. [01:12:46] S4: No. Absolutely. I think it's a scientific question just like all the other ones we answer when we're when we're creating the model. If one of the features we want to the model is that there's certain capabilities which are tamped down while all other capabilities are still state of the art, that is in itself a scientific problem. And yes, some of the methods you mentioned are interesting. We look into some things like that. It's an active research area, and this technology is evolving very, very fast, regardless of whether it's a closed weight model or an open weight model, and we need to be dynamic with respect to these things. So in the future, if there's some other risk, we will approach it accordingly, but we consider it a scientific problem problem solve to rather than the fundamental philosophical challenge to the way in which we we operate our, you know, our open weights business.
01:13:36 · Nathan Labenz
[01:13:36] S0: People like me who talk a lot about AI safety these days and are willing to, like, bite a fair amount of bullets to, you know, to try to keep things from going totally haywire on us. A large percentage of us are, like, longtime techno optimist libertarians who, you know, have previously said things like information wants to be free and, you know, are, like, very much sort of spiritually on the side of open source, which is why I end up looking for these, like, technological solutions as opposed to, you know, sort of policy or policing wherever wherever possible. Later in the closing conversation, after Sagar left, I came back to Mistral and open release. Yeah. Well, I I feel like I should say once more that while I do worry about all this getting out of hand, I do, at the same time, very much feel the concentration of power risk factor. And, you know, I I I don't take lightly at all the the potential problems there or the value that people are creating when they do create open source options for people to use. I really, really hope that we get some of these technical solutions to work well. The question of, like, just how much urgency they really feel about this kind of stuff, I hope it's high, because I do think we are gonna hit a point pretty soon where it is gonna become untenable to just release these things in, you know, just the sort of throw everything into the mix, pop out whatever pops out, throw that into the public. Jordan Schneider from ChinaTalk has has come around a lot on this with respect to China. I don't necessarily know exactly what his, feeling was previously, but recently, I've heard him say over and over again that, like, don't you think that before too long, the Chinese government is just going to allow this, like, willy nilly open sourcing anymore, and there's gonna have to be some, like, way to make a good case that this is actually gonna be okay? I think that is almost certainly coming and potentially quite soon. But on some timeline, and it's not a super long timeline, we've really got to sort that out. And without it, it seems like we're gonna end up not being able to have open source. I just... I... Governments are gonna have a real hard time tolerating that. Prakash's position was that you can't stop anyone from training a model or distributing the weights. So the only place to gate is deployment.