"The Cognitive Revolution"
01:12:22 · Sagar Vaze, Nathan Labenz
But we fundamentally believe in the open wage principle, we think that it applies in this case too, as hopefully illustrated by the examples I just gave. [01:12:33] S0: Then I asked about the approach I'd raised with Prakash. Localize the dangerous know how in a few of a mixture of experts, models, experts, put those behind know your customer checks, and release everything else. I asked whether he saw any promise in it. [01:12:46] S4: No. Absolutely. I think it's a scientific question just like all the other ones we answer when we're when we're creating the model. If one of the features we want to the model is that there's certain capabilities which are tamped down while all other capabilities are still state of the art, that is in itself a scientific problem. And yes, some of the methods you mentioned are interesting. We look into some things like that. It's an active research area, and this technology is evolving very, very fast, regardless of whether it's a closed weight model or an open weight model, and we need to be dynamic with respect to these things. So in the future, if there's some other risk, we will approach it accordingly, but we consider it a scientific problem problem solve to rather than the fundamental philosophical challenge to the way in which we we operate our, you know, our open weights business.
01:13:36 · Nathan Labenz
[01:13:36] S0: People like me who talk a lot about AI safety these days and are willing to, like, bite a fair amount of bullets to, you know, to try to keep things from going totally haywire on us. A large percentage of us are, like, longtime techno optimist libertarians who, you know, have previously said things like information wants to be free and, you know, are, like, very much sort of spiritually on the side of open source, which is why I end up looking for these, like, technological solutions as opposed to, you know, sort of policy or policing wherever wherever possible. Later in the closing conversation, after Sagar left, I came back to Mistral and open release. Yeah. Well, I I feel like I should say once more that while I do worry about all this getting out of hand, I do, at the same time, very much feel the concentration of power risk factor. And, you know, I I I don't take lightly at all the the potential problems there or the value that people are creating when they do create open source options for people to use. I really, really hope that we get some of these technical solutions to work well. The question of, like, just how much urgency they really feel about this kind of stuff, I hope it's high, because I do think we are gonna hit a point pretty soon where it is gonna become untenable to just release these things in, you know, just the sort of throw everything into the mix, pop out whatever pops out, throw that into the public. Jordan Schneider from ChinaTalk has has come around a lot on this with respect to China. I don't necessarily know exactly what his, feeling was previously, but recently, I've heard him say over and over again that, like, don't you think that before too long, the Chinese government is just going to allow this, like, willy nilly open sourcing anymore, and there's gonna have to be some, like, way to make a good case that this is actually gonna be okay? I think that is almost certainly coming and potentially quite soon. But on some timeline, and it's not a super long timeline, we've really got to sort that out. And without it, it seems like we're gonna end up not being able to have open source. I just... I... Governments are gonna have a real hard time tolerating that. Prakash's position was that you can't stop anyone from training a model or distributing the weights. So the only place to gate is deployment.
01:16:05 · Nathan Labenz
He asked how withholding would actually work. I haven't seen really anything to date that has shown a unlearning, you know, in a really profound or successful way or that refusals can be made super reliable or that, like, you know, tamper proofing can really work. If you're actually, you know, giving people the weights and they, you know, can obviously modify the weights however they want. So that has not seemed super promising to me, although, you know, I'm always eager to hear about a breakthrough. It feels to me like that is quite different though from a scheme where the model just doesn't learn certain things in the first place. And I really think we could use a ton more science around that. Whether it is localizing knowledge to to particular experts or pretraining filtering, there's, of course, the galaxy brain take that, like, you know, you have this whole world model, and then you have, like, one little gap in it, and you'll be able to fill in those gaps. I think there's, like, probably some truth to that, but there's some gap clearly that, like, would be big enough that you would not be able to fill in the dangerous parts with the high resolution, you know, that you would need to actually do bad things in the world. This is the sort of thing that I think that excess compute, you know, could really, be very productively used for at the frontier companies. It would be amazing for them to say, like, here's kind of how big of a knowledge gap you have to leave in a model for that to be, like, meaningfully hard, you know, for the... The model's not just gonna, you know, fill it in via insight. And, also, like, you're not just gonna be able to do, like, a couple, you know, marginal documents worth of training and kind of fill in that that gap in the way that we do see refusal being, like, removed at, you know, generally speaking, like a pretty small, like, almost like vanishingly small amount of additional training. So I would love to understand that stuff better. Yes. Like, really determined people can set up their own cluster, collect their own bio dataset, and, you know, do the additional training on top of the open weight model to create the dangerous thing. And, like, probably at some point that happens, but that's, like, more than just needing a few GPUs. Right? That's more than, like... That's like a real effort.
01:18:34 · Nathan Labenz, Prakash Narayanan
So you you would need multiple kinds of expertise. Like, that data is not that accessible. There are these, like, you know, moves to try to rein in that data and not let it, you know, just hang out all over the Internet. Prakash had heard a claim that morning that Claude was refusing to help clone video games. [01:18:56] S6: I I think I think what I fear is that they're pissing off the gamers. Right? And gaming. And you say Claude is not gonna help you, like, make these games? You're just asking for trouble. You're asking for the most security competent group, the most technically competent, and the people with the most time to burn, and it's a passion for them. NVIDIA was built on gamers. Like, John Carmack was a gamer. Right? You tell John Carmack that he can't do something. Like, you are in trouble. Right?
01:19:30 · Nathan Labenz
[01:19:30] S0: I I think that's apt. I mean, you... I think it is definitely the case that, like, we should be pretty cautious around where we deploy these things because we don't want to burn them on stupid things that don't really matter that much. It'll be like, you know, of all the ways to fail in this, it'll be really dumb if it's like, well, we were interested in protecting, you know, Disney IP. And so because of that, we, like, put in these guardrails, but then, you know, the fan fic community was so strong that they came over the top and, like, broke all of our safeguards. And now everything's, like, running wild, and it all was because, like, we were too draconian on things that didn't really matter. I mean, that's a bit of a caricature story, but it's... You know, that's basically the same story as the gaming one you're telling. And I can see that happening in different domains, and it really would be incredibly stupid. So I I do think we need to be, you know, pretty disciplined about, like, just how far we try to create these, like, knowledge gaps or refusals because everyone does potentially mobilize another community to try to undo them. And I agreed too that you're not gonna convince gamers that, like, well, hey. Because you're doing this, you know, you're creating this other downstream problem. Well, good luck having that. You know, you're not gonna win that fight in the Discord. So better to just probably let them have their fun and then, you know, keep our key techniques where we, like, really, really care and really have to have them. One more from Friday's closing right after Sagar left. Mistral's technical report and a Claude four system card story I told from memory. The card itself says, Anthropic had mistakenly omitted the only fine tuning dataset that included harmful system prompts, and that this was not the sole cause. So I'll be very interested to read the more open report on exactly, and hopefully, you know, we'll see how how open they are about all of the technical details. There have been these, like, footnotes in multiple system card reports over time. One that I really was kind of obsessed with for a minute was when... So Anthropic had forgotten to include a particular dataset called sysprompt harmful, which was meant to teach the model that it should not respond...
01:22:05 · Nathan Labenz, S7
You know, it should not directly follow a harmful system prompt. So it did follow the directions of a harmful system prompt, and this made it all the way through to the model card. We don't roll back and, you know, train again. Nope. We just kinda figure out a way to patch. I've really been very curious as to, like, how that stuff actually works and, like, how reliably we should think of it as working. That's the week. Thanks for listening to the AI and the AM weekly highlights. The narration was my cloned voice. Please tell us what worked and what didn't. See you in the morning. [01:23:31] S7: Aimed out and let her go. Ain't no calling her back. Water flat as a plate. All we do now is wait. This is the one we've got. The one we've got. Halfway out, something knocking, something wrong down below. Can't turn around, can't stop or grease up to the elbow. All hands all night. Nobody's saying it's fine. Come on in one or two ways. Everybody happy or everybody sad. No in between or halfway. Hold on. Hold on. Hold on. Hold on. This is the one we've got. The one we've Waterline keeps rising, call goes out for one more hand. There's a faith in fixing.