"The Cognitive Revolution"
01:00:09 · Sagar Vaze
[01:00:09] S4: The second is that, you know, this threshold slot budget does evolve over time because precraning does essentially become more efficient as we... As our level of scientific understanding increases. So a great example of this is sort of MOE architectures. I mean, you look, say, example, GPT-three, this was a dense model trained with three x the number of active parameters as the current Mistral Large four. Particularly, GPT three was trained with 175 b parameters total, but it was a dense model, so all parameters were activated for each token. Whereas ML four is a sparse mixture of experts, such that it's... There's one third the number of parameters activated per token, so only 50,000,000,000 parameters. But the total number of parameters is still 1,000,000,000,000. Your flop budget is proportional to the number of active parameters, so the number of, you know, parameters you're activating each token during training. And what this means is that you can train much larger models, much more capable models, but much less flops in total. So this figure that I referenced, so one e 25, this is in itself a moving target. So I wouldn't use that as an absolute target for a capable model throughout time. There would have been some one e 20 fives much less performant one or two years ago. And I would argue actually some larger models also much less performant today. But as pre training science progresses, as we learn how to make better use of the flops, this number itself will change. And at least what we're seeing is, at the moment, looking at the other open source labs, our understanding of, you know, what are the best models being created at the moment, training a 1,000,000,000,000 parameter model on one e 25 is sort of a rite of passage you have to go through in order to be able to to justify scaling even further.
01:01:56 · Nathan Labenz, Sagar Vaze
[01:01:56] S0: I asked how much uncertainty is left once a run that size is launched and whether they had ever aborted one. Halfway through this run, he said, they found something wrong with a big share of their compute already spent. [01:02:12] S4: Your other question was about, okay. So you do all this prep, and then, you know, what happened when you actually launched the model? And again, I think that every precraner will have, you know, the same stories to tell you, which is that precraning is kind of like you... You're you're doing all these scaling laws, you're sort sort of trying to align the tanker, and then you sort of let it go and you have to let it crane for however long you budgeted for it. And it's a huge amount of compute and capital expenditure in that run. And sometimes sometimes things were great, and the the loss goes down exactly as you expect it to. And sometimes there are surprises along the way and you have to be creative and find innovative solutions. So we had to figure out, okay, how can we fix it? And it was sort of all hands on deck, bunch of ablations, bunch of analysis of the model. And then we we found a solution. We found a way to be... We called it a... We found a way to patch it. We called it patch trial internally. And this it worked even better than we expected, and the the result is the model you see today. And, yeah, part of part of scaling compute is gonna be building the bones of how to iterate on this process more reliably.
01:03:26 · Nathan Labenz
[01:03:26] S0: After Sagar left, Prakash came back to the compute gap. 3,800 GPUs against the hundreds of thousands at the Frontier Labs. Yeah. I think that's gonna be a really hard deficit to make up for. I mean, we talked about this within the context of Google, I think, a few times. You know, when you own a quarter of the world's compute supply, you can have a a missed cycle, and you can, like, figure out a way to get back on the horse. And seems like this week, they've done that. Right? We barely even talked about carbon. I I mean, with Mistral, it does seem like the one big thing is that they do have, like, state backing. You know, they... I I I really do think if they miss a model cycle in a purely capitalist context, they're maybe done. You know? You... Like, if if you are just relying on return seeking investors to fund your growth and you miss at this point, that may be fatal. There's not a lot of sovereign AI plays out there that are at all credible, and literally, there might just be one. So in that context, I do think that gives them not... It doesn't translate directly to compute, but I do think it translates to at least, like, reasonable expectation that they will be around to fight another day because there's just so many interests that want to see that happen that I can't imagine, you know, even if everything were to totally flop, would the whole continent of Europe just be like, well, I guess we have zero now. It's all China and The US from here. Like, I I just can't see that, being an acceptable answer. Llama three, he he mentioned. I did look up. Llama three was... Get the exact dates here. July 2024, and it was trained on 3.8 times 10 to the 25 flops. So llama three more than two years ago, nearly four times the input compute that Mistral Large Four has today. That is a... There's a... I think a lot can be, unpacked from that one statistic. Part seven, what the open model shouldn't know. After Freeman left, Prakash and I took up whether you can keep a capability out of a model at all. Prakash's case started with his account of the 50¢ army being paid to scrub the Internet of things like Tiananmen Square.
01:05:58 · Prakash Narayanan
[01:05:58] S6: And so in order to get these guys to understand what they're supposed to censor, they had to teach them what Tiananmen Square was. So the CCP ended up having these party classes to teach the the 50 centers what was Tiananmen Square, and what you should be looking for when you want a sensor. And I think that is gonna be largely true of almost all things LLM, where these guys are trying to teach it what not to do, and they're going to specifically take out these behaviors and the LLM is going be left wondering, all right, so these are all the good behaviors and I can have antonym to every good behavior, that antonym should go into this solution space and this latent space that I should be seeing that antonym in doesn't exist.
01:06:46 · Nathan Labenz
[01:06:46] S0: Prakash's conclusion was that you can't carve a capability out of the latent space. My reply? I'm I'm kinda more bullish on some of these, like, not necessarily unlearning, but, you know, I I just recorded a podcast earlier today where I was talking a bit about how feasible it is. I mean, there's there's multiple levels of control, of course, but, like, there are efforts underway to basically lock up certain classes of biodata that are being newly generated now because the worry is that they'll create dangerous capabilities and models if they're they're used in training, and not necessarily to shut that out, you know, forever from everyone, but at least to have it on kind of a know your customer basis or whatever. It does seem that those... What what few experiments have been done on, like... And, you know, I'm thinking here of ESM and also some work, the EVO series of models that came out of ARC Institute and Stanford and whatever big collaborations, of course. It seemed like they showed that if you do withhold certain classes of data, you can get generally capable models that that still kind of don't know about these areas of, censorship. So I'm I'm not so sure that that can't work. Took that question to Sagar when he came on. First though, Mistral built Large four with cyber defense as a focus, and Prakash asked how that changed the training.
01:08:22 · Sagar Vaze
[01:08:22] S4: So so the cyber capabilities and agent decoding capabilities in general are something which are typically given to the model in the in the post training stage. So most of the stuff we've discussed today so far has been about pre training. So Right. Where a large percentage of the flops go at at the start. Right. And then in post training is where you say, okay, well, what are people going to really use this model for? And that's when, you know, you start giving it these these new capabilities of finance, law, defense, etcetera. Earlier in the year, there were there were some events where some very capable models had restricted access to both certain geographies as well as to certain enterprises. And at the same time, we were seeing that these very capable models had these incredible cyber capabilities, which often were being jailbroken by malicious actors in order to to infiltrate or expose vulnerabilities in in, you know, legitimate defenders. And an imbalance which we saw here is that these defenders were then trying to, you know, use these same models to patch their own vulnerabilities, but were finding that their requests were refused. Because in practice, for very good reasons, these closed labs want to make sure that the models are not misused. They've got classifiers under the hood to identify malicious cyber requests from a standard request. But unfortunately, in practice, if you're a defender and you're trying to shore up your infrastructure, your defensive request looks quite a lot like a malicious request because you have to try and find vulnerabilities in your own code base to patch them. And so we thought it was imperative for there to be an open weight cyber model of similar capabilities to these closed models such that, you know, enterprises could scalably defend themselves.