The MAD Podcast: How AI Gets Built — with Matt Turck
00:03:36 · Nathan Benaich
[00:03:36] S0: Yeah. Yeah. Yeah. Exactly. And that's like I think up from 1% a couple of months ago and that was August, which sounds really far away. So God knows what it is in October. And so there's that I think that was that was really interesting. But then there was there was another... A stat that I really liked from a blog post that they had in June this year about recursive self improvement, which explained the concept. Then they presented this kind of dual arm experiment where they were testing out different model generations versus a human in this setup where first the human would undertake a research agenda and, you know, go a couple of steps down this path deciding what to work on doing the work. And then at some point, that experiment would fork and the human would continue or a specific model generation would continue. And it would be, you know, the old sonnet versions to Opus and then Fable. And then and the experiment would end and human graders would look at the human path versus the model path. And it was something like the old generation of models humans preferred like 10% of the time or something like that. But for the Mythos class models, humans preferred that research direction 64% of the time, which suggests that models are getting better at predicting the right steps or the right interesting kind of, you know, sort of leafs in the in the nodes to pursue. And then the second kind of piece of data I liked was some research from OpenAI with their Codex usage amongst their AI staff where they tried to classify the sort of token use in the overall process of their of their research. And that would be using tokens for deciding what research question to test, how to test it, what experiment you should set up, like writing the training code, running the, you know, the data engineering pipelines, writing the infrastructure, etcetera, and then marketing the paper, writing the paper, etcetera. And the vast majority of the token usage was on kind of explaining the research writing, which kind of makes sense, you know, sort of knowledge work followed by all the infrastructure for running the experiment. Very, very few tokens were used for what question should I explore and like, why should I explore it and what's the right path?
00:05:42 · Nathan Benaich, Matt Turck
And so I like this question a lot because like I was trained in science and biology and I just remember, you know, this is like ten years ago, fifteen years ago, not that long. But you know, at the start of the PhD, you would sit there with, you know, a ton of papers and you know, you go on the internet and read, read, read, read, read, try to understand what's the status quo, how do people test certain ideas, where are the gaps, what experiment should I run, and then in concert with your supervisor, like decide on the path. And the best supervisors would be the ones who could guide you down the most fruitful path to get hopefully your nature paper and then off you were. And the worst supervisors were the ones that like didn't really know or didn't have great ideas. So that idea of like research taste, which is what kind of gets bandied around on Twitter a lot, is I think one of the hallmarks of intelligence and is one of the critical components towards RSI working well and is one that I just like really enjoy because it speaks to my interest in science. [00:06:41] S1: Yeah. Indeed taste is sort of become a cliched Yeah. Word over the last couple of years. But in the specific context of RSI, just to play back what you just said, it's effectively choosing the right problem. Yeah. That's the right Yeah
00:06:41 · Matt Turck, Nathan Benaich
[00:06:41] S1: Yeah. Indeed taste is sort of become a cliched Yeah. Word over the last couple of years. But in the specific context of RSI, just to play back what you just said, it's effectively choosing the right problem. Yeah. That's the right Yeah [00:06:57] S0: exactly. And you know a lot of famous inventors will say, you know, it's like invention is a result out of good taste and a lot of the work is choosing what direction to work on, not necessarily just boiling the ocean. And so if we wanna be using tokens correctly, using our compute budgets correctly, but also showing that our systems are smart then they shouldn't just be And on boring [00:07:20] S1: conquering taste is something that you see happening in your various conversations as something know something happening in the next few months? [00:07:30] S0: Maybe longer, at least a year. I think the problem is like how do you objectively measure taste? I mean in art I guess you can because it's like perfectly observable and it's... And maybe in like wine you can sort of express like why this taste is better than that taste. But in like science and engineering where there's different paths to getting to the right results, it's not immediately clear. Like would it be the model that consumes the less tokens to get to an answer that humans believe is the best? Is it like the journey that matters more? And I think maybe the whole like uproar in the maths community of, you know, Astra solving or whatever model it was solving various like [00:08:11] S1: There's dope. Open questions.
00:08:11 · Matt Turck, Nathan Benaich
[00:08:11] S1: There's dope. Open questions. [00:08:12] S0: Yeah, exactly. It's like we're, okay, we got the answer, but like a human or a system didn't sort of go through the pains of navigating how to get to the answer. And it's like the process that's important, not like not the end result. So to me, like somehow quantifying like what a good process looks like, it'd be important to determining whether the output of, like, taste is true or not. [00:08:33] S1: Yep. What would you say the mood is at the almost end of 2026? I remember when we chatted two years ago, one of the key highlights or takeaways from the conversation was that, doomerism was largely gone. But now there's like this whole revival. We had Redwood Research on the on the pod talking about, all sorts of, terrible scenarios. So what do you make of that?
00:08:57 · Nathan Benaich, Matt Turck
[00:08:57] S0: Yeah. You're right. I think it was like two, three years ago there was the... It was it was strictly speaking ex risk, like extermination of humanity and like loss of control. And then we sort of went away from that and largely forgot it and sort of just like hill climb towards capabilities. And then now now it's sort of back, but I'd say it's both like ex risk loss of control, but much more short term like cybersecurity risks. And to me, like the cyber risk is like much more real than like AI exterminating humanity. As well as the cybersecurity risk could eventually lead to those kinds of cataclysmic scenarios. But I think what the report shows with... And it was roughly written around the time of the Hugging Face incident with OpenAI kind of coming out and, you know, us learning more about what happened. And then since then, the last few weeks has been, you know, breaches of PHP and government's services getting like, you know, taken down in Asia and other places. I think it's becoming very clear that, you know, regardless of if you believe in like safety or not safety, like we need security. So like a bunch of the predictions that I have in the report talk about escalating cyber threats, whether it's, you know, exfiltrating model weights from labs, which I thought was gonna be like AI driven and turns out like, a human could just try to plug in a USB stick. [00:10:13] S1: Yeah. So let's talk let's talk about that. Let's talk about cybersecurity then. So that specific prediction, what what is it?
00:10:13 · Matt Turck, Nathan Benaich
[00:10:13] S1: Yeah. So let's talk let's talk about that. Let's talk about cybersecurity then. So that specific prediction, what what is it? [00:10:20] S0: Yeah. So the prediction is that there would be an AI driven, like, cyber attack to a frontier lab that would attempt and successfully exfiltrate model weights. [00:10:32] S1: The model. Yeah. [00:10:33] S0: Entire thing. Yeah. Yeah. The model. What happened just like a week or two later was actually like a human effectively tried to steal, you know, IP whether he tried to steal like the actual model weights or not. But like certainly... [00:10:44] S1: Why was that incident?