The MAD Podcast: How AI Gets Built — with Matt Turck
00:06:41 · Matt Turck, Nathan Benaich
[00:06:41] S1: Yeah. Indeed taste is sort of become a cliched Yeah. Word over the last couple of years. But in the specific context of RSI, just to play back what you just said, it's effectively choosing the right problem. Yeah. That's the right Yeah [00:06:57] S0: exactly. And you know a lot of famous inventors will say, you know, it's like invention is a result out of good taste and a lot of the work is choosing what direction to work on, not necessarily just boiling the ocean. And so if we wanna be using tokens correctly, using our compute budgets correctly, but also showing that our systems are smart then they shouldn't just be And on boring [00:07:20] S1: conquering taste is something that you see happening in your various conversations as something know something happening in the next few months? [00:07:30] S0: Maybe longer, at least a year. I think the problem is like how do you objectively measure taste? I mean in art I guess you can because it's like perfectly observable and it's... And maybe in like wine you can sort of express like why this taste is better than that taste. But in like science and engineering where there's different paths to getting to the right results, it's not immediately clear. Like would it be the model that consumes the less tokens to get to an answer that humans believe is the best? Is it like the journey that matters more? And I think maybe the whole like uproar in the maths community of, you know, Astra solving or whatever model it was solving various like [00:08:11] S1: There's dope. Open questions.
00:08:11 · Matt Turck, Nathan Benaich
[00:08:11] S1: There's dope. Open questions. [00:08:12] S0: Yeah, exactly. It's like we're, okay, we got the answer, but like a human or a system didn't sort of go through the pains of navigating how to get to the answer. And it's like the process that's important, not like not the end result. So to me, like somehow quantifying like what a good process looks like, it'd be important to determining whether the output of, like, taste is true or not. [00:08:33] S1: Yep. What would you say the mood is at the almost end of 2026? I remember when we chatted two years ago, one of the key highlights or takeaways from the conversation was that, doomerism was largely gone. But now there's like this whole revival. We had Redwood Research on the on the pod talking about, all sorts of, terrible scenarios. So what do you make of that?
00:08:57 · Nathan Benaich, Matt Turck
[00:08:57] S0: Yeah. You're right. I think it was like two, three years ago there was the... It was it was strictly speaking ex risk, like extermination of humanity and like loss of control. And then we sort of went away from that and largely forgot it and sort of just like hill climb towards capabilities. And then now now it's sort of back, but I'd say it's both like ex risk loss of control, but much more short term like cybersecurity risks. And to me, like the cyber risk is like much more real than like AI exterminating humanity. As well as the cybersecurity risk could eventually lead to those kinds of cataclysmic scenarios. But I think what the report shows with... And it was roughly written around the time of the Hugging Face incident with OpenAI kind of coming out and, you know, us learning more about what happened. And then since then, the last few weeks has been, you know, breaches of PHP and government's services getting like, you know, taken down in Asia and other places. I think it's becoming very clear that, you know, regardless of if you believe in like safety or not safety, like we need security. So like a bunch of the predictions that I have in the report talk about escalating cyber threats, whether it's, you know, exfiltrating model weights from labs, which I thought was gonna be like AI driven and turns out like, a human could just try to plug in a USB stick. [00:10:13] S1: Yeah. So let's talk let's talk about that. Let's talk about cybersecurity then. So that specific prediction, what what is it?
00:10:13 · Matt Turck, Nathan Benaich
[00:10:13] S1: Yeah. So let's talk let's talk about that. Let's talk about cybersecurity then. So that specific prediction, what what is it? [00:10:20] S0: Yeah. So the prediction is that there would be an AI driven, like, cyber attack to a frontier lab that would attempt and successfully exfiltrate model weights. [00:10:32] S1: The model. Yeah. [00:10:33] S0: Entire thing. Yeah. Yeah. The model. What happened just like a week or two later was actually like a human effectively tried to steal, you know, IP whether he tried to steal like the actual model weights or not. But like certainly... [00:10:44] S1: Why was that incident?
00:10:46 · Nathan Benaich
[00:10:46] S0: This was where an OpenAI employee who'd been there for a little bit essentially like tried to implement a listening device into a conference room that he figured out by asking the executive... The sort of office staff which rooms were most commonly used by executives and then tried to conceal the device by wrapping it in toilet paper and putting it behind a TV, which is where it created this like iconic Internet meme of like a human like pairing behind TV to see if the thing is like connected correctly, which is sort of like comical at this point. And then that person was reported to the FBI and then there was an investigation. So I think, yeah, we oftentimes forget that even though AI is getting really capable and can orchestrate cyber threats, lot of the risks are actually human driven. But I think that's a big issue particularly as like there's increasing competition between nation states of like who has the best AI systems. This is driving obviously like economic growth and economic instability and political instability and US like model labs have started to sort of communicate crackdowns on distillation. So, you know, maybe some nations are trying to, like, get access to model capabilities in other ways that might not be distillation based. But generally speaking, think, you know, as companies become richer, as individuals become become wealthier through this ownership of AI and, like, disparity ends up increasing, which we see in other data in the report of, like, some companies spending a lot and gaining a lot and other companies, like, having employees still using ChatGPT to, like, write emails. I think this kind of tension breeds, you know, increasing risk and, you know, with frontier models like in the open domain becoming very good at cyber hacks, particularly the Chinese ones, you know, the sort of like threat surface just grows and the number of actors who can use these tools to do bad things grows.
00:12:40 · Matt Turck, Nathan Benaich
[00:12:40] S1: Yeah. It's a whole complexity of the role of open weights in cyber because in the Hugging Face incidents, Hugging Face was able to protect itself. Thanks to an open weight model while the closed source model refused to help. [00:12:56] S0: Yeah. That I find is the most scary, maybe the second most scary fact in all of this. And there was a Chinese model. I think it was GLM five five point three or something. [00:13:05] S1: And maybe since this is is kind of, like, a retrospective kind of episode, maybe just walk people in in a few sentences through the Hugging Face incident because we had Thomas Wolf on the on the pod, but it may fall in the category of something that people heard about like a thousand times who actually don't know what actually happened.