The MAD Podcast: How AI Gets Built — with Matt Turck
00:17:39 · Nathan Benaich
[00:17:39] S0: you provided? Described is like be... To be good at defense, need to be good at offense. So, you know, one of the ways you could do that would be like a frontier AI that constantly tries to hack your own infrastructure. It could be, for example, like, you know, new clouds are getting more and more popular, you know, to run AI workloads. And so, like, potentially they could buy products from labs that would constantly try to, like, hack their own infrastructure, find issues, and remediate them. That would be like two two avenues because I... Or or there would be like some kind of maybe partnership with like CrowdStrike or others. I think NVIDIA has like a consortium of companies that's working together for cyber for cyber reasons. You'd have to go through some kind of like AML or KYC process, not for finance, but to like certify that you're like a non malicious company and then you maybe get access to these services. Or maybe going a bit further, you know, there's some folks, you know, at at Frontier Labs that, you know, would love to just kind of almost like hit pause on capabilities and then just spend six months just like hardening the world against, like, you know, potential cyber issues, and that would would take a ton of money. So maybe to do that, you would have to almost frame this as like, this is a question of national security and defense and we should use some of our like defense budget to spend on tokens to secure all of our critical infrastructure so that, you know, airports don't go down, like Internet shopping doesn't go down, like, you know, logistics don't go down, etcetera.
00:19:08 · Matt Turck
[00:19:08] S1: Yep. And still in in your predictions, you have some other other good ones, which is slightly scary. A deployed agent copies itself outside its environment and operates after its original instance is shut down.
00:19:23 · Nathan Benaich
[00:19:23] S0: Yep. What is... Yeah. I mean, I'm reading with a sci fi. No. Well, like, you know, even a couple of weeks after that, there was some like, you know, research at Anthropic where, you know, models could like copy each other on an edge device. But yeah, I just think there's there's like a lot of behaviors that we don't that we don't currently understand. For example, models are like generally trained as like individual entities, like not necessarily to collaborate with each other. And there was some work not that long ago that looked at say, hey, can we compare a frontier model like acting by itself or a set of frontier models trying to collaborate to solve a task? And the swarm was like worse than the individual model because it's like kind of not trained for that. But just today there's some, you know, new work from general reasoning where they've trained models to act as a collective through reinforcement learning from collective rewards. So this idea of like, hey guys, you know, work as an organization, figure out your own organizational structure, work together to achieve a goal. And through that, we see like better performance with like lower token budget. The downside so far is that while you you you can put these models that are trained through collective rewards into these like cyber scenarios, like this one that tries to approximate a bit of the Hugging Face incident where they collaborate on a message board to try to do some, you know, guardrailed cyber attack. And, you know, none of the models actually do anything bad, but if they do think about doing something bad over training, they increasingly do not want to tell the human about it. So I think I put that prediction there basically because like I think it's possible because we just don't understand how we don't fully understand like how agents work by themselves and like, you know, as swarms I think it becomes even more complicated.
00:21:16 · Matt Turck, Nathan Benaich
[00:21:16] S1: Okay. All right. So maybe to, close this part of, the conversation on, AGI, RSI, cyber, and basically things getting out of hand. Where are you on the pacing the frontier idea? Is that something that you think makes sense? Something that's realistic or not? [00:21:37] S0: I can see multiple like angles on this one. You know, in in some way if you ask the general public, okay, all the lab leaders are saying we have something that we just like increasingly can't control that's dangerous, that has caused these incidents. Like, do you think it's worth continuing or like just figuring out what's going on before you make it worse? And I think a logical, like, immune reaction is like, yeah, we should probably, like, figure it out before we go much further. I think the practical reality is that there's so much money at stake from all sorts of different parties in the economy that I think it... That kind of level of collaboration and coordination is really difficult and is sort of like peak game theory where maybe some some participants, like, support it because they are behind. Other ones support it because it plays to their general agenda, whether their agenda is like fair or not fair or seeking another goal. I I I don't know. But I think the the practical things are like who decides it? How do you measure it? How do you implement it? When do you know when to stop or start. There's some proposals like sending like external evaluators into labs to sort of measure the stuff. Trying to see [00:22:48] S1: big revenge of KPMG.
00:22:48 · Matt Turck, Nathan Benaich
[00:22:48] S1: big revenge of KPMG. [00:22:51] S0: That's a good one. Or trying to like somehow survey like workloads that are running on data centers, maybe like, you know, put compute thresholds on certain workloads, but like that stuff has been tried in the past and it's like very difficult to monitor. [00:23:04] S1: Yeah, especially when one thinks about the international part of the prisoner's dilemma, right, especially US versus China. [00:23:12] S0: Yeah. So like this part of the report, like I don't have like the definitive answer but I've tried to do my best to present the different perspectives on things and then let people conclude with their with their own data. But I find it's... If we extend this towards like, you know, this nimbyism debate, which we predicted in the last year's report of, you know, most people would not wanna, you you know, live next to a data center and then this would start to become a politically important component to midterms. [00:23:37] S1: Yeah. That was a resoundingly correct prediction. Right? [00:23:42] S0: Yeah. Yeah. Yeah. It seems so Where more and more states have tried to implement a moratorium or have done so or established new criteria by which grid connections can happen. Like, for example, you know, data centers would have to figure out and bring their own power on-site or they would have to make certain investments in the communities that they want to build in.
00:23:42 · Nathan Benaich, Matt Turck
[00:23:42] S0: Yeah. Yeah. Yeah. It seems so Where more and more states have tried to implement a moratorium or have done so or established new criteria by which grid connections can happen. Like, for example, you know, data centers would have to figure out and bring their own power on-site or they would have to make certain investments in the communities that they want to build in. [00:24:03] S1: Yeah. And while you added, so maybe just go into this a bit more, explain what the current state of things. Obviously, we all know that there is opposition to data center construction, but, like, any color, you can add. [00:24:15] S0: Yeah. Well, there's there's... They're like projects, I think, worth about $50,000,000,000 that are currently stalled, through a variety of these these reasons. Like Texas, Pennsylvania have been ones that have generally, like, paused paused new data center connections. There's a there's a stat on polling of, like, American American people where individuals would prefer to live next to a nuclear power plant than live next to a data center, which is quite wild to think of. [00:24:44] S1: And what what do you think that is by the way? Is that lack of understandingeducation? Is that just terrible PR