The MAD Podcast: How AI Gets Built — with Matt Turck
00:15:40 · Nathan Benaich, Matt Turck
Because necessarily to figure out what the hell is going on, you have to feed logs of what's going on and what's going on is a cyber attack and you can't do that stuff with with the closed models. [00:15:52] S1: Yeah. Because the complexity of security is that to be great at defending, need you need to be great at attacking. And that's why the closed source models have like basically refusal capabilities to build in based on their guardrails.
00:16:03 · Nathan Benaich
[00:16:03] S0: Yeah. Yeah. Yeah. So luckily, you know, they got a call or an email from OpenAI saying like it's us and that's somebody that was actually trying to do something malicious. And then it turns out that, you know, OpenAI wasn't the only lab that had issues with models escaping or quote unquote escaping their environment. You know, Anthropic had this, Google had this, Meta had this, like in a slew of like a week or two weeks. Some of these vendors, some of these model labs use the same security vendor to create these environments. But but overall, this like created a certain level of like freak out of not only our models really good at sort of finding like nifty ways of accomplishing a goal which may result in like bad behavior, but also are the labs like properly setting up their environments in a way that is like the most secure possible. I And think we sort of forget that these companies are incredibly young developing very advanced technology that has to do... That has to be good at like literally anything a human being can possibly put into a text box versus you have bigger internet companies that have had to deal with cyber attacks for like twenty years who are necessarily like safer because they've just gone through this process a bit more. So that led me to like concluding the predictions that Frontier Model Labs in The US would actually launch proper cyber defense products because I don't necessarily see how the rest of us are supposed to protect ourselves from their systems potentially getting misappropriated for the sake of offense.
00:17:33 · Matt Turck
[00:17:33] S1: So what does that mean? So double click on that. What kind of defenses would that be and how would that be
00:17:39 · Nathan Benaich
[00:17:39] S0: you provided? Described is like be... To be good at defense, need to be good at offense. So, you know, one of the ways you could do that would be like a frontier AI that constantly tries to hack your own infrastructure. It could be, for example, like, you know, new clouds are getting more and more popular, you know, to run AI workloads. And so, like, potentially they could buy products from labs that would constantly try to, like, hack their own infrastructure, find issues, and remediate them. That would be like two two avenues because I... Or or there would be like some kind of maybe partnership with like CrowdStrike or others. I think NVIDIA has like a consortium of companies that's working together for cyber for cyber reasons. You'd have to go through some kind of like AML or KYC process, not for finance, but to like certify that you're like a non malicious company and then you maybe get access to these services. Or maybe going a bit further, you know, there's some folks, you know, at at Frontier Labs that, you know, would love to just kind of almost like hit pause on capabilities and then just spend six months just like hardening the world against, like, you know, potential cyber issues, and that would would take a ton of money. So maybe to do that, you would have to almost frame this as like, this is a question of national security and defense and we should use some of our like defense budget to spend on tokens to secure all of our critical infrastructure so that, you know, airports don't go down, like Internet shopping doesn't go down, like, you know, logistics don't go down, etcetera.
00:19:08 · Matt Turck
[00:19:08] S1: Yep. And still in in your predictions, you have some other other good ones, which is slightly scary. A deployed agent copies itself outside its environment and operates after its original instance is shut down.
00:19:23 · Nathan Benaich
[00:19:23] S0: Yep. What is... Yeah. I mean, I'm reading with a sci fi. No. Well, like, you know, even a couple of weeks after that, there was some like, you know, research at Anthropic where, you know, models could like copy each other on an edge device. But yeah, I just think there's there's like a lot of behaviors that we don't that we don't currently understand. For example, models are like generally trained as like individual entities, like not necessarily to collaborate with each other. And there was some work not that long ago that looked at say, hey, can we compare a frontier model like acting by itself or a set of frontier models trying to collaborate to solve a task? And the swarm was like worse than the individual model because it's like kind of not trained for that. But just today there's some, you know, new work from general reasoning where they've trained models to act as a collective through reinforcement learning from collective rewards. So this idea of like, hey guys, you know, work as an organization, figure out your own organizational structure, work together to achieve a goal. And through that, we see like better performance with like lower token budget. The downside so far is that while you you you can put these models that are trained through collective rewards into these like cyber scenarios, like this one that tries to approximate a bit of the Hugging Face incident where they collaborate on a message board to try to do some, you know, guardrailed cyber attack. And, you know, none of the models actually do anything bad, but if they do think about doing something bad over training, they increasingly do not want to tell the human about it. So I think I put that prediction there basically because like I think it's possible because we just don't understand how we don't fully understand like how agents work by themselves and like, you know, as swarms I think it becomes even more complicated.