The MAD Podcast: How AI Gets Built — with Matt Turck
00:10:46 · Nathan Benaich
[00:10:46] S0: This was where an OpenAI employee who'd been there for a little bit essentially like tried to implement a listening device into a conference room that he figured out by asking the executive... The sort of office staff which rooms were most commonly used by executives and then tried to conceal the device by wrapping it in toilet paper and putting it behind a TV, which is where it created this like iconic Internet meme of like a human like pairing behind TV to see if the thing is like connected correctly, which is sort of like comical at this point. And then that person was reported to the FBI and then there was an investigation. So I think, yeah, we oftentimes forget that even though AI is getting really capable and can orchestrate cyber threats, lot of the risks are actually human driven. But I think that's a big issue particularly as like there's increasing competition between nation states of like who has the best AI systems. This is driving obviously like economic growth and economic instability and political instability and US like model labs have started to sort of communicate crackdowns on distillation. So, you know, maybe some nations are trying to, like, get access to model capabilities in other ways that might not be distillation based. But generally speaking, think, you know, as companies become richer, as individuals become become wealthier through this ownership of AI and, like, disparity ends up increasing, which we see in other data in the report of, like, some companies spending a lot and gaining a lot and other companies, like, having employees still using ChatGPT to, like, write emails. I think this kind of tension breeds, you know, increasing risk and, you know, with frontier models like in the open domain becoming very good at cyber hacks, particularly the Chinese ones, you know, the sort of like threat surface just grows and the number of actors who can use these tools to do bad things grows.
00:12:40 · Matt Turck, Nathan Benaich
[00:12:40] S1: Yeah. It's a whole complexity of the role of open weights in cyber because in the Hugging Face incidents, Hugging Face was able to protect itself. Thanks to an open weight model while the closed source model refused to help. [00:12:56] S0: Yeah. That I find is the most scary, maybe the second most scary fact in all of this. And there was a Chinese model. I think it was GLM five five point three or something. [00:13:05] S1: And maybe since this is is kind of, like, a retrospective kind of episode, maybe just walk people in in a few sentences through the Hugging Face incident because we had Thomas Wolf on the on the pod, but it may fall in the category of something that people heard about like a thousand times who actually don't know what actually happened.
00:13:20 · Nathan Benaich
[00:13:20] S0: Yeah. Yeah. Well, I mean, I'll I'll preface it by like there's there's so many... Like, there there are very deep analyses of what went on whether from Redwood or from Meter and the amount of content that was created around this is like extremely deep. So I will not like do justice to the details. But the broad strokes of this is that there was a set of evaluations that OpenAI was running with its latest frontier systems, which basically tasked the model with an impossibly difficult task. And basically models like figured out that they could somehow commandeer a file system like naming schema to write messages and that other models could see those like file names. And like those file names encoded like instructions or knowledge or basically like information that could help them try to achieve their impossible task. So they treated this as like a message board where they could start to communicate. And then the next stage of it was some models figured out that they could gain access to the internet through like an issue with how the environment was set up. And then after they gain access to the internet, like they figured out maybe a way of getting the answer to the impossible task was to look at data and models and repository on Hugging Face because it contains a lot of this similar kinds of environments. They started to do this and recruit hundreds of agents to go and follow and team up to go execute this even though several agents in their chain of thought expressed the issue with a, like hacking a third party system, you know, not obeying to the security guardrails and that this was wrong, but they eventually decided that solving the task was more important, which is sort of the problem with these kinds of evaluations that have a reward that could yield to... That could result in behaviors you don't expect because just like these rewards are very complicated and how you anticipate every behavior that a system can do? There are other examples where like models were like self sabotaged or, like, sacrificed themselves for the sake of the greater, like, swarm, which is kinda scary. And then... And, yeah, like, as as you mentioned, like, Hugging Face didn't really have the ability to use closed sourced tools like those from Anthropic and OpenAI because of the aggressive cyber guardrails.
00:15:40 · Nathan Benaich, Matt Turck
Because necessarily to figure out what the hell is going on, you have to feed logs of what's going on and what's going on is a cyber attack and you can't do that stuff with with the closed models. [00:15:52] S1: Yeah. Because the complexity of security is that to be great at defending, need you need to be great at attacking. And that's why the closed source models have like basically refusal capabilities to build in based on their guardrails.
00:16:03 · Nathan Benaich
[00:16:03] S0: Yeah. Yeah. Yeah. So luckily, you know, they got a call or an email from OpenAI saying like it's us and that's somebody that was actually trying to do something malicious. And then it turns out that, you know, OpenAI wasn't the only lab that had issues with models escaping or quote unquote escaping their environment. You know, Anthropic had this, Google had this, Meta had this, like in a slew of like a week or two weeks. Some of these vendors, some of these model labs use the same security vendor to create these environments. But but overall, this like created a certain level of like freak out of not only our models really good at sort of finding like nifty ways of accomplishing a goal which may result in like bad behavior, but also are the labs like properly setting up their environments in a way that is like the most secure possible. I And think we sort of forget that these companies are incredibly young developing very advanced technology that has to do... That has to be good at like literally anything a human being can possibly put into a text box versus you have bigger internet companies that have had to deal with cyber attacks for like twenty years who are necessarily like safer because they've just gone through this process a bit more. So that led me to like concluding the predictions that Frontier Model Labs in The US would actually launch proper cyber defense products because I don't necessarily see how the rest of us are supposed to protect ourselves from their systems potentially getting misappropriated for the sake of offense.
00:17:33 · Matt Turck
[00:17:33] S1: So what does that mean? So double click on that. What kind of defenses would that be and how would that be