00:06:11 · Rebecca Balan, Tomer Schwartz
[00:06:11] S1: to double click on something you said, that a lot of different organisations found the bug at the same time. And you're saying this is because everyone's kind of using the same AI tools to do this, to search for vulnerabilities? [00:06:25] S2: Well, can't speak for everyone, right? But what I can say is that at Wiz, what we're using is a sort of custom harness that we've built. We call this Project Atlas, which builds on top of all the frontier AI models. It leverages both the the DeepMind capability, the the OpenAI capabilities, the the Anthropic capabilities. And in reality, what it what it allows us is to scale a lot of different ideas and try different potential attack paths at the same time. So it gives us a really broad scope to be able to pull up a lot of research ideas and a lot of potential issues. And from there, what we're trying to do is to distill it down to the most impactful ones and work on them. At the same time, I do I do know a little bit from our work with OpenAI and Calif. They're doing very similar things and trying to use their own models and and and new ideas to scale up research with AI, and they're finding, in many cases, very similar bugs to to the bugs that we're seeing.
00:06:25 · Tomer Schwartz, Rebecca Balan
[00:06:25] S2: Well, can't speak for everyone, right? But what I can say is that at Wiz, what we're using is a sort of custom harness that we've built. We call this Project Atlas, which builds on top of all the frontier AI models. It leverages both the the DeepMind capability, the the OpenAI capabilities, the the Anthropic capabilities. And in reality, what it what it allows us is to scale a lot of different ideas and try different potential attack paths at the same time. So it gives us a really broad scope to be able to pull up a lot of research ideas and a lot of potential issues. And from there, what we're trying to do is to distill it down to the most impactful ones and work on them. At the same time, I do I do know a little bit from our work with OpenAI and Calif. They're doing very similar things and trying to use their own models and and and new ideas to scale up research with AI, and they're finding, in many cases, very similar bugs to to the bugs that we're seeing. [00:07:24] S1: And how much of the the bug finding is really... Is it really AI versus, you know, human work that's supercharged? Right? Like, Calif found the bugs in a security audit says... You know, this is from our our security editor, Zach. He said that AI helped with the exploitation, but there was, you know, it was 90% human work. Calif's own line on this was that no model is simply capable of unearthing all bugs, at least for now.
00:07:49 · Tomer Schwartz
[00:07:49] S2: Well, I sort of agree. There's still an art and a science to this in how do you direct all of these models and how do you actually tailor them together to to be able to find such impactful bugs. But in reality, I didn't think that we could have found it without AI. So the the opportunity for a team of 10 of the most talented people that I know that are powering the the Wiz vulnerability research team right now, if we had taken all of them and told them, hey, maybe maybe find some some PHP RCE bugs, Maybe it it would have taken them a couple months of work, maybe even twelve months of work. And eventually, who knows? Maybe maybe they would have found the same the same finding. But what's cool about this is that we managed to take all of those people and tell them, hey, why wouldn't you try 20 different ideas on various different targets and just run them on top of Atlas and find a ton of different potential vulnerabilities and maybe do it over the weekend by the time we come back next week to see everything, we had a model working for us, trying a ton of different things that maybe a human researcher would not have found up. Right? So it's kind of cool to think that right now we can get 100x, 1000x the work of a researcher, and they can actually pick and choose what are the best what are the best research ideas and what are the best solutions and where do we need to pull this thread a little more. And in in many cases, with the capability of AI models today, it might be a full exploit already available where we just need to validate and we need to understand what the vulnerability is.
00:09:23 · Rebecca Balan, Tomer Schwartz
[00:09:23] S1: Yeah. I mean, it's great for defenders. It's also great for attackers. Right? And that's why we saw such a push to get this bug fixed because attackers are using AI to send out mass attacks and to 100x their own capabilities. Can you tell me a little bit more about Atlas? Atlas is your harness. Using, you said, OpenAI, Anthropic models. Are you using latest site? Are you using Mythos in that? Do you have access to that yet? Which models are you using and playing with? [00:09:51] S2: We constantly change it. With the pace in which AI models get better and better, we've to update this all the time, and there's also various different steps where we find that certain models are better at as opposed to others. [00:10:06] S1: Can you give me an example?
00:10:06 · Rebecca Balan, Tomer Schwartz
[00:10:06] S1: Can you give me an example? [00:10:08] S2: So let's talk about binary exploitation versus code comprehension. Right? These are very different tasks. If I take a very large code base and I'm looking for, hey, what kind of what kind of interactions do do we see here between components and what do they assume about how other components are using them? Maybe maybe there's one model that's really good about processing large chunks of data and compressing it into a whole bunch of insights that we can use later. But then if the the only thing we need is to to get to a certain part of the code base and figure out how to trigger this remotely, this is a completely different task. So we are trying to evaluate different models all the time. We we actually have a publicly facing benchmark that we we release. Fortunately, we also have access to unreleased models as well with the the labs that we're working with. So what we're trying to do is to just sort of do a best of breed kind of combination and see what's what's the best solution for this problem. [00:11:05] S1: How much of the mix is open models? Because I know a lot of people I've spoken to, including, you know, Clint along from Hugging Face, say that open models are really great cybersecurity because you don't run into any of the guardrails when you're red teaming or searching for vulnerabilities that you might with some of the frontier models. Is that what you're seeing as well?
00:11:05 · Rebecca Balan, Tomer Schwartz
[00:11:05] S1: How much of the mix is open models? Because I know a lot of people I've spoken to, including, you know, Clint along from Hugging Face, say that open models are really great cybersecurity because you don't run into any of the guardrails when you're red teaming or searching for vulnerabilities that you might with some of the frontier models. Is that what you're seeing as well? [00:11:22] S2: Well, open weight models are good, and they're getting better and better. But honestly, we're fortunate to be a part of Google right now, so we have access to pretty good models at unique pricing, I would say. But other than that, I think that that when you're really thinking about scaling research in in massive ways, maybe some of the startup companies are listening to us right now in podcast or or thinking about this too, we did a lot of different experiments with open weight models, and we do find them to be extremely effective. We had a couple of iterations where we did use them. It's just that the current model that we're using, the current harness that we're using is not using any open weight models. No. [00:12:04] S1: Not using any open weight models. Interesting. [00:12:06] S2: Not right now, but that situation could change in a month or two. It's kind of fun to see how quickly the the AI world is is moving.