00:12:06 · Tomer Schwartz, Rebecca Balan
[00:12:06] S2: Not right now, but that situation could change in a month or two. It's kind of fun to see how quickly the the AI world is is moving. [00:12:16] S1: Yeah. Absolutely. Now, you know, because it's moving so quickly, which was part of the problem with, you know, rushing to get the fix out once the word was out that there was this bug, Did you learn anything from this? Did Wiz learn anything from this in terms... Because I know you had a schedule when you wanted to release this publicly. Have you learned anything in terms of how to handle doing this quietly and quickly? What was it like inside? Did it ruin your weekend? What happened?
00:12:40 · Tomer Schwartz
[00:12:40] S2: There's always learnings out of this, and coordinated disclosure is something that is fundamentally flawed. We want to bring more and more people into the inner circle to make sure that defenders are aware of the the vulnerability and they have time to prepare. But then on the other side, we can't really announce it to the world without adversaries know... Knowing about this. And this is not a model that we invented, obviously. This this is a problem in the cybersecurity industry for decades. The question of how to do it well and how to do this in a very timely process, I think, is is right now a big question that the industry needs to tackle with the the the imminent vulnpocalypse that we're seeing right now. So I think it was a a a pretty fast turnaround time compared to to all the things that needed to happen. From detection to fix, I think it was roughly two months, which is pretty amazing given the fact that this this was a a very unique architecture flaw within PHP that was was lurking there for twelve years. Right? It was it was very diff... Difficult for us to come up with a solution that is reliable enough, that covers every single aspect of this unique vulnerability, that is backwards compatible with some of the extensions that PHP has. There were a lot of different constraints that we had to operate in. And given the fact that we couldn't really bring in a lot of people into that inner circle, I think we managed to do this, again, comparably pretty quickly. The other side of this is what happened towards the end of that cycle where we realized, well, this is something that can take the Internet down quite literally. So we got to work with the cloud providers, with the the web application firewall providers, with, you know, lot of different vendors that are associated with this that need to be aware so they can make sure that they release on time, they have the the detection rules out, that we have we have the full set of visibility and and potential... Not full defense, but partial defense against what's what's gonna happen. And once you realize how big this industry is, you realize that, you know, maybe that inner circle is all all of sudden hundreds of people. And maybe they're talking to some some of their colleagues because they have to pull them in to be able to operate. So I don't I don't blame anyone for this leaking.
00:15:00 · Tomer Schwartz, S0, Rebecca Balan
It's just... It's it's it's a function of how coordinated disclosure is happening. And luckily, we were pretty much ready to that situation. We had sort of a plan b that we we could have operated bay... Based off of. And I think that we made the right decisions in in in every given point in time. [00:15:18] S0: Every company today is looking for an edge. Better insights, faster decisions, greater resilience. But as AI drives new growth, it also creates new demands on energy and infrastructure. Schneider Electric is connecting energy, automation, and digital technologies to help organizations make smarter decisions and operate more efficiently. Because the future won't be shaped by AI alone, it will be shaped by energy and industrial intelligence. Schneider Electric, advancing energy tech. [00:15:45] S1: So just to like take a step back for coordinated disclosure for anyone who isn't familiar with this, researchers will privately notify software maintainers, give them time to prepare a fix before the vulnerability is made public. Right? And so PHP Foundation would have given big companies and hosts a bit of a head start. Maybe some people had already, you know, implemented a fix or started to before it became public. I don't know.
00:15:45 · Rebecca Balan, Tomer Schwartz
[00:15:45] S1: So just to like take a step back for coordinated disclosure for anyone who isn't familiar with this, researchers will privately notify software maintainers, give them time to prepare a fix before the vulnerability is made public. Right? And so PHP Foundation would have given big companies and hosts a bit of a head start. Maybe some people had already, you know, implemented a fix or started to before it became public. I don't know. [00:16:07] S2: Yeah. So the intention wasn't to give certain companies a a preferential treatment and tell them, hey. Go patch this because, you know, we feel you're... That you're more important than than the rest. That was not our intention, and we were very clearly communicating this. I think that our intention was to say, for those choke points that really have massive leverage for the part of the Internet, the part of the industry that relies on PHP, we gotta make sure that at the very least, they're ready for the release. Right? So what I mean by that is let's talk about how distribution works in open source. Right? So the PHP Foundation is a foundation. It has an amazing team that maintains the PHP code base. [00:16:57] S1: A team of volunteers. Right? Well,
00:16:59 · Tomer Schwartz
[00:16:59] S2: I I I can say for certain, I think they're volunteers, but, you know, if if... We need to ask the PHP Foundation about about how it works. I think I think they are. But, basically, we worked with them night and day. Right? For a couple weeks, they they were on Slack at every given hour of the day, over weekends as well, just trying to push this forward and and make sure that we have a comprehensive fix. But once they finish their work, then maybe there is a patch available, maybe everyone's happy with it and it protects the PHP code base against these vulnerabilities, then they have to put together an official release. But this is going to be a binary release on the php.net website. Does not necessarily mean that if you're using Ubuntu, if you're using Red Hat, if you're using the the default AWS Linux image, it does not really mean that you have a binary distribution that contains the patch. So to be able to release all of these binary releases in the same time, we gotta give those teams a little time to prepare and tell them, hey. Here's the patch. Here's what you need to do a couple of days before. Make sure that you have a plan and and the the fixed versions available to your customers as well, and make sure that you hold on to them before we actually release this at the set time. Now, at the same time, we realize that with this level of impact, we want to to contact, you know, some of the cloud providers maybe and and some of the web application firewall vendors to tell them, hey. Here here is a set of detection roles that we can work on together to see whether or not attackers are currently using this vulnerability. And that was important for me for for two reasons. First, you know, once the patcher... Patches are out, just like we we discussed before, adversaries can take them, reverse engineer them, figure out where the vulnerability is, and and build an exploit. The other thing is that while working on this coordinated disclosure, we wanted to make sure that there are no attacks in the wild. And that was really critical to us to know, well, hey. Maybe we need to push the pedal even farther. Maybe we need to release patches without coordinating with anyone. Because if people are at risk right now, we wanna make sure that there's a fix available.
00:19:11 · Tomer Schwartz, Rebecca Balan
So we rolled out what's called sort of silent detections or monitoring or just having some better level of visibility to what's what's going on out there. And we worked with a lot of vendors that that have this level of unique visibility to be able to have threat and to know how to make decisions along the way. [00:19:29] S1: Wow. Yeah. Seems like it was an an all hands on deck situation. So, you know, Cloudflare said it saw millions of exploit attempts. Amazon says it saw exploitation about a week before patches were out. I do want to kind of get into like who's still vulnerable, like what is still vulnerable, what works still needs to be done. But before we do, just one thing I wanted to ask you about, because I read that the bugs were spotted in 2024, but they weren't considered exploitable then. So what changed? Like these are, you know, over a decade old. So how does a bug sit for that long?