00:12:40 · Tomer Schwartz
[00:12:40] S2: There's always learnings out of this, and coordinated disclosure is something that is fundamentally flawed. We want to bring more and more people into the inner circle to make sure that defenders are aware of the the vulnerability and they have time to prepare. But then on the other side, we can't really announce it to the world without adversaries know... Knowing about this. And this is not a model that we invented, obviously. This this is a problem in the cybersecurity industry for decades. The question of how to do it well and how to do this in a very timely process, I think, is is right now a big question that the industry needs to tackle with the the the imminent vulnpocalypse that we're seeing right now. So I think it was a a a pretty fast turnaround time compared to to all the things that needed to happen. From detection to fix, I think it was roughly two months, which is pretty amazing given the fact that this this was a a very unique architecture flaw within PHP that was was lurking there for twelve years. Right? It was it was very diff... Difficult for us to come up with a solution that is reliable enough, that covers every single aspect of this unique vulnerability, that is backwards compatible with some of the extensions that PHP has. There were a lot of different constraints that we had to operate in. And given the fact that we couldn't really bring in a lot of people into that inner circle, I think we managed to do this, again, comparably pretty quickly. The other side of this is what happened towards the end of that cycle where we realized, well, this is something that can take the Internet down quite literally. So we got to work with the cloud providers, with the the web application firewall providers, with, you know, lot of different vendors that are associated with this that need to be aware so they can make sure that they release on time, they have the the detection rules out, that we have we have the full set of visibility and and potential... Not full defense, but partial defense against what's what's gonna happen. And once you realize how big this industry is, you realize that, you know, maybe that inner circle is all all of sudden hundreds of people. And maybe they're talking to some some of their colleagues because they have to pull them in to be able to operate. So I don't I don't blame anyone for this leaking.
00:15:00 · Tomer Schwartz, S0, Rebecca Balan
It's just... It's it's it's a function of how coordinated disclosure is happening. And luckily, we were pretty much ready to that situation. We had sort of a plan b that we we could have operated bay... Based off of. And I think that we made the right decisions in in in every given point in time. [00:15:18] S0: Every company today is looking for an edge. Better insights, faster decisions, greater resilience. But as AI drives new growth, it also creates new demands on energy and infrastructure. Schneider Electric is connecting energy, automation, and digital technologies to help organizations make smarter decisions and operate more efficiently. Because the future won't be shaped by AI alone, it will be shaped by energy and industrial intelligence. Schneider Electric, advancing energy tech. [00:15:45] S1: So just to like take a step back for coordinated disclosure for anyone who isn't familiar with this, researchers will privately notify software maintainers, give them time to prepare a fix before the vulnerability is made public. Right? And so PHP Foundation would have given big companies and hosts a bit of a head start. Maybe some people had already, you know, implemented a fix or started to before it became public. I don't know.
00:15:45 · Rebecca Balan, Tomer Schwartz
[00:15:45] S1: So just to like take a step back for coordinated disclosure for anyone who isn't familiar with this, researchers will privately notify software maintainers, give them time to prepare a fix before the vulnerability is made public. Right? And so PHP Foundation would have given big companies and hosts a bit of a head start. Maybe some people had already, you know, implemented a fix or started to before it became public. I don't know. [00:16:07] S2: Yeah. So the intention wasn't to give certain companies a a preferential treatment and tell them, hey. Go patch this because, you know, we feel you're... That you're more important than than the rest. That was not our intention, and we were very clearly communicating this. I think that our intention was to say, for those choke points that really have massive leverage for the part of the Internet, the part of the industry that relies on PHP, we gotta make sure that at the very least, they're ready for the release. Right? So what I mean by that is let's talk about how distribution works in open source. Right? So the PHP Foundation is a foundation. It has an amazing team that maintains the PHP code base. [00:16:57] S1: A team of volunteers. Right? Well,
00:16:59 · Tomer Schwartz
[00:16:59] S2: I I I can say for certain, I think they're volunteers, but, you know, if if... We need to ask the PHP Foundation about about how it works. I think I think they are. But, basically, we worked with them night and day. Right? For a couple weeks, they they were on Slack at every given hour of the day, over weekends as well, just trying to push this forward and and make sure that we have a comprehensive fix. But once they finish their work, then maybe there is a patch available, maybe everyone's happy with it and it protects the PHP code base against these vulnerabilities, then they have to put together an official release. But this is going to be a binary release on the php.net website. Does not necessarily mean that if you're using Ubuntu, if you're using Red Hat, if you're using the the default AWS Linux image, it does not really mean that you have a binary distribution that contains the patch. So to be able to release all of these binary releases in the same time, we gotta give those teams a little time to prepare and tell them, hey. Here's the patch. Here's what you need to do a couple of days before. Make sure that you have a plan and and the the fixed versions available to your customers as well, and make sure that you hold on to them before we actually release this at the set time. Now, at the same time, we realize that with this level of impact, we want to to contact, you know, some of the cloud providers maybe and and some of the web application firewall vendors to tell them, hey. Here here is a set of detection roles that we can work on together to see whether or not attackers are currently using this vulnerability. And that was important for me for for two reasons. First, you know, once the patcher... Patches are out, just like we we discussed before, adversaries can take them, reverse engineer them, figure out where the vulnerability is, and and build an exploit. The other thing is that while working on this coordinated disclosure, we wanted to make sure that there are no attacks in the wild. And that was really critical to us to know, well, hey. Maybe we need to push the pedal even farther. Maybe we need to release patches without coordinating with anyone. Because if people are at risk right now, we wanna make sure that there's a fix available.
00:19:11 · Tomer Schwartz, Rebecca Balan
So we rolled out what's called sort of silent detections or monitoring or just having some better level of visibility to what's what's going on out there. And we worked with a lot of vendors that that have this level of unique visibility to be able to have threat and to know how to make decisions along the way. [00:19:29] S1: Wow. Yeah. Seems like it was an an all hands on deck situation. So, you know, Cloudflare said it saw millions of exploit attempts. Amazon says it saw exploitation about a week before patches were out. I do want to kind of get into like who's still vulnerable, like what is still vulnerable, what works still needs to be done. But before we do, just one thing I wanted to ask you about, because I read that the bugs were spotted in 2024, but they weren't considered exploitable then. So what changed? Like these are, you know, over a decade old. So how does a bug sit for that long?
00:20:00 · Tomer Schwartz
[00:20:00] S2: Well, there are many parts of many code bases that for those who are very familiar with these code bases, they can say, well, this area is a little more sensitive than others. But there's a big difference between a a simple bug, a QA issue, if you will, and something that can be leveraged by by attackers. It's it's it's very different situations. I think that even though, you know, information about these kinds of problems have appeared as early as 2020 and and even before, and when we talked to the PHP team, they said that they're... They were thinking about this particular area as a somewhat problematic area. No one could have built an exploit for this. Right? No one reported that they have a fully working exploit that manages to trigger these bugs remotely in a reliable way and in a way that can be abused by attackers. I think what happened is, again, it goes back to AI. Now that we can not only spend our time and energy, but also spend a countless amount of tokens and cognitive load that we can of move between different ideas. It gives us the ability to scale and try to build exploits in a much much different way. So I think what happened is that all of a sudden, Us defenders have such great tools to be able to try all of those things and and and cover the basics, if you will, and all of those potentially sensitive areas that we can surface a unique set of of new vulnerabilities. Right? And and this is why we keep saying that this is not the only critical vulnerability that is about to be published in the next couple weeks or months. Right? I think that we... The industry was used to this yearly cycle where maybe we had a one headliner per year of, like, the the most critical vulnerability that was found in that year. I I think that cadence is very quickly changing.