On Tuesday, 20 October, I was scrolling through Mastodon in the morning. Many of the accounts I follow work in infosec, just like I do. That’s where I first heard about the PHP2Shell vulnerability: details were unclear, but all PHP installations were affected. Pre-auth remote code execution, so absolutely critical. No fix available yet, but there were rumors about the first exploits being developed already. A patch would be released at 4 pm; until then, you should take anything running PHP and connected to the internet offline. Okay, I thought, this might be a busy day. As a reminder, I’ve only recently ended up in vulnerability management.
Measure 1: I figured out how to take the two old WordPress sites I still run offline. I also took my wife’s WordPress blog offline and informed her manager, since her company is still running on WordPress too (not for much longer though).
Measure 2 came with the start of the working day. Among other things, I looked at the recommendations for handling the vulnerability on the websites of Huntress, Watchtowr and PHP itself. At the same time, I informed my client about the vulnerability and developed an action plan before the daily meeting a little later. I have to give my client credit: the issue was immediately given high priority.
We immediately set about identifying the PHP installations and, as a first step, confirming our assessment that no PHP code was running in publicly accessible network zones (DMZs). That made things considerably less stressful.
What was interesting about this first day of working on php2shell was that the usual vulnerability-handling process doesn’t work in cases like this. The vulnerability scanner didn’t know about the vulnerability yet, since it hadn’t been published as a CVE. Accordingly, the scanner couldn’t list the PHP installations that were affected. Instead, part of my day was spent searching huge CSV files for the string “PHP”. The downside of such short names is that they often also appear in base64-encoded or other high-entropy strings.
Where we found PHP files, we contacted the relevant departments and asked for details. For example, it wasn’t clear in every case whether these were active PHP installations or merely software modules that might not even be enabled. In one case, we were actually told that PHP wasn’t running on the supposedly affected software. It was only when I showed the contact the installation paths I’d found that they realised PHP obviously was running there after all. I’m afraid this is one of the most unpleasant aspects of this vulnerability: PHP runs in the background on such a wide variety of systems that even the people operating them are not necessarily aware of it.
While all the PHP installations were being catalogued in the wiki, the relevant departments basically had two options. If they operated the PHP installations themselves, they were to schedule a window for the update as soon as the patch became available — availability notwithstanding, a vulnerability like this requires the patch to be installed immediately. Where PHP instances were bundled into a third-party software package or even a vendor appliance, the relevant contacts were to get in touch with the vendor immediately and let us know when a patch would be made available. Where that takes too long, we’ll implement mitigations, or further restrict access to the affected servers.
I actually find it fascinating to experience a process like this from end to end. My impression is that my client is reasonably well prepared. Nobody went into the dreaded “headless chicken mode”; everything seemed to follow an orderly process, which moved fairly quickly for such a large organisation in the financial sector — not without a few escalations along the way. What did I learn from this? I’d say the following:
- Predefined processes really do prevent chaos and panic
- Having to deal with vulnerabilities before they have been formalised and published is an important exception that organisations need to be prepared for
- You need technical solutions to find every installation of components such as PHP, because even the people responsible for the relevant hosts often can’t tell you off the top of their heads whether the component is running on them.