Recent malware attacks by WannaCry and NotPetya (thought to be Petya as it masquerades as the Petya ransomware that was first discovered in 2016), have caused massive business disruption at thousands of organizations worldwide. Since most malware exploit known vulnerabilities for which a vendor-issued security fix exists, it begs the question: if there was a cure, why was it not applied on the affected computers?
Sometimes, the answer is negligence. The patch was there, but those in the organization simply did not think it was important to install it. Sometimes, it is a resource issue: although there was awareness, not enough resources were made available to perform the patching in time. Other times, however, there is no negligence, quite the opposite: the complexities of the information system infrastructure and the importance to maintain business continuity demand a more measured, cautious approach to the application of any change to the infrastructure, including the installation of security patches. Many security practitioners find themselves in organizations that are in this group. Unfortunately, it is often these kinds of organizations that are likely to be hit the hardest by malware and sustain the greatest damage.
Many security practitioners find themselves in organizations that are in this group. Unfortunately, it is often these kinds of organizations that are likely to be hit the hardest by malware and sustain the greatest damage.
There are a few observations that can be made here.
Installing security patches is risky, but so is implementing other changes
It is possible that simply by installing a security fix, a business application starts malfunctioning or stops working altogether. This can break business processes and cause damage. However, this is true for all software changes, and not unique to security-related fixes. A common approach to managing this risk is delaying mass-deployment of changes, including patches, until all (or at least the most critical) business applications have been thoroughly tested in-house to ensure they all work correctly.
While thorough testing of security patches before mass installation sounds good in theory and is considered the only responsible approach by many, testing in an enterprise setting often takes a significant amount of time (several days or weeks) and as a result, has the unintended consequence of malware popping up in the wild before the patch is applied. Besides, the coverage and depth of such in-house testing, even when it is performed, is usually far from complete, therefore the risk reduction is only partial. The conclusion: patching is risky, but malware bringing your infrastructure to its knees can be devastating. This is a risk that needs to be managed.
When a malware breakout hits, emergency patching suddenly seems acceptable
Experience shows that in the face of an imminent malware threat, even those that are most reluctant to apply security patches without thoroughly testing them first tend to be softened and give a green light to an emergency patch procedure. Why is that? Probably because suddenly the risk balance changes.
Prior to a malware breakout, whether to patch or not is a choice between the slight possibility of a disruption (applying a patch, which is a kind of change and change is always risky) versus another option with zero perceived risk (no change, letting everything run as before – “if it ain’t broke, don’t fix it”). And when there is a malware breakout, immediately there is a high likelihood of significant business disruption caused by the malware, and the risk of breaking something by patching suddenly pales in comparison. After all, security patches are written by the good guys, malware by the bad guys. The problem is that by this time, it is often too late, damage has been done by the malware.
The presence of unpatched computers presents a risk even to those computers that are fully patched
Now, this is where the playing field has begun to change. While one attack vector of a malware may be closed, malicious software with multiple ways of propagating and doing damage present a far greater threat to the IT infrastructure. As a result, reducing the attack surface is more important now than ever, and this should be a great motivating factor toward the comprehensive, timely installation of security fixes and the use of other security technologies such as network access control, to enforce an environment where all devices have the appropriate patches installed.
It is not a question whether patching is necessary, but rather, how to strike a balance between reducing operational risks caused by the deployment of security patches and being quick enough to provide effective protection against occasional outbreaks, realizing that the time window between the detection of a vulnerability and active exploitation in the wild is shrinking dramatically.
Here are a few recommendations.
You can rely on others who have gone before you
Security professionals: you are not alone. There are others out there, too, who are faced with the dilemma of whether or not to install a particular patch. You are not the only one testing it, others are testing, too. And some will even do the unthinkable: install the security patch without testing, trusting the software manufacturers that the patches won’t break everything. Just as you rely on information on Twitter, Facebook and professional portals to learn about the effects of a particular malware, you can use the same resources to identify a potential problem with a security patch.
By way of analogy, if you want to try a new kind of mushroom, your options are not limited to choosing between thorough laboratory testing done all by yourself, or taking a bite and seeing what happens. You can also significantly reduce the chance of dying of poisoning by simply watching others eat it and waiting a day or so to see if they are still OK. It doesn’t remove all risks but still is an effective reduction.
Patching doesn’t necessarily need to be a “big bang”
Except for a few special settings, not all computers in a network are essential to keep the business running. A couple machines that malfunction may cause some inconvenience, but not necessarily present a business continuity challenge. As a result, some companies have chosen to select a small subset of their infrastructure (ideally, machines from several different departments, serving various business functions) that are used to pilot software changes, including security patches.
Changes and patches are installed on these machines and are being closely observed while running them for some time before going ahead with large-scale deployment to the entire infrastructure. While such an approach may not be feasible at all organizations, it can be an alternative way of testing patches that deserves consideration, as a similar approach can dramatically shorten the time to patch.
Early detection and response is of paramount significance
In the event of a malware outbreak, time is essential. As the proverb says, an ounce of prevention is worth a pound of cure – similarly, a minute of early detection and appropriate rapid response can save days or weeks of cleanup and recovery. It is possible, and in fact, strongly recommended, to work toward speeding up the patching cycle, and even introduce technology that ensures only machines with the right security posture are allowed to access the organization’s information resources.
Yet, it is not possible to prevent all attacks, and when a malware outbreak hits, the damage done is often exponentially proportional to the time it takes to discover and successful contain the threat. This time can only be shortened by the use of advanced monitoring technologies with appropriate coverage that are able to look at the various events in near real time, alerting personnel to unusual and potentially risky activities and providing them with the right tools to quickly drill down and understand the underlying causes of anomalies, as well as established response procedures and personnel to carry out these procedures. Prevent attacks as much as possible, and whatever you can’t prevent, do everything you can to detect it early and be prepared to respond immediately.