Lessons from the Front Line: Decreasing Time to Patch

This article is the first in a series from the BlackBerry Incident Response (IR) team examining critical behaviors that either stop incidents from happening or greatly reduce their impact. Due to the recent trend of high-profile critical vulnerabilities (at times on externally facing devices), these lessons are becoming increasingly important. The following examples have kept defenders particularly busy patching and hopefully checking for signs of compromise:

• Citrix - CVE-2019-19781
• F5 - CVE-2020-5902
• Palo Alto - CVE-2020-2021
• Cisco - CVE-2020-3330, CVE-2020-3323, CVE-2020-3144, CVE-2020-3331, & CVE-2020-3140
• Microsoft - CVE-2020-1350

Insight

As IR consultants, we usually become involved in situations that have escalated to the point that in-house teams need additional help — as a result, we see plenty of extreme conditions. Having the ability to investigate the root cause of so many breaches allows us to see key inflection points of incidents that end up being a defining moment, for better or for worse.

Obviously, the best place to stop any attack is before it happens, since this is by far the cheapest and least disruptive option. However, this is not always possible. The primary cause of breaches seen by our incident response team usually stems from one (or more) of three issues: human weakness (phishing), weak external access mechanisms, or unpatched external vulnerabilities. Even if you have a very mature and layered security posture, it is likely that a motivated and well-funded threat actor will eventually gain initial access. All is not lost though, as there are some key mitigations that help stop the majority of attacks or at least limit the damage.

Trends

The first mitigation is as old as the first computer bug, but time and time again this is something overlooked by even some large and experienced organizations: Vulnerability management. Vulnerability management is often considered to be “just” patch management, but the complexity and risk can be enormous. After all, the defender needs to patch everything, and the attacker can benefit from finding just one unpatched vulnerability — if it is the right one.

In fact, in our recent increased trend of IR investigations resulting from the Citrix Vulnerability assigned CVE-2019-19781, we noticed that threat actors (TA's) were front loading their exploit efforts. In essence, taking advantage of the publicized vulnerability in an accelerated time frame, not with the intention of carrying out their objectives at the time (due to their own bandwidth limitations), but to install backdoors and revisit at a later time.

It is a race to gain a foothold into the environment before externally facing systems can be patched. Once the backdoor is in place, patching the original vulnerability is not enough to remove the attacker’s access to the environment.

In the case of the most recent F5 (CVE-2020-5902) vulnerability, NCC saw scans on their honeypot less than three days after the vendor disclosed the vulnerability, initial exploitation in under four days, publicly released exploits (including Metasploit) available after only five days, and increased exploit activity after only six days. The timing of this vulnerability was also problematic with the situation rapidly evolving just before and during the July 4^th holiday weekend, which may have slowed down the responsiveness of security teams.

There are various recommendations that provide rough timeframes for applying and verifying patches based on vulnerability risk. However, some organizations still operate on the assumption that they are safe until an exploit has been publicly released or systems are being exploited in the wild. We saw this with Citrix CVE-2019-19781, where nearly a month and half after the vulnerability was announced it was estimated that one in five organizations had still not patched:

Figure 1: Perceived safe period to patch vulnerable systems

However, this patching timeframe is drastically reduced due to various threat actors sprinting to create workable exploits and drop backdoors en-masse at multiple organizations. The risk of a reduced timeframe should be considered especially if the vulnerability is easily exploited:

Figure 2: Actual risk in vulnerability disclosure and patching

Key Takeaways:

• Patch vulnerabilities as quickly as possible, in particular, ones that are externally facing and easily exploitable.
• Consider the possibility that threat actors may have already compromised your environment and patching may only afford you a false sense of security unless you have the visibility and capability to hunt for IOCs associated with the vulnerability.

 Tips for Creating an Effective Vulnerability Management Program:

• Ensure the vulnerability management program has executive support
• Assign an owner who bears responsibility and takes it seriously
• Track the latest vulnerabilities, patches, and exploit releases
• Discover, learn, and begin to know your environment
• Scan everything, scan completely, scan often
• Prioritize vulnerabilities by:
  • Risk
  • Publicly available exploits
  • System criticality
  • Public exposure
• Assign the vulnerabilities to system owners and track them to remediation
• Rescan and validate the remediation
• Track the environment’s current state and progress — share with stakeholders
• Wash, rinse, repeat
  

Conclusion

Vulnerability management is not made any easier by the fact that most externally accessible applications are often client-facing, revenue generating and/or critical services. Patching these systems has an inherent risk of disruption.

Even if you have been lucky enough to have never experienced a breach or ransomware infection, it does not mean that it will never happen. If it has happened, the pain of recovery stings sharply and fades over time — so never let a “good incident” go to waste. If your organization knows this pain, use it as fresh perspective to adjust the company’s appetite for this risk. If you have not learned first-hand from an incident, we hope that our articles assist in adjusting priorities and providing quick tips to make a big difference.