There are many benefits of being part of a professional services group; one of them is the ability to see trends across multiple organizations and industries. The first trend that the BlackBerry Incident Response (IR) Team is seeing is an uptick in breaches perpetrated from exploiting CVE-2019-19781.
If this number is not seared into your mind already, it is a critical vulnerability found in the Citrix/Netscaler Application Delivery Controller (ADC), Citrix/Netscaler Gateway, Citrix SD-WAN WANOP appliances, and even includes Virtual Appliances (VPX) hosted on any Citrix Hypervisor including ESX, Hyper-V, KVM, AWS, GCP, Azure, SDX appliances. Successful exploitation allows an attacker to execute arbitrary commands, often leveraged to establish backdoor access to the environment serving as a beachhead for further access.
Additionally, the BlackBerry IR Team noticed a recent change in attacker Tactics, Techniques, and Procedures (TTPs) where a new method of persistence identified on a compromised Citrix host used some interesting and clever techniques to decrease the need to touch disk. This is covered at a high level below and will be detailed in a future, more comprehensive article.
Figure 1: Backdoor execution process
A third, more alarming, trend seen by the BlackBerry IR team is the number of victims who believed they had remediated this vulnerability but were still compromised. This is the primary reason for raising awareness and providing guidance on detection and mitigation. One potential reason for the false sense of security is that once a host is compromised, closing the initial vulnerability is not enough to remediate any backdoors that may have already been installed; this is akin to closing the proverbial door after the horse has bolted.
The vulnerability was first announced by Citrix on December 17th, 2019, and security researchers quickly confirmed the exploitability, though no proof of concept (POC) code was initially released. Publicly available POC code was first released on January 10th, 2020, followed shortly by a number of separate but additional POC releases by various parties in the security community. Honeypot Operators indicated that they observed an increase in resource requests related to the vulnerability shortly after the POC code releases.
For most organizations, the primary window of vulnerability is the time between the public POC code releases on January 10th and when they finally mitigated the vulnerability. However, given that the exploit was developed independently by a number of parties after its existence was announced, everyone should consider the possibility of advanced attackers developing their own exploit code independently and taking advantage of this vulnerability before the release of public exploit code available on January 10th.
It is also worth noting that in addition to the window of opportunity previous described there was a firmware bug acknowledged by Citrix on January 16, 2020, that proved that the initial mitigation was ineffective on some Citrix ADC Release 12.1 builds. This provided attackers an additional opportunity to exploit the vulnerability and gain a foothold in the victim’s network.
If you have any of the following products, please take some time to double and triple check that they are sufficiently remediated, tested, and checked for indicators of compromise (source):
- Citrix ADC and Citrix Gateway version 13.0 all supported builds before 13.0.47.24
- NetScaler ADC and NetScaler Gateway version 12.1 all supported builds before 12.1.55.18
- NetScaler ADC and NetScaler Gateway version 12.0 all supported builds before 12.0.63.13
- NetScaler ADC and NetScaler Gateway version 11.1 all supported builds before 11.1.63.15
- NetScaler ADC and NetScaler Gateway version 10.5 all supported builds before 10.5.70.12
- Citrix SD-WAN WANOP appliance models 4000-WO, 4100-WO, 5000-WO, and 5100-WO all supported software release builds before 10.2.6b and 11.0.3b
Testing for the Vulnerability
If you have one of these appliances or virtual appliances, you can test for the vulnerability using a number of different tools – including one verification tool provided by Citrix:
Figure 2: Example execution and output from the Citrix vulnerability check script
The Metasploit Framework also includes a module to scan for the presence of the vulnerability.
Remediation
If a system is found vulnerable, please follow the remediation steps provided by Citrix. Even after performing these steps, remember to rerun the vulnerability test above to verify that the fix was effective.
Additionally, we recommend following the principle of least privilege for any domain accounts used to administer Citrix appliances. We observed a distinct difference in outcomes from incidents that allowed easy lateral movement into the enterprise environment from elevated credentials stored on the Citrix device. A review of common security best practices including network segmentation is also advisable.
Indicators of Compromise (IoCs)
While double checking remediation efforts, we recommend checking for signs of compromise. The good news is that there are tools and manual checks available for this as well. The easiest being the scanning tool Citrix co-released with Mandiant for identifying common IOCs. This will provide both indicators of compromise and indicators of attempted vulnerability scanning.
It should be noted that running this tool is a good starting point but not a 100% guarantee that it will identify all compromises. If concerned about a compromise from this or any other vulnerability, it is best to engage in a full compromise assessment.
Additional manual analysis can involve checking for the following:
- Persistence mechanisms, in particular looking for unknown crontab entries
- Cron history for suspicious entries, our IR team discovered scripts that utilize curl to download files and execute them in memory
- Shell history logs for indication of malicious activity including the clean up of files, such as the XML templates, related to the initial compromise
- Web server access logs for indications of malicious activity
- Anomalous files including some of the more well-known backdoors like “NOTROBIN”, xml templates, scripts and web shells
- Anomalous running processes including those running as “nobody”
It is also worth noting that the BlackBerry IR team has seen some uncertainty in interpreting the results from the various security tools available. For example, scanning output indicating that a compromise may have taken place, but this was either not identified as a risk or not properly investigated. We have also seen cases where victims have been unclear of the purpose of the scripts executed and the context of what the output means.
For example, instances where the verification tool whose only purpose is to confirm if devices are vulnerable to CVE-2019-19781 was mistakenly used to determine if a compromise had occurred. If further verification or assistance is needed in understanding the output from any analysis or scanning tools, we recommended contacting your trusted security partner and/or IR retainer service.
Conclusion
Due to the uptick in Incident Response inbounds resulting from a CVE-2019-19781 exploit, we hope that this public service announcement reminds everyone to check and double check against the existence of this vulnerability and for potential compromise. In our next article, we will cover the offensive side of discovering and exploiting this vulnerability using real-world examples from the BlackBerry Cylance Red Team.
