BlackBerry Effective Against BlackCat/ALPHV and menuPass in MITRE ATT&CK Evaluations

Evaluating the Efficacy of Managed Detection and Response (MDR) Services in the Dynamic Threat Landscape of 2024.

The dynamic threat landscape in cybersecurity presents a constant challenge to organizations. From the development of advanced malware to more elaborate social engineering techniques being used by threat actors, defenders must maintain constant vigilance and stay up to date with the latest security strategies to keep assets safe and secure resources.

To help organizations combat today's sophisticated cyber threats and enhance their threat detection capabilities, MITRE conducted a managed detection and response (MDR) evaluation that emulated two notable threat actors: menuPass and BlackCat.

Highlights of BlackBerry Results

Here are some of the achievements of BlackBerry’s Cylance^® AI during this evaluation:

Actionable Detections: BlackBerry placed in the top five for Actionable Detections, a significant ranking that underscores our comprehensive approach. The CylanceMDR™ solution by BlackBerry, supported by the Cylance portfolio, detected threats from the very beginning of the attack chain.

Alert Fatigue Prevention: BlackBerry issued 75 percent fewer alerts than the vendor who issued the highest number of alerts during the evaluation. BlackBerry delivered actionable alerts at each critical point of the attack chain, thereby providing prompt insights for the customer and preventing alert fatigue.

Best-in-class Visibility: Through a combination of detected activities, observed tactics, techniques, and procedures (TTPs) plus forensic analysis, the BlackBerry team accurately identified the threat actors menuPass and BlackCat/ALPHV at the earliest stages in both emulated scenarios, along with the specific malware they used.

Protective Security: CylanceMDR successfully detected and provided remediation suggestions for the most critical steps of the attack in both campaigns.

Above and Beyond: The BlackBerry team not only provided alerts for urgent actions and sent timely escalations, but also provided detailed daily reports summarizing each day's activities, showcasing BlackBerry’s commitment to thorough analysis and clear communication.

About the Evaluation

MITRE Engenuity’s ATT&CK^® Evaluations: Managed Services — Round 2 represents a pivotal benchmark in assessing the capabilities of vendor participants in analyzing and describing adversary behavior within the MITRE Framework. This evaluation serves as a test for the efficacy of Managed Detection and Response (MDR) solutions in combating sophisticated cyber threats.

Although there were 11 vendors in Round 2 of testing, BlackBerry is one of just eight vendors in the market that have participated in both MITRE evaluations for MDR/Services. During this second evaluation, participants utilized a self-supplied toolset to demonstrate their service capabilities, providing analysis in the same format they would offer to their customers.

What makes this evaluation particularly challenging for participants is its “black box” approach to adversary emulation. Participants are kept in the dark regarding the emulated adversary or adversaries until after the execution is complete.

This round of testing also included a new category: Actionable Detection. MITRE was looking for a timely “scenario detection,” with clear, specific information such as activity, time, location, users and context, enabling a security team to take immediate and effective action to mitigate or respond to a detected threat.

Conducted over a rigorous five-day period in a cloud computing environment (AWS), the complete evaluation emulated two notable threat actors and demanded that participants provide their analysis as if MITRE Engenuity were one of their customers.

Meet the Adversaries: menuPass and BlackCat

Both menuPass and BlackCat are sophisticated entities within the cyber threat landscape; however, their operational tactics and techniques demonstrate distinct differences:

menuPass, also recognized as APT10, primarily engages in state-sponsored espionage, concentrating on gaining long-term strategic access and extracting data across numerous global industries, with a significant emphasis on Japanese targets. Members of menuPass are known to have acted in association with the Chinese Ministry of State Security (MSS). Their approach is methodical, using spear-phishing and custom malicious open-source access tools like QuasarRAT to infiltrate and persist within their targets' networks.

BlackCat, known to operate as ALPHV, represents the new wave of ransomware-as-a-service (RaaS). This prolific Russian-speaking group is notorious for its aggressive monetization strategies, leveraging ransomware attacks that threaten data encryption and leakage to extort money from its victims, often impacting businesses regardless of their geographic or sectoral alignment. While menuPass prefers stealth and sustained infiltration, BlackCat prioritizes rapid financial extortion and public disruption.

Test Results for CylanceMDR

MITRE Engenuity rigorously tested 11 MDR solutions for five continuous days, running 24 hours per day, with this round marking the second iteration of this closed-book evaluation. In this case, the evaluation was divided into two adversary groups: menuPass and BlackCat.

This evaluation tests a vendor’s ability to detect threats that prioritize stealth, leverage trusted relationships and system tools, and inhibit data recovery. The assessment consisted of 15 major steps, encompassing a total of 173 sub-steps, to emulate a complete attack chain.

The CylanceMDR™ solution by BlackBerry, supported by the Cylance portfolio, successfully detected and provided remediation suggestions for the most critical steps of the attack. Out of 11 MDR solutions tested, BlackBerry placed in the top five for Actionable Detections, an evaluation category which assesses the clarity, context and practical guidance of alerts for effective threat response.

Our top five placement highlights our proactive efforts both in reporting potentially malicious activity and in providing detailed context and thorough remediation information.

Out of the 173 sub-steps in the test, 43 key steps were selected by MITRE for final scoring. BlackBerry successfully addressed 35, showcasing the robustness of our solution. It is important to remark that for the eight remaining steps, BlackBerry was fully aware of the attack. As one example, for step 1.B.4 on Reflective Code Loading, we successfully identified the injected Notepad++ process but did not automatically report the in-depth technical details to the customer, as this is not part of our standard procedure.

Despite this, there was extra data in the reverse engineering reports that were produced internally. These reports can be made available to a customer upon request, but in the case of this emulated attack, simply identifying and reporting this data did not fulfill the detection criteria requested by MITRE.

In the context of the MITRE evaluation versus a real-world attack, however, there would have been low to no impact to the customer, as all suspicious artifacts were immediately escalated as malicious due to their behaviors and the activity they performed.

BlackBerry vs. Alert Fatigue

In this round of testing, MITRE graded vendors on Total Alerts sent. This part of the evaluation measured the total volume of alerts received, regardless of whether they pertained to actual malicious attacks, regular user activity, or false positives.

While alerts are important, they can also introduce a significant amount of noise into the SOC. This is a crucial point to understand: Issuing alerts on 100 percent of issues does not equate to effective threat protection. Organizations need to strike a balance between automated defensive actions and signal noise.

During the five-day period of the evaluation, CylanceMDR issued 1207 fewer alerts than the vendor who issued the highest number of alerts during the test, or 75 percent fewer.

By design, BlackBerry products powered by Cylance AI make automated decisions on behalf of the user to preemptively stop attacks before they can cause damage, striking a balance between the volume of robust and actionable reported events, and a lower number of alerts.

This optimal balance helps minimize alert fatigue for security teams, ensuring the MDR service remains both effective and manageable, regardless of the size of the organization or security team.

BlackBerry Results: In Depth

Infrastructure

For this year’s MITRE Evaluation, the environment was designed with added complexity in comparison to those of previous evaluations, simulating an enterprise network that included trusted networks across multiple buildings and external connections to contractors.

• The first scenario emulated menuPass’s compromise of two subsidiaries of a fictitious global pharmaceutical company. The emulation replicated menuPass’s use of living-off-the-land techniques, reflective code loading to evade detection, and DLL side-loading to execute the SigLoader, FYAnti, QuasarRAT, and SodaMaster payloads in memory.
  
  
• The second scenario introduced a cohabitating ALPHV BlackCat affiliate’s deployment of BlackCat ransomware to Windows^® and Linux^® ESXi servers in one of the subsidiaries, highlighting defense evasion, data encryption/destruction, and system recovery obstruction behaviors.
   

Figure 1: MITRE MDR Round 2 complex infrastructure layout.

Solution Deployment

One of the key aspects of CylanceMDR is the simplicity of our installation in both workstations and servers, with quick deployment and fine-tuning that ensures a smooth start to operations in a short period of time.

The MITRE MDR environment was no exception, as we were able to deploy and configure CylanceMDR in less than two business days.

The baseline for CylanceMDR is the usage of CylancePROTECT® and CylanceOPTICS®, two best-in-class products powered by Cylance AI. Using these advanced technologies, our analysts can, on demand, deploy any extra forensic tool they need on any given endpoint or server via our Python Refract Package Interface.

This is how our team deploys utilities such as memory forensic tools for in-depth analysis and the recovery of malicious samples that were deleted by the attacker after execution.

Threat Detection and Response Performance

Through a combination of detected activities and observed TTPs in both campaigns, plus forensic analysis, the BlackBerry team accurately identified the threat actors menuPass and BlackCat/ALPHV, along with the specific malware in use.

As mentioned earlier, MITRE emulated a total of 15 steps, with 173 sub-steps, primarily focusing on 43 core sub-steps for measuring activities. Here’s a brief overview of the process:

• The chained attack commenced with the use of stolen credentials to access a subsidiary’s server, where the emulated threat actor menuPass deployed malware to establish a persistent foothold.
• Through the use of QuasarRAT, they conducted reconnaissance to discover critical network components.
• The attackers escalated privileges using captured credentials and moved laterally to more valuable targets within the organization.
• By leveraging various sophisticated tools and techniques, they exfiltrated key data, including the Active Directory database, and prepared for further lateral movement.
• Ultimately, the attackers collected and exfiltrated sensitive information from the network, covering their tracks by clearing event logs to avoid detection.

BlackCat Attack

• BlackCat’s attack began when an access broker infiltrated a contractor organization using stolen credentials (from the previous scenario), providing remote desktop protocol (RDP) access to a critical network host.
• Once inside, the attackers used discovery tools to map out the network and harvest credentials.
• BlackCat then disabled security measures, escalated privileges, and moved laterally across the network to identify and exfiltrate sensitive data.
• The final phase of the attack involved deploying ransomware payloads on both Linux and Windows systems, encrypting files and leaving ransom notes.
• This multi-stage attack culminated in widespread data encryption, significantly disrupting the victim organization’s operations.
   

Figure 2: Managed Detection and Response attack chain timeline.

Speed and Accuracy of Threat Containment and Remediation

The MITRE MDR Evaluation comprised 15 steps, during which BlackBerry demonstrated proactive threat detection by alerting at very early stages in both scenarios, as shown in Figures 3 and 4 below.

Figure 4: Threat detection and actionability in attack chain – steps 10-15 conducted by BlackCat/ALPHV.

As seen in figures 3 and 4 above, CylanceMDR detected threats from the very beginning of the attack chain. If the BlackBerry Incident Response (IR) team had been permitted to act in response to these detections, the suggested fixes or the preventive measures provided by the products themselves would have blocked this attack from the first step.

On each alerted step, the BlackBerry IR team provided MITRE with context, remediation suggestions and specific details of the observed activity.

Although MITRE evaluated only 45 of the 173 operational sub-steps, our early detection of the attack chain would have rendered the eight unreported steps irrelevant in a real-world attack scenario.

Actionable Reporting

Thanks to the diligent efforts of BlackBerry MDR analysts who escalated cases, and the dedicated team working tirelessly in the background to gather evidence, BlackBerry successfully fulfilled the new "Actionability" metric introduced by the MITRE Team for nearly every step.

The BlackBerry team went above and beyond requirements, not only by sending timely escalations but also by providing daily reports.

At the conclusion of the activity, we delivered a highly detailed 173-page report accompanied by several appendices and diagrams of the attack to provide visual detail. This comprehensive analysis thoroughly described the attack chain, the impact of the breach, and the full timeline of events, showcasing BlackBerry’s commitment to thorough analysis and clear communication.

The MDR Journey: Team Preparation and Constant Training

To ensure BlackBerry MDR services consistently perform at their best, we provide continuous training for our analysts and reverse engineers. Simultaneously, we keep our products, telemetry and detections finely tuned by frequently conducting real-world threat emulations using the latest and most significant threat actors and tools.

Our methodology is focused on delivering positive security outcomes, designed to help customers safeguard their organizations. We enhance this approach with monthly purple teaming exercises, rigorously testing our product capabilities in various scenarios. This methodology aligns with the classic "Think red, act blue" strategy.

Thanks to our proactive mindset and constant preparation, the MITRE MDR evaluation was seamlessly integrated into our routine operations. We treated it as business as usual, deploying our solution and allowing our MDR team to operate just as they would for any customer.

Figure 5: The BlackBerry MDR journey.

MITRE ATT&CK Evaluations vs. Real World Attacks

It is important to note that MITRE does not rank or score as part of their evaluations. Although numbers can be useful from a security practitioner’s perspective to show how our products measure up, context and timing are the most relevant pieces of information brought to light by MITRE’s testing.

While the MITRE ATT&CK Evaluations testing revolves around detection and visibility, there can be a point of diminishing returns. Although MITRE publishes test results on a 100 percent scale, the evaluators will be among the first to tell you that achieving such a score on any section or category of the test is no guarantee of protection in the real world. 

In a real-world scenario, there is no such thing as 100% protection. Anyone who claims otherwise does not understand the nature of cyber threats. At the end of the day, the biggest question is the outcome: Did you stop the progression of the attack?  

Conclusion

The BlackBerry team effectively delivered actionable alerts at each critical point of the attack chain, thereby providing prompt insight for the customer while preventing alert fatigue. BlackBerry provided immediate alerts for urgent actions, and detailed daily reports summarizing each day's activities.

Additionally, at the conclusion of the engagement, the team presented a highly detailed and exhaustive final report. This report included a detailed timeline of activities, reverse engineering reports for the dropped artifacts, and visual diagrams to enhance the customer’s understanding of the attack. This underscores the customer-centric nature of the CyanceMDR solution, leveraging its simplicity and the expertise of our analysts to deliver superior service.

A Note of Thanks

We’d like to thank the MITRE Engenuity team — in particular, the ATT&CK Evaluation team — for their continuous support, and for helping all vendors elevate their products and services to the next level, for the common benefit of the security industry.

Related Reading

• Top 20 MITRE ATT&CK Tactics Threat Actors Use Now
• BlackBerry AI Cybersecurity Effective Against Turla
• BlackBerry Demonstrates Prevention Against Wizard Spider and Sandworm Emulated in MITRE ATT&CK Evaluations

_{Disclaimer: The views and opinions expressed in this blog are those of BlackBerry, and do not necessarily reflect the views or positions of any entities they represent.}