BlackBerry’s Fact Check alarms were sounding this week as vendors set their hype meters to maximum volume to claim “victory” in the recent MITRE ATT&CK® Evaluations. Facts were among the first casualties.
Fact Check: With MITRE Turla Testing, Your Results May Vary
Here are a few facts to bear in mind as you navigate this latest post-MITRE hype-fest:
1. FACT: MITRE Turla threat emulation testing was conducted with 31 cybersecurity vendors participating. Charts produced by marketing teams that show only a subset of the participating vendors are inherently misleading and apparently designed to throw certain competitors under the proverbial bus.
MITRE does not provide such stack-ranked test results, specifically to discourage these kinds of targeted assaults that pit one vendor against its primary competitors. Any participant that fabricates such tables is acting in contradiction to the spirit of the testing, instead using it in a transparent attempt to score points against competitors they wish to disparage.
2. FACT: The recent MITRE tests did not include managed services testing, which is actually scheduled for the next round of MITRE tests. Beware of any vendor that, in squawking about “100%” scores on certain sections of the MITRE Turla tests, casually adds that they also scored well “in the Managed Security Services Providers testing.” The MSSP testing they are referring to actually took place last year and did not involve Turla emulation. They are mixing and matching results from entirely different tests, and hoping no one will notice.
3. FACT: There is no such thing as 100% protection. Anyone who claims otherwise either does not understand the nature of cyber threats, or is deliberately trying to mislead you. While MITRE did publish test results on a 100% scale, the evaluators will be among the first to tell you that achieving such a score on any section of the test is no guarantee of protection in the real world.
4. FACT: No vendor scored 100% on all areas of the MITRE Turla emulation, and those who have cherry-picked “perfect scores” to promote are leaving out areas where they fared poorly. Check MITRE’s official published results to get the full story.
5. FACT: Different vendors approach this testing with different goals in mind. While the MITRE ATT&CK Evaluations testing revolves around detection and visibility, there can be a point of diminishing returns. In the real world of threat protection, both visibility and protection are paramount. At the end of the day, the biggest question is the outcome: Did you stop the progression of the attack?
Setting the Record Straight
BlackBerry is the longstanding market leader in security innovations and the pioneer of AI cybersecurity. Our Cylance® AI was built from the outset to provide world-class protection capable of stopping Turla and other sophisticated APT (advanced persistent threat) attacks faster and more efficiently.
The MITRE ATT&CK Evaluations put emphasis and focus on detection and while this is important, this can also introduce a significant amount of noise into the SOC. Achieving 100% ability to issue alerts does not equate to effective threat protection. Organizations need to have a balance between automated defensive actions and signal noise. CylanceENDPOINT™ utilizes AI to achieve best-in-class protection while simultaneously reducing the alerting load that’s burning out your security teams.
By design, our Cylance AI makes automated decisions on behalf of the user to effectively stop attacks before they can cause damage and with fewer alerts, reducing alert fatigue for security teams.
FACT: In MITRE tests, CylanceENDPOINT demonstrated it was able to successfully protect against Turla, disrupting attacks early in the kill chain before they were able to do any damage.
BOTTOM LINE: It is risky to draw direct comparisons of products from the testing data alone. Users are advised to look at the actual test data, draw their own conclusions, and ask vendors to explain any apparent discrepancies between what MITRE has reported and how the vendor subsequently interpreted that data for marketing purposes. Even MITRE makes this point.
Finally, users should be advised that even if a vendor claims a perfect score on a test, your actual results may vary.
For similar articles and news delivered straight to your inbox, subscribe to the BlackBerry Blog.
Related Reading
- Introducing BlackBerry Fact Check
- Microsoft Defender vs. CylanceENDPOINT
- Three Words that Describe BlackBerry in 2023
- Customers Have Spoken: BlackBerry Named a Customers' Choice for Endpoint Protection Platforms on Gartner® Peer Insights™
- BlackBerry Summit Speaker Lineup: U.S. DHS, Government of Canada, Mercedes-Benz, AWS, Adobe, Siemens, Spotify and More
