NSS Labs has just released their first advanced endpoint test, entering the organization into the advanced endpoint testing market, and we are pleased to announce that Cylance has earned the NSS Labs Recommended Rating for Advanced Endpoint Protection (AEP).
This is by far the most comprehensive advanced endpoint security public test to date, and as with all new endeavors, there will be growing pains - some minor and some major. Given that this was NSS Labs’ first test, there were many execution challenges, delays, and also (speaking frankly) a series of unexpected surprises.
In this hyper-competitive world of endpoint security, the merits of testing will always be dynamic. We at Cylance will continue to work with the testing industry to evolve and improve testing to meet the expectations of consumers when reading these testing reports. To that end, because we have been publicly advocating for changes in testing methods, we were pleased to see NSS Labs involve vendors in reviewing and commenting on their testing methods from the very beginning.
Testing Methodology
No doubt, executing advanced tests against 15 vendors while trying to acquire valid malware within a small window of time to execute is a daunting task. Their testing framework is massive and automated, and this was no small undertaking.
The NSS Lab AEP test primarily covered detection and prevention efficacy, as well as total cost of ownership (TCO) calculations. The test did not cover performance impact or user experience of the products. The recent 2017 Gartner Magic Quadrant for Endpoint Protection Platforms (EPP) specifically calls out Cylance for “easy deployment and management, low performance impact and high pre-execution detection rates against new threat variants”. Cylance was also cited as the “fastest-growing EPP vendor in the market”.
Cylance and several other vendors first agreed to allow NSS to test products privately. All of us wanted to first flush out any quirks in their framework. We all agreed to be publicly tested based on these exact same testing methodologies from the private testing.
NSS executed against this, but what we learned during the process of this public testing is that there is always room for improvement. In this public test, NSS Labs added additional testing methods without the foreknowledge of the vendors and added an arbitrary weighting of results to these tests. The percentage of efficacy applied given this arbitrary weighting scale is perplexing.
Where we have concerns is with the penalty weighting of evasion techniques within this test across all vendors. This is where we feel that NSS needs to reevaluate their own weighting scale and make the facts available upfront to all vendors, and not arbitrarily decide this weighting scale at the last minute.
An arbitrary percentage of the test was based on seven evasion techniques, while the other 1,840 advanced tests used accounted for the remainder. We questioned how NSS had come to that skewed weighting score, and we will continue to work with them to improve overall transparency in their testing methodology going forward.
Evasion Techniques Examined
Evasion techniques are a serious concern for all vendors, for as we all know, there are many ways to evade any and all endpoint security products. Evasions are the closest testing can get to emulating an unbiased, unknown set of malware. At Cylance we take evasions very seriously and believe that having a broad set of evasion techniques can only make these tests more effective at determining a product's efficiency in the real world. However, let’s agree to weight them appropriately, or else the test is unfairly skewed for everyone.
The scoring of Cylance currently stands at 99.69% efficacy. Cylance has addressed these penalizing evasion techniques called out in the report and we are now awaiting NSS Labs to re-test us for verification. We believe we can now ignore the penalty shown on Cylance.
Here, it’s key to understand the TCO within the NSS Labs scoring matrix. TCO is based on the product purchase, product maintenance, installation, threat alerting and monitoring and upkeep. Prevention is always better than detection, and if one can prevent the malware from executing in the first place, as Cylance’s product CylancePROTECT® does, this reduces your TCO considerably.
We at Cylance appreciate NSS Labs independently proving what our customers have been saying for years - CylancePROTECT is a proven endpoint security solution for replacing legacy antivirus products with AI powered prevention, blocking the most advanced cyber threats, pre-execution. This is a giant stride forward in evolving security testing methodologies to match real-world environments, and we look forward to more progress in the coming months and years as we remain committed to advancing these testing methodologies.
You can learn more about NSS Labs’ new advanced endpoint test here: https://www.nsslabs.com/linkservid/13FCD19B-5056-9046-9323B872FF06DD0A
You can find the results of the NSS Labs Advanced Endpoint Test here: https://pages.cylance.com/2017-02-14-CNT-NSS-Report-2017.html
This CSO Online article by Steve Ragan does a great job looking at what else is wrong with this system, and how efforts are being made by outfits like AV-TEST to fix the problems inherent in the current testing models, and to create more realistic results that are not based on pay-to-play testing house setups.
Finally, with regards to the Anti-Malware Testing Standards Organization (AMTSO), Cylance has been heavily involved with this organization since September 2016 and recently joined its ranks in December 2016.
We have been involved in many standards working groups to include contributing to the testing protocols standards for the testing of anti-malware solutions. We see this an opportunity to contribute to the development of real-world scientific tests that are reproducible, statistically valid, and objective.
As always, we also encourage you to Test for Yourself. Res Ipsa Loquitur! (“The thing speaks for itself!”)
