We have seen countless times where attackers have demonstrated the ability to bypass the protection offered by conventional and traditional endpoint security solutions. While layers of network-based defense can help with prevention, endpoints are not always on the corporate network, and malware is still bypassing these layers and making it through to the end user.
The last line of defense is on the endpoint, and most everyone desires a preventative security solution that provides complete and utter protection against the most advanced threats.
Given new technological capabilities that have been introduced into the market, there also needs to be advanced testing capabilities that go beyond the conventional and traditional-based testing we have been accustomed to seeing.
Thus, Cylance led the engagement with some of these testing organizations and collaborated to create advanced testing methodologies in order to test and verify new malware prevention capabilities.
AV-TEST results can be found here.
Background
I used to work in the Office of the CTO for a major computer manufacturer who sold millions upon millions of endpoints each year as their Distinguished Engineer for Security, so suffice to say, I have years and years of rigorous evaluations under my belt.
One thing that became very clear to me early on: the test results we generated were vastly different than those being marketed by some so-called ‘independent’ testing firms.
How different? Let’s just say the cliché ‘night and day’ does not even come close to doing it justice. The private testing houses produce report after report where some of the antivirus (AV) vendors tested repeatedly score 100% efficacy in identifying malicious malware and blocking it.
The problem is, everyone and their brother knows that in the real world these products do not perform anywhere near 100% - if they did, much of our work in security would be finished and there wouldn’t be such a high demand for security experts.
But that is obviously not the case, as organization after organization continues to suffer major and minor security incidents because their AV missed a variant, for no other reason than the vendor had not seen it before, and therefore there was no signature to counter it.
We decided it was time for a change – a drastic change.
We have been working with the AV-TEST Institute, a leading independent security research company, to create a testing model that represents real-world scenarios employing unique malware samples that no AV products have ever seen before.
Testing Done Right: AV-TEST Report Data
To begin with – and contrary to how some other less ethical testing houses operate - we tested against other AV vendors and anonymized the results. This is fair, and should be a practice all testing companies should adhere to when testing is being sponsored.
Next, I worked with AV-TEST in creating new advanced testing methodologies. Now, if the vendor did not have the capabilities in their product to test well against these methodologies, they were removed from testing. We have seen time and time again where testing methods used don’t reflect the realities of the product being tested, and we did not want to contribute to that ill-advised practice. Everyone in this test claims to be able to detect unknown, zero-day malware.
Then - and this next item is where I give huge props - AV-TEST agreed to actually create their own unique malware to test against the AV products. I flew to Magdeburg, Germany and personally worked with AV-TEST to create the following three testing scenarios:
Test Case 1 – Holiday Test (Zero-Day)
The purpose of this test is to reproduce a real-world scenario whereby an end user goes on holiday for a given period. Upon returning from their holiday, the end user returns to their endpoint and gets infected prior to the endpoint being able to update its protection measures, a.k.a. signatures.
This is indeed a real-world scenario we have heard from our customers looking to purchase Cylance’s products and services. We also see cases in some enterprises where they have delayed signature updates by a quarter. This scenario is also a fairly easy way to test for zero-day detection and prevention capabilities.
The test itself froze the product VM at Day Zero and then it would be removed from being online. We then wait seven days, then AV-TEST collected 100 new malware (PE's) that are considered newly discovered seven days post the freezing of the product.
On day seven, AV-TEST would bring up the frozen product VM without connectivity to the Internet so that the AI/ML/signatures are essentially seven days old. They then run the newly discovered malware (PE's) against the vendors for detection and prevention efficacy.
The first test case covered 100 new samples that had to be detected offline and with seven-day-old signature databases. The results displayed in figure 1 show that Cylance achieved by far the best result in this test.
CylancePROTECT® detected over 97% of the samples before execution. The best other product detected 67% and the average of Vendor 1 to Vendor 5 was only 42% detection rate - however, only when combining static and dynamic detection. The static detection alone was even lower at 28% for the average of vendors 1 to 5. The best product scored 63%, compared to the 97% scored by Cylance.
Results of Test Case 1:
Figure 1: 'Holiday Test'
Test Case 2 – AV-TEST Zero-Day Development
The second test simulated a targeted attack where an attacker was able to introduce an executable file onto the system. These executables were created by AV-TEST to simulate certain types of attacks that had to be detected and blocked by the products. These executables are based upon common advanced attacks seen today.
The new zero-days are executed on systems first in offline mode to validate each endpoint security solution’s ability to detect true unknown attacks without connectivity to the cloud, and then online to show the impact of cloud queries.
Let me make this as big of a deal as I can… AV-TEST is the first testing organization to create their own malware with self-developed tools. This is significant and hugely important to validate the security solutions’ ability to detect never-before-seen malware. And guess what…
The results in figure 2 show the success in detecting new binaries that simulate attacks on an endpoint. The figure displays the offline detection. Again, CylancePROTECT detected all threats even before execution, and even without Internet connectivity.
Results of Test Case 2:
Figure 2: Simulated Offline Attack Test
Only two other vendors besides Cylance were able to detect some files offline before execution. Most of the other detections came from behavioral analysis during execution of the tools. Two vendors did provide additional detection when the test was carried out with online detection. You can see those results in the full report.
Test Case 3 – Drive-By Malware Without URL Filtering
The URL itself is not malicious, it’s the content of the website that is malicious. This test turns off the URL Filtering on all products under test to determine if they can truly detect the malicious nature of the visited website, or if they are simply blacklisting by reputation.
Products simply can’t keep up with the scale in which malicious URLs are created. The true answer to drive-bys is the ability of the product to detect and prevent the malicious content itself.
Not surprisingly, we are seeing a similar picture as in the previous tests. CylancePROTECT again detected nearly all test cases with static detection, and the one remaining case was detected during execution of the downloaded binary.
Results of Test Case 3:
Figure 3: Malware Distributed by Websites
Conclusion
There you have it. Three new advanced test cases to employing unknown zero-day malware scenarios. Again, I hand it to AV-TEST for stepping up and seeing the need to adapt and add new testing methodologies, and their willingness to actually create their own unique malware.
AV-TEST results can be found here.
More testing organizations must follow AV-TEST’s lead in developing their own malware to truly test advanced prevention capabilities of endpoint security products, and we are seeing this here with AV-TEST’s willingness to collaborate with Cylance on their development.
We cannot stress enough how revolutionary these new testing methodologies from AV-TEST are - they provide unprecedented results on the ability of advanced products to detect and prevent unknown never-seen-before malware.
AV-TEST is truly advancing the field of endpoint protection testing to a degree that should compel other testing companies to follow suit. We are working with a few other companies to do just that, but it is going to be a long fight to bring some of the others up to speed.
Vendors pay a lot of money for skewed testing that produces what appears to be high efficacy results compared to their competitors. This broken system has been working well for both parties but has also been utterly failing to protect the consumers of these products.
We are pleased that the regular tests performed by AV-TEST did indeed show that some of the products from our competitors do provide a reliable level of protection from common commodity malware - if of course the products have access to the cloud, can use all protection layers, and have updated signature databases.
But as shown in the results above, this is no longer enough to be able to protect from new and unknown threats. All endpoint protection providers need to apply these advanced testing methodologies in an honest and open fashion, and let the consumers of these products see unbiased testing results so they can make informed decisions.
The fine work done by AV-TEST in developing these new standards for testing will undoubtedly help to improve the security posture for countless organizations and individuals, but there is still a lot of work to be done.
Finally, with regards to the Anti-Malware Testing Standards Organization (AMTSO), Cylance had been heavily involved with this organization since September 2016, and recently joined its ranks in December 2016.
We have been involved in many standards working groups to include contributing to the testing protocols standards for the testing of anti-malware solutions. We see this as an opportunity to contribute to the development of real-world scientific tests that are reproducible, statistically valid, and objective.
We also encourage you to Test for Yourself. Res Ipsa Loquitur! (“The thing speaks for itself!”)
Chad Skipper
VP Industry Relations & Product Testing
