My previous employer – not to name any names – was one who sold millions upon millions of endpoints each year. I was a part of the Office of the CTO, and my main focus was the security strategy for these endpoints. One of my many challenges was to find and test the very best in endpoint security products – technology that could prevent commodity threats from executing on the endpoint, and provide advanced threat protection (ATP).
While conducting my search, I was shocked to discover that I could not rely on security testing house results, because most security vendors scored 100% in tests, which those of us who work in the industry know is not possible. If it were, then everyone could take their ball home because the game would be over. Those impossible testing house results were completely at odds with the results I obtained in my independent testing, but more on that later.
Fast forward 18 months and I found myself at Cylance – in large part due to the incredible results I’d obtained from Cylance’s products during my testing. My first call of duty after joining Cylance was to engage with the testing houses that publicly test security software. I will not discuss the names of any of these testing houses, but what I found in some of these places is astonishing to say the least.
Let me start by saying that in some, I found examples of positivity and growth, usually coupled with the willingness to change and work with new security technologies such as machine learning which have produced a paradigm shift in the security industry of late. In others, unfortunately, I encountered nothing but fraud, bias, software piracy and extortion. I know these are strong accusations, but I call it like I see it.
Why It’s All About the Benjamins
As with most things in life, when you dig a little deeper, it’s all about the Benjamins. Some testing houses are scared to change, because change will impact their revenue streams. They are afraid that testing and showing the true results (or lack of results) from the most well-known AV vendors will result in those vendors discontinuing their relationships with said testing house, resulting in loss of revenue. This is, in effect, fraud. Some of these testing houses refuse to show poor efficacy results so as not to impact their bottom line, and in that refusal, perpetrate a fraud against you, the consumer of that security testing report. It’s astonishing that these business practices of courting security vendors to reflect high efficacy results in exchange for payment are alive and well. To me, this is completely unacceptable.
The defrauding and manipulation of the public with these tests also stems from vendors who pay so that their test results will show 100% efficacy. These reports not only deceive the buyer, but they also set up impossible standards for the entire security industry. Repeat after me: there is no such thing as a 100% efficacy rate in security. There is no single silver bullet that will provide total, unbreachable protection against every type of malware in every situation - ever!
If a vendor does get 100% on an anti-malware test, they either:
1) Paid for perfection, bribing the testing house to hide the negative results of their tests.
2) Tested using a statistically invalid sample set of malware like 100 samples.
3) Tested with samples not in any way reflective of real world attacks.
4) All of the above.
Uncovering Bias and Extortion in AV Testing Houses
Now, it goes without saying that there is always a bias of some sort when it comes to technology. If humans are involved in the testing, we will always have a percentage of bias. Some of the time that’s due to the relationship between the vendor and tester, and in other cases it’s the testing houses’ preference for a certain type of technology – even technology that is now outdated. But I get it. There’s something very comforting about the familiar. For example, I have a bias for certain car manufacturers because of my perception of them (right or wrong) as being good quality, reliable brands. Maybe there are better, newer brands of automobile out there, but because I’ve been burned a few times, I’m going to stick to what I know, thank you very much. So I always buy the same brand of car, the brand I’m used to.
A certain testing house pirated Cylance’s software at one point, because they believed we had unproven marketing claims. They wanted Cylance to come over to their table and pay for testing – at their testing house. They felt that the efficacy rates reported by users of our next-generation, artificially intelligent products were unproven because Cylance didn't use so-called public testing methods, which are (surprise, surprise) biased… because they can be bought.
Well, I have some news for you: Cylance will never ‘pay to play.’ In fact, we invite you to Test For Yourself. Personally speaking, I did so myself while at my former employer. The jaw-dropping results I obtained stood in stark contrast to the results ‘created’ at most testing houses.
The next ‘big whopper’ moment was when we were approached by a testing house to be included in their own analyst testing. Notwithstanding the atrocious testing methodologies commonly utilized by this place, Cylance was asked to personally pay for this testing in order to be granted various ‘rights’, such as a choice of the types of malware used in the test (which real life does not give us), an opportunity to challenge the testing results (excuse me?) and also, worst of all, to have edit rights to the final article. If we did not give in to this extortion, then the Cylance product would be forcibly tested by this testing house, purposely using bare-bones policy settings that would make our product inert. Those skewed results would then be publicly posted.
Once again, this is extortion, plain and simple. Buying test settings does not give the consumer accurate results. Picking and choosing which malware is to be used in the test does not help the end user. Having a glowingly positive EDITED article written about your test results does not help your client when a piece of ransomware you ‘chose’ not to test against breaks through your inadequate traditional defenses and lays waste to their company’s systems.
AGAIN I say: Test For Yourself. Res Ipsa Loquitur! (“The thing speaks for itself!”)
A Bright New Dawn for AV Testing
Now, having stated this, I must say that I have been extremely impressed with some testing houses. Not all of them fall into the same boat as those I have described above. Some testing houses have been completely open to updating their testing methodologies in a manner capable of effectively scrutinizing machine learning efficiencies, and in a manner that is reflective of the paradigm shift Cylance has introduced to the world. These testing houses see the need for bold new technology to heal the wounds of the past, and view the honest testing of this technology as vital to the greater good of the security industry.
It is with these testing houses that Cylance is currently engaged and working to provide comprehensive public testing results. Cylance welcomes any other vendor with advanced capabilities to step up and engage as Cylance leads the fight in confronting those testing houses which operate in a way that does harm to the good name of this industry, and undermines those who wish to advance it.
VP of Product Testing & Certifications, Cylance