Last week at the BlackHat convention in Las Vegas, Lidia Giuliano and Mike Spaulding gave a presentation entitled, “Lies, and Damn Lies: Getting Past the Hype of Endpoint Security Solutions.” Upon starting the presentation, Mike joked that they could have renamed it “Lawsuits, and Damn Lawsuits” for reasons that I’ll explain below.
With a combined 35 years in Information Security between them, Mike and Lidia have seen their share of infosec marketing hype. However, to effectively deal with rampant ransomware in their environment, they wanted to cut through all the hype.
Just like all security teams, they knew they needed to focus on protecting themselves, rather than buying into all the marketing buzzwords of the day. In their case, their business-centric goals were to reduce incidents, reduce people costs, keep the reputation of the firm, and keep the business running.
Testing Real-World Scenarios
As they began talking to vendors about potential solutions, they quickly realized that they would need to create their own test framework to effectively test their own real-world scenarios. They initially selected eight vendors, but once Mike and Lidia explained that they would essentially be doing a bakeoff, three vendors quickly dropped out.
Vendors dropping out of their test framework project speaks volumes – many vendors must realize that their products perform in the subpar range during real-world testing.
Throughout the course of the project, vendors gave Lidia and Mike varying kinds of bad advice:
- Some vendors tried to seduce them into testing only within the vendor’s own cloud-based environment. Mike and Lidia saw right through this sham – the vendor controls the entire environment, which, of course, makes their product look superior and more effective than the competition.
- Some vendors told them to only protect critical servers. Yet, in their environment, file shares were used extensively – and if one Patient Zero endpoint is not protected or is using traditional antivirus (AV), then the entire file share could be infected.
- Some vendors claimed to have “Real Time APT Protection” – yet had no memory-based analysis. These vendors are just using buzzwords that just don’t jive with how their product actually works.
- Some vendors claimed to be a “Leader in Cloud-based Endpoint,” but such solutions do not work for a roaming user with no Internet connection. Being a “leader” carries no weight if your product doesn’t prevent malware in any environment that you’re likely to encounter.
Lidia explained how she methodically worked from Oct 2016 through to May 2017 on business requirements, setting up a very large test framework and documenting her results. She also realized that it would be important to simulate never-seen-before malware variants, often referred to as zero-day attacks, so she setup her system to mutate malware to better represent a real-world attack scenario.
Attackers don’t typically just reuse malware that is known and easily blocked by traditional AV solutions – they want to get into the corporate network, so they often make minor changes to known malware to slip past the guards at the front door. The vendors in Lidia's bakeoff whose solutions rely heavily on hash-based signatures that have already been seen before tried to convince her to drop this approach, but Lidia knew better. And she persisted.
Creating a Realistic Attack Scenario
Ultimately, Lidia and Mike’s test framework grew to contain tens of thousands of samples. Lidia wrote scripts to automate much of the testing. In this way, they were easily able to test and retest on different OS’s, with and without network connectivity, with known samples and mutated samples – and easily replicate tests weeks or months later. Essentially, they created a much more realistic attack scenario than many of the existing testing procedures accounted for.
The difference in effectiveness and performance among the five solutions tested in various categories was stunning. The presentation walked though vendors named A through E. One vendor, vendor C, was clearly superior to the others in several categories of testing, both in effectiveness and performance. The failures of the other four vendors in many cases were simply striking.
Mike and Lidia had originally planned to share the detailed results, including all the vendor names. Yet, as word of their presentation spread, they were threatened with legal action from some of the vendors. As recently as Monday of last week, they had planned on at least announcing the name of the one vendor who ranked by far the highest in the overall testing.
However, after receiving a Cease and Desist notice last Monday, and speaking with their legal counsel, they decided to avoid any further legal hassle for their client and opted to anonymize all the vendor names in their bakeoff, at least for now.
Why You Should Test For Yourself
I applaud Lidia and Mike’s near-heroic efforts to bring truth and honesty to this field, and their attempts to combat marketing hype. At Cylance, we fully support this kind of in-depth analysis of anti-malware efficacy that Mike and Lidia have researched, as well as the framework for testing. Their efforts present organizations with real results so that they can make the right decisions for their companies and cut through all the buzzwords.
Here at Cylance, the advice we give to customers has always been to test our products in your environment, so you can see for yourself how effective they can be.
Of course, there are no silver bullets in this industry. Yet, customers who take the time to test within their own environment will be well rewarded for their efforts, in that their environments will be significantly safer than if they just chose a vendor based on their marketing materials.
I encourage you to reach out to Mike (@fatherofmaddog) and Lidia (@pink_tangent) to check out their research. You may even want to join me in thanking them for their courageousness in the face of legal challenges for trying to do the right thing. Even better, they also have templates and a guide for testing available to the community free of charge. Kudos to Mike and Lidia for taking on this important topic.
