Henry Ford, who was the first to use the assembly line to mass produce automobiles, is famously credited with saying, “If I had asked people what they wanted, they would have said faster horses.”
Before Cylance, I was in charge of the Global Malware Research organization for one of the world’s biggest and best-known antivirus (AV) companies. My team and I were responsible for writing signatures for every piece of known malware. I was hired ten years ago with the sole mission to bring the company's malware detection rates back to prominence.
Before I could fix the problem, however, I needed to identify exactly what the problem was. The first thing I did was to shadow our employees to assess the high-level operations of the globally distributed team. I needed to get an in-depth look at their workflow as they analyzed malware and wrote signatures to protect our customers. The next thing I did was go on a world tour to meet with as many customers as possible, in order to fully understand their pain points.
I asked our customers what security issues kept them up at night, and what they felt we could do to make their lives easier. Guess what many of them told me they wanted? "Faster signatures."
All signature-based antivirus technologies operate pretty much the same way behind the scenes, and herein lies their downfall. I am going to take you through each of the steps required to create those “fast signatures,” to explain why the continued use of signatures by the AV industry is a losing proposition.
Figure 1: How Traditional AV
Vendors Create a Signatures
How Traditional AV Vendors Create a Single Signature
Let's suspend reality for a moment and say that every traditional AV product has signatures for every single known piece of malware to date, as of this present moment. Even though this is not true, let's frame their operational problems in the context of just the next 24 hours, to give you a sense of the magnitude of the problem these traditional AV companies currently face.
T=0 | It's common knowledge that the malware authors have the element of surprise on their side. Before releasing a new version of their malware to the world, they can test every endpoint security product on the market to see if their malware is detected. Approximately 500,000 new suspected malware samples are seen by Cylance every 24 hours. New malware is known in the AV industry as zero-day malware. The clock starts ticking at this moment in our timeline, which we’ll call T=0. |
T=1 | In order to write a signature for a new piece of malware, traditional AV must first get their hands on the physical file. Therein lies the problem. In order for a zero-day to be identified as such, at least one ‘sacrificial lamb’ user has to become infected by the zero-day and report it. The clock is now ticking. However long it takes for that first Ground Zero victim to be struck down, and for security teams to then become aware of and collect that piece of malware, adds to the time it takes to create a signature for that malware. |
T=2 | Before a signature is written for the malware, it first must be determined that it is indeed malware. Writing a signature for a file that is misclassified as malware is known as a false positive. |
T=3 | Once a malware sample has been correctly classified as malicious, it’s now time to write the malware’s signature. It is first churned through backend automation to see if it’s possible to auto-generate a signature. If not, it gets escalated to human malware researchers, who will reverse-engineer the sample and apply their years of experience to analyze and break it down. |
T=4 | The next decision that needs to be made is to decide what kind of signature should be written. It takes a few seconds to write a hash-based signature, but that signature will forever after only detect that one individual malware file. Malware researchers could decide to write a pattern-matching heuristic signature that will detect a larger swath of malware in the same family, but that is a larger job that can take anywhere from hours to days. Remember, the clock is ticking, the malware is on the rampage in the wild, and the traditional antivirus company needs to get protection out to their customers ASAP. |
T=5 | Now that the AV company has finally chosen and written the final signature, they need to upload it to the cloud so their customers can download it. |
T=6 | The Security Administrators at their client companies may then decide to test the new signature file before deploying it, to ensure it correctly detects the malware and doesn't have false positives, which will cost the company time and money. |
T=7 | Depending on the size of the company, the Security Administrator now has to deploy the new signature file to each and every endpoint in their network. Lather. Rinse. Repeat. |
How long did the clock tick from the moment the malware was released, to the moment the clients of the traditional AV company had the corresponding signature deployed to every device they needed to protect? Hours? Days? Weeks? Months? Even, dare we say, years, in the case of targeted attacks?
It’s a sobering thought that every time a new piece of malware shows up in the wild, every traditional AV company has to jump right back to the beginning of the process again, at T=0. Now, multiply this process by the 500,000 new suspicious samples that Cylance sees every single day, and you’ll see why malware is currently out of control, laying siege to our economy and affecting everyone from first responders such as hospitals and police teams, to small business CEOs and mega-corporations, to the very critical infrastructure we rely on for our survival such as oil, water and power.
If traditional AV companies cannot triage all 500,000+ new daily samples and write enough signatures to detect all the malware released within 24 hours, they fall further and further behind. It doesn't take a PhD in mathematics to realize that these kinds of numbers stack up quickly and dramatically.
A decade ago, I wish I’d known what I know now – that there is a better way forward that doesn’t involve using labor-intensive signatures. At the ‘Big AV’ company, my team of brilliant malware researchers and I took it for granted that we had only one way to give our customers what they wanted – by creating faster signatures. But looking back, my time at that company reminds me of that episode of "I Love Lucy" when she was working on the chocolate factory assembly line:
VIDEO: Lucy Working on the Chocolate Factory Assembly Line
In my next blog, we will give insight into how Cylance's approach to detecting malware using math-based detection and artificial intelligence is a complete departure from this assembly-line method.
Stay tuned!
Hiep Dang
Director of Product Management
