On July 15th, the results of the latest Business Security Test 2022 (March – June) by AV-Comparatives were released. Security solutions from 18 leading cybersecurity companies, including Bitdefender, were tested for their ability to prevent compromise of Windows endpoints.
AV-Comparatives is an independent, 3rd party security testing lab certified by the European Expert Group for IT-Security (EICAR) and ISO 9001:2015 certified by TÜV Austria (technical standards body) for independent tests. Operations of this organization are audited by TÜV Austria every year, to ensure that tests are unbiased and independent.
At Bitdefender, we strongly believe that independent, 3rd party testing is critical for informed customer decisions. The ultimate competitor of all participants are threat actors, and independent evaluations help security vendors learn from these exercises and improve our security technology.
In the following three sections, we discuss the most important takeaways and offer recommendations for choosing the best security solutions for your business needs. We always recommend drawing your own conclusions and not solely relying on interpretations by security vendors (even ours!)
Effectiveness
Effectiveness refers to the overall ability of a security solution to detect and block threats and is one of the most important metrics when choosing an endpoint security solution (an obvious point, but important to say). AV-Comparatives has a well-documented methodology, mimicking real-world conditions as much as possible.
Each Business Security Test report has two tests for protection:
- Malware Protection Test – considers a scenario where the malware pre-exists on the disk or enters the test system via e.g., the local area network or removable device, rather than directly from the Internet.
- Real-World Protection Test – mimics online malware attacks that a typical business user might encounter when surfing the Internet.
The Malware Protection Test is a more traditional anti-malware test. A total of 1007 recent malware samples were subjected to on-access scans (file copy operation), followed by an on-execution scan if necessary. Pre-execution and on-execution scans can use various modules, such as command-line scanner, fileless protection, memory and process monitoring, and others.
Bitdefender scored a 99.9% malware protection rate in the Malware Protection Test, compared to the industry average of 99%. A difference of less than 1% might look insignificant – but it leads to a compromise of 9x more systems on average based on the malware samples size.
The Real-World Protection Test is the more interesting of these two tests. In this test, 733 cases such as malicious URLs or drive-by downloads are tested over an extended period of 4 months. All protection features of the product can be used to prevent infection – not just signatures or heuristic file scanning. The modern endpoint security platform (Bitdefender XDR) is a much more robust suite of technologies with many layers of protection.
Bitdefender achieved the highest malware protection rate of 99.9% in the Real-World Protection Test, compared to the industry average of 98.86%. The only other security vendor that achieved this level of protection is also using our Bitdefender engine.
This was the best result of all tested security solutions and validates Bitdefender is built for real-world threats (it’s also interesting to note that some other so-called major industry vendors did not participate in this independent, 3rd party test). Bitdefender provides protection not only for the business market, but also for consumers, and an extensive OEM ecosystem. This gives us access to telemetry data from hundreds of millions of sensors globally, allowing us to use one of the most actionable threat intelligence feeds in the market. We are also very proud that one-third of the participating security vendors are running one or multiple Bitdefender technologies, adding further validation to the benefits of our technology and expertise.
Consistency
While effectiveness is the most popular metric, there is another metric that is even more important — Consistency. Consistency is effectively delivered over time and is critical for becoming cyber resilient. Understanding how a product behaves over the long term is one of the best insights into what you can expect from it in the future.
Between 2018 and 2022, AV-Comparatives completed 9 Business Security Tests. In 7 out of 9 (78%), Bitdefender offered the highest real-world malware protection rate. The next closest vendor ranked top in just 4 tests, with a significantly higher number of false alerts.
When talking about consistency, consider not only the historical perspective (how a vendor performs over time in one specific test), but also how a vendor performs in tests from other independent organizations with a well-defined methodology. AV-TEST is another popular independent research institute focused on endpoint protection and Bitdefender achieved the highest possible score in their latest test. And the one before that too.
Finally, when considering consistency, pay attention to all the security layers. AV-Comparatives (and AV-TEST) are focused on prevention, while tests such as MITRE ATT&CK Evaluations are oriented towards detection capabilities. When choosing a security solution, review available results for both prevention and detection capabilities – as both are of high importance to becoming more cyber resilient.
In the words of Bruce Lee: “long-term consistency trumps short-term intensity”. An attack must only succeed once to deliver a lot of harm.
Actionability
To increase the malware protection rating, vendors could decide to configure more aggressive policy settings. While these settings can artificially improve results by blocking the execution of partially suspicious files, the solution would not be usable in the real world due to increased noise (false alerts or alarms). To demonstrate this, AV-Comparatives also publishes the number of false alerts that each solution generated, which can be used to measure how accurate detections are and how much noise is generated by each vendor.
Detection of false positives is an important criterion for any endpoint security evaluation. A high number of false alerts impacts different parts of the organization, from end users (security should not limit productivity) to business (with an increase of TCO/ROI), to the security team itself (an overworked team).
In the latest report, Bitdefender generated 2 false alerts in the real-world protection test, compared to the industry average of 6.5. Zero false alerts were generated in the malware protection test!
A useful perspective on the impact of false positives can be found in a 2021 report from AV-Comparatives called the Endpoint Prevention and Response EPR Comparative Report. This report includes the costs incurred due to false positives for each product, as well as the cost of a potential security breach. In this report, the Bitdefender solution had the lowest 5-year TCO per agent of all products evaluated, due to our strong combination of prevention and response capabilities and a very low rate of false positives.
Conclusion
The best protection against modern attacks is to implement a defense-in-depth architecture. Start with reducing your attack surface and then leverage automated prevention controls to eliminate, as early as possible, most security incidents. For the few incidents that may get through your defenses, rely on detection and response capabilities, adopted as a service or as a product.
Implementing multiple layers of security should not be a check-the-box exercise – the quality of each layer should be thoroughly and regularly evaluated. While a high malware protection rate and a low number of false alerts is a good indicator of an accurate security solution, consider also other decision factors – for example, if a solution is Native XDR or Open XDR, and how easy it is to implement, configure, and use the product (watch the technical spotlight of our XDR solution).
3rd-party, independent testing with a well-defined methodology offers invaluable insights into the capabilities of leading cybersecurity companies so you can make informed decisions. Cybersecurity is a game of cat and mouse, with both sides continuously innovating and improving tools and techniques, and security vendors need to prove their solutions are effective, accurate, and provide consistent results.
We’re incredibly proud of the latest AV-Comparatives Business Security Test results and are excited that our solution, the result of hard work across several departments, and cybersecurity experts, is once again leading the endpoint security industry.
