Third-Party Testing: What Is It and Why Is It Valuable?

We all wrestle with the challenges of security in today's digital marketplace. The security landscape and potential attack surfaces continue to expand, and malware and exploits continue to become more sophisticated. However, one of the most significant security challenges that organizations face is simply deciding which solutions they want to incorporate into their security strategy. Vendors are multiplying at a dizzying pace, and anyone who has even been halfway around the block knows that data sheets and marketing materials aren't nearly as reliable as they could be. Moreover, given time and resource constraints, setting up a testbed and evaluating all potential solutions by hand is rarely a viable option.

This is why third-party testing of security products and solutions plays such a critical role in thwarting cyber criminals. 

What is Third-Party Testing and Why is it Important?

The reason is simple: organizations need effective security solutions that meet an evolving set of requirements. The fact that every organization’s network demands and business objectives are unique makes the selection process even more complicated. And to make things worse, far too many security vendors don’t do a very good job of providing data that enables a fair comparison between competing solutions. Besides often not providing enough information, data sheets can emphasize—and sometimes even inflate—good points, obscure product flaws, and rely on internal test results that don't replicate real-world environments.

Third-party testing plays an important role in addressing this issue because it requires the product to be evaluated by an accredited, unbiased third party to ensure product safety and quality assurance. Not only does this maintain product safety and fairness, but it also helps to obtain credibility in the industry.

Addressing the Challenges of Comparison Shopping

Looking for the right security products can feel worse than comparison shopping at the grocery store. Items next to each other on the shelf may appear similar at a glance, but when you look closely at the labels, you find that one is priced per ounce, while the next is labeled with a price per unit. Another calls itself “healthy,” yet contains too many grams of fat. Fortunately, the FDA and equivalent agencies around the world are tasked with protecting the health of citizens, and so they have the authority to ensure that the labels on the food products you buy reflect what's inside the package. And because they use the same measurements, standards, and processes you can make valid comparisons.

Unfortunately, there is no such authority for security solutions, which is why third-party testing facilities are so essential. They provide a comparative assessment of solutions using standardized testing criteria and methodologies, allowing organizations to take an educated look at solutions through a common lens that would not otherwise be possible.

Everyone Benefits from Third-party Testing

It's not just consumers who benefit from third-party testing. Vendors who regularly participate in these sorts of tests usually learn as much as their potential customers do from the results. Testing methodologies provide critical input to vendors about evolving enterprise requirements, while test results can help confirm they’re on the right track (or provide evidence for necessary course corrections) in terms of corporate expectations and comparisons to other products on the market. Independent testing can even help manufacturers better understand market shifts in the options being made available by competitors to make informed choices about where to focus engineering efforts.

With the advent of digital transformation, for example, the networks that security tools were designed to protect are undergoing profound and often radical change. They are broader, more complex, and subject to a more sophisticated threat landscape than ever before. Effective testing methodologies often reflect these new requirements, meaning that yesterday’s winners who sit on their laurels can quickly become less relevant as test results reflect new requirements.

Staying Ahead of Evolving Security Requirements

To be effective, independent testing must be based on open methodologies (refined continuously based on enterprise requirements), impartially applied across available products, and then quantifiably reported.  This means that organizations that rely on testing results to evaluate products need to do more than merely look at the results. They have to have confidence in the impartiality of the testing methodologies and ensure that the testing itself reflects the evolving challenges today’s network requires.

Here are a few examples that illustrate the new demands of network environments:

1. Organizations now expect next-generation firewall (NGFW) solutions to be able to provide effective SSL inspection, and that functionality needs to be integrated into any firewall test results on which organizations rely. The same is true for technologies such as integrated sandboxing and SD-WAN, reflecting the changing nature of threats, connectivity, and traffic.
2. Data center security gateways not only need to continue delivering high performance, but also provide advanced security functions such as segmentation, deeper levels of inspection, and seamless integration with cloud-based data and workflow resources.
3. Endpoint security must now provide advanced exploit prevention and utilize machine learning to more effectively address today's more sophisticated threats.
4. Breach prevention must combine the ability to not only detect known and unknown threats but also automatically respond to detected cyber events to stay ahead of fast-moving threats.
5. New tests also need to be continuously introduced. Web Application Firewalls (WAFs) are a relatively new area of testing, reflecting the growing need for dedicated protection of the web services portion of the network. Likewise, cloud services, IoT protection, securing OT environments, application integrity, and cross-functionality between traditionally isolated security solutions will all need their own, or to be added to existing testing processes to better reflect the rapid changes taking place in today's networks.

Examining the Challenges of Third-party Testing

Of course, not all testing is the same. This is why companies looking at test results must be also aware of some of the challenges. Here are two critical considerations:

1. Not all tests are created equal. When examining test results, you must understand what was tested. Different tests, even conducted by the same lab, look at different things and may have different objectives. Some are very narrowly focused. Some value things like efficacy over performance. Most do not evaluate critical elements such as interoperability, visibility, or collaboration. This is why it is important that the testing organization publishes how their testing is conducted, and that you ensure that their methodologies match your criteria.
2. Not all test results are created equal, either. It is also vitally important to know something about the organization that produced the test results you are reviewing. Many, such as AV-Comparatives, AV-Test, ICSA, NSS Labs, SE Labs, or Virus Bulletin operate with a high degree of integrity. A few other labs that are not members of AMTSO might not even follow the testing standards and provide a vendor with just about whatever test results they want.

Third-party Testing is Not Perfect, But It Is Important

For organizations addressing digital transformation, many of the current test methodologies being used by labs and testing centers provide critical insight into emerging requirements, enabling IT teams to evolve their security infrastructure appropriately. They help organizations narrow down potential solution candidates based on things such as superior effectiveness, performance, innovation, and value. They are an excellent place for organizations to start looking for validated solutions to ensure their security meets their evolving customer needs and internal digital business requirements. 

But also remember that most tests evaluate a product in isolation and that the solution you choose not only needs to be at the top of its game but also function as part of your larger security architecture. In addition to selecting a third-party-validated solution, also be sure to look for things such as interoperability and the ability to share and respond to threat intelligence as part of a coordinated response that’s tied to an open security fabric. This will ensure you’re leveraging the right approach that unifies all security technologies to improve threat response time and better protect your network

Read more about the Fortinet Security Fabric and the Third Generation of Network Security. 

Visit Fortinet’s FortiGate SD-WAN homepage to learn more about this advanced security solution.