Business & Technology

Independent Testing for Cybersecurity: 4 Types to Know

By David Finger | April 20, 2021

Organizations are investing in security. According to Gartner, 78% of organizations use 16 or more security tools and more than $150B is spent on information security every year. Further, the Gartner hype cycles for cloud, network, application, and endpoint security cover more than 60 products.

Despite all of these security solutions and spending, it remains difficult to definitively answer a key question: “How secure is our organization?” Let alone, “Are we protected from [the latest] cyber attack?” Independent testing provides a way to answer these questions in an unbiased way.

5 Types of Independent Testing to Use

There are multiple types of independent testing designed to help organizations improve their cybersecurity posture. The best option will depend on your organization’s cybersecurity needs. Here’s an overview of each.

Security Scoring

Whether internally developed or established by industry available tools, key performance indicators (KPIs) can be used to assess your cybersecurity posture across all security configurations and controls. KPIs are one way to answer the question of how secure an organization may be either as an absolute, based on its historical levels, or as compared to organizations of similar size, geographies, or business. Using KPIs can provide a relative assessment that can be considered reasonable. But simply being better than the average does not necessarily mean that your security is adequate for your level of risk.

Penetration Testing 

To understand your real risk of an incident, you can engage a red team of ethical hackers to attempt to breach your security configurations, controls, and teams. These groups are experts in the latest tools, techniques, and tactics. They act like cyber criminals and attempt to breach your defenses, which is an excellent way to stress test every aspect of your security, including employee awareness. This approach helps you determine which defenses are strong and which are weak. A key limitation is that it is dependent on the expertise of the red team and it only occurs at a single point and defined scope of attack.

Breach Attack Simulation

Breach attack simulation (BAS) is similar to penetration testing. Like penetration testing, it attempts to assess the totality and effectiveness of your defenses, but it uses automation tools to seek entry, rather than human experts. BAS can be run regularly and broadly, rather than at a single point in time or scope. However, the attacks are more programmatic, so they may be less sophisticated or customized than penetration testing.

Independent Effectiveness Testing

In addition to the organization-specific assessment of overall security, expert test labs run independent assessments of specific security tools. These assessments often benefit from a much larger sample set of attacks, since they are relevant to a broad set of organizations. And in many cases, they can provide comparative scoring for security tools of the same type. The common downside is that they operate in a lab, rather than in the real world. The conditions may vary from those of your organization, particularly over time. The assessments also typically focus on just one type of control, such as network security, email security, or endpoint security. They rarely test combinations of controls.

MITRE Engenuity ATT&CK Evaluations

MITRE Engenuity’s ATT&CK Evaluations are another useful tool. The evaluations test a range of security tools that are typically in the same security category and expose them to a single or small number of sophisticated cybercriminal campaigns. These campaigns are comprised of a series of tactics and techniques that are designed to accomplish a defined cyber mission. The key benefits of this approach are:

  • Enterprise security teams to see the inner workings of security controls. They can understand not only what the solution detects but also why and how it performed. Seeing the process can give teams more confidence in the type of protection they have. The evaluation goes beyond a single attack, sample set, point in time, or control. Evaluation results also can be combined across controls for a more comprehensive view of coverage or exposure.
  • Security vendors get an independent assessment of their product’s capabilities through the lens of the cybercriminal and a real-world campaign. They also have a collaborative community that can help them continuously improve the capabilities of their security products.

The primary drawback is that tactics and techniques evolve over time and the evaluation results are constrained to the scope of the campaigns that are run. They also focus only on detection of the attack technique, with no ability to assess what else (including legitimate operation) might be flagged by the control. 


Answering tough questions like “How secure are we?” or “Are we protected from [fill in the blank]?” requires considering a range of resources. If your objective is to do more than the average organization, security scoring is a great tool. If your objective is to push your security posture to higher levels, penetration testing and/or breach attack simulation are great aids. For granular assessments of individual security controls at points of exceptional risk, independent effectiveness testing can help. And for planning and implementing a rigorous and resilient defense based on capabilities across controls in aggregate, the MITRE ATT&CK Evaluation is a valuable tool. Finally, if you have questions that relate to a specific cyberattack or campaign, you should talk to each security vendor to get the answers you need.

To view our complete results, please visit the MITRE Evaluation website for Carbanak + FIN7.