Know and reproduce the scanner’s OWASP Benchmark results

Fluid Attacks uses the OWASP Benchmark

Fluid Attacks  is committed to delivering highly accurate security testing results. This means minimizing errors from the automated tools and security analysts involved in the Continuous Hacking  solution. To measure accuracy, Fluid Attacks utilizes several benchmarks, including the OWASP Benchmark Project .

The Open Worldwide Application Security Project  (OWASP) is a non-profit foundation dedicated to helping improve software security. It operates as an open, online community where anyone can contribute resources and expertise related to application security (AppSec). This collaborative environment allows developers and security professionals to learn from each other and stay up-to-date on the latest threats and best practices.

The OWASP Benchmark Project  is a free Java test suite created in 2015 to assess the accuracy, speed and coverage of automated software vulnerability detection tools. It helps developers and security professionals identify the strengths and weaknesses of various AppSec testing solutions, allowing for objective comparisons between them.

The OWASP Benchmark allows for the evaluation of different types of security testing tools that apply any of these techniques:

1. Static application security testing  (SAST): Analyzes source code
2. Dynamic application security testing  (DAST): Tests running applications without access to source code
3. Interactive application security testing (IAST): Analyzes code and application behavior through agents and sensors while the application is running

A tool’s performance is measured by its ability to correctly identify vulnerabilities (true positives) and secure code (true negatives) while minimizing incorrect assessments (false positives and false negatives). This evaluation provides a benchmark for choosing the right security testing tool for your software development lifecycle (SDLC).

OWASP Benchmark measuring system

The OWASP Benchmark utilizes two primary metrics to evaluate tool performance:

1. True Positive Rate (TPR) or Sensitivity: This indicates the percentage of actual vulnerabilities correctly identified by the tool. A higher TPR means the tool is more effective at finding real security risks.
2. True Negative Rate (TNR) or Specificity: This indicates the percentage of safe code correctly identified as non-vulnerable. A higher TNR means the tool is less likely to generate false alarms.

As mentioned earlier, two other classifications of assessments are possible, however, they are not treated further in this page:

1. True negatives: Correct reports of code, inputs or ports as being secure (these are desirable, as they allow you to address real security risks)
2. False negatives: Incorrect reports of code, inputs or ports as being secure (these are highly undesirable, as they can lead to a false sense of security and the potential deployment of vulnerable versions of your system into production)

The following illustration may help you better grasp the result categories, where the bigger circle encloses what the tool reports as vulnerable:

Fluid Attacks’ results

To ensure high levels of accuracy, Fluid Attacks  tested its SAST automated vulnerability detection tool —a core component of the Continuous Hacking plans — against the OWASP Benchmark test suite .

The following scorecard shows how Fluid Attacks’ scanner compares to other vulnerability detection tools measured against the OWASP Benchmark test suite:

As demonstrated in the image above, Fluid Attacks’ scanner consistently outperforms other vulnerability detection tools:

1. 100% True Positive Rate: The scanner accurately identifies all actual vulnerabilities in the test suite.
2. 0% False Positive Rate: The scanner does not flag secure code as vulnerable.

This exceptional performance translates to a perfect OWASP Benchmark Score of 100%, significantly exceeding industry averages:

1. Almost 3x higher than the average score of commercial (paid) vulnerability detection tools
2. More than 1.5x higher than the highest-scoring non-commercial (free) tool

What is most important, Fluid Attacks  cares about what you care:

• Finding all vulnerabilities before they impact your business
• Maintaining your team’s efficiency with zero false positives

Reproduce Fluid Attacks’ OWASP Benchmark results

All Fluid Attacks products are open source . You can download, inspect, and suggest to modify the source code  behind them. Being open-source gives customers confidence in Fluid Attacks’ transparency and security .

To verify the OWASP Benchmark results, follow these steps (or skip ahead to the one-step alternative):

1. Meet the requirement: Install Docker  to be able to run Fluid Attacks’ main CLI scanner.

2. Install the scanner: Download the Docker container  and pull the image.

3. Clone the OWASP Benchmark v1.2 repository:

   git clone https://github.com/OWASP-Benchmark/BenchmarkJava.git benchmarkcd benchmark

4. Create a configuration file: Use the following content:

   checks:  - F004  - F008  - F021  - F034  - F042  - F052  - F063  - F089  - F107  - F112  - F130namespace: OWASPmultifile: trueoutput:  file_path: /working-dir/results/Benchmark_1.2-Fluid-Attacks-v2024.csv  format: CSVsast:  include:    - src/main/java/org/owasp/benchmark/testcode/    - src/main/java/org/owasp/benchmark/helpers/DatabaseHelper.java    - src/main/java/org/owasp/benchmark/helpers/SeparateClassRequest.java    - src/main/java/org/owasp/benchmark/helpers/Thing1.java    - src/main/java/org/owasp/benchmark/helpers/Thing2.java    - src/main/java/org/owasp/benchmark/helpers/ThingFactory.java    - src/main/resources/benchmark.properties

5. Execute the scan:

   docker run --rm -v .:/working-dir fluidattacks/sast:latest sast scan /working-dir/config.yaml

   Once the scan completes, the results are saved in a CSV file named Benchmark_1.2-Fluid-Attacks-v2024 located in the results/ folder of the cloned repository.

6. Install the OWASP plugin: You need an OWASP plugin to create a scorecard from the results. Fluid Attacks currently uses a modified version of the plugin to ensure compatibility with the latest scanner version. (A pull request  to add native support is open in the official OWASP Benchmark repository.)

   To install the plugin locally (make sure you have Maven  installed):

   cd ..git clone https://github.com/alejolagosm/BenchmarkUtils.git benchmark_utilscd benchmark_utilsmvn installcd ../benchmark

7. Add YAML file to use the plugin: Inside the benchmark repository, add a YAML file named benchmark_config.yaml with the following contents:

   expectedresults: expectedresults-1.2.csv

8. Run the plugin to generate the scorecards:

   mvn org.owasp:benchmarkutils-maven-plugin:create-scorecard -DconfigFile=benchmark_config.yaml

9. Open the results in your browser: For example:

   On Firefox:

   firefox scorecard/Scorecard_Home.html

   On Google Chrome:

   google-chrome-stable scorecard/Scorecard_Home.html

   Tip

   Free trial Search for vulnerabilities in your apps for free with Fluid Attacks’ automated security testing! Start your 21-day free trial  and discover the benefits of the Continuous Hacking  Essential plan . If you prefer the Advanced plan, which includes the expertise of Fluid Attacks’ hacking team, fill out this contact form .