How to Pick the Security Tools for your DevSecOps

14 January, 2023

Introduction to the Security Tool Challenge

“Invest in Outcomes, not Tools and Capabilities” – Paul Proctor, Gartner.

Security tools are overrated. They are even starting to become part of the problem. Companies are looking to improve their security postures by introducing security tools. Typical DevSecOps security tools (e.g., SCA, SAST, DAST) are great at finding vulnerabilities. One can immediately “see” the added value of these security tools by simply taking a free demo version for a spin. Hence, by bringing them in companies can demonstrate measurable improvements to their security postures. 

Unfortunately, the security posture improvements driven by tools is a mirage. First of all, the tools often have quality issues and high false positive rates. Thus we end up obsessing over the vulnerability counts, rather than focusing on the actual security risks. Secondly, the time required to go over the findings and fix them is seriously underrated. One of our largest customers has once asked us to run our software images against their (state-of-the-art and expensive) security tools. 9120 findings with 80 having a severity score of 10 out of 10! Finally, tools provide a false sense of security. Overall, this strategy has a minimal impact in terms of security risk reduction.

How to pick DevSecOp tools
An export of the  9120 findings of a vulnerability scanner against one of the software systems we have developed.

In this blogpost, I will discuss the problem in further detail. I will also present which tools Codific is using and how have we selected them. Don’t get me wrong, I love automation and tools are an integral part of that. However, as a security expert I am convinced that 80% of your security gains are unlikely to come from tooling. Tooling will help you boost productivity, but they won’t solve the security challenge by themselves.

Codific’s Journey in Software Security

Codific is a software security firm by design. Before starting the firm I was a researcher at the Imec-DistriNet lab of KU Leuven focusing on security and privacy engineering. I joined the club of the cool security guys in 2007 when together with 9 of my DistriNet colleagues we won the third place at an international capture the flag contest. Thus when I founded Codific, security was a non-negotiable software quality attribute. Up until a couple years back we didn’t have any security-specific tooling in place. Perhaps except for a few team red attack security tools (nikto, nmap just to name a few). Today we have a few tools in our DevSecOps pipeline. Nonetheless I strongly believe that their absolute overall added value to our security posture is relatively low. Our software assurance programme based on OWASP SAMM is what gives us most of our security superpowers.

Security Tools Could be Part of the Problem

According to Mark Curphey (OWASP founder and current board member), a security tool crash is coming. Amongst the conditions that are preparing the perfect storm is that security teams want less tools. The sheer number of findings generated by the tools is the key motivation behind this. From a vendor perspective, it is the vulnerability findings that demonstrate the immediate added value of the tools. On the receiving end though, many of these findings are often not exploitable. As high as 90-95% of all Common Vulnerabilities and Exposures (CVE) identified in the National Vulnerability Database (NVD) are not exploitable in any given configuration. Hence, security teams get overwhelmed by the volume of vulnerabilities detected by modern scanners.

Let’s have a quick look a the most commonly security tool types deployed within a DevSecOps pipeline.

Static Application Security Testing (SAST) / Static Code Analysis (SCA)

SAST tools examine the source code to detect and report weaknesses that could lead to vulnerabilities. These tools can detect various types of vulnerabilities such as: 

  • SQL injections
  • Cross-site scripting (XSS)
  • Insecure file operations
  • Authentication and authorization flaws
  • Buffer overflow
  • Unvalidated input
  • Hardcoded sensitive information
  • Insecure configuration
  • Insecure use of cryptography.

SCA is similar to SAST, however it may include additional checks for specific coding practices, security best practices and standards, and compliance to policies and regulations.

Dynamic Application Security Testing (DAST)

DAST tools analyze the runtime behavior of an application to identify potential vulnerabilities. These tools can detect some vulnerabilities similar to SAST such as: 

  • SQL injections
  • Cross-site scripting (XSS)
  • Insecure file operations
  • Authentication and authorization flaws
  • Buffer overflow
  • Unvalidated input
  • Insecure use of cryptography.

DAST tools can also detect vulnerabilities that are not possible to detect with SAST alone. In particular, DAST can detect vulnerabilities that arise from the interactions between different parts of the application, such as:

  • Insecure session management
  • Insecure cookie management
  • Insecure communication (e.g. HTTP header injection)
  • Insecure storage

Software Composition Analysis

Software Composition Analysis (SCA) tools are used to scan the third-party libraries and dependencies used in an application to identify known vulnerabilities and potential security risks. These tools work by analyzing the application’s dependencies, identifying the versions of those dependencies, and comparing them against a database of known vulnerabilities. SCA tools can typically:

  • Identify known vulnerabilities in the dependencies and libraries used by the application
  • Provide information about the severity of identified vulnerabilities
  • Provide guidance on how to remediate or mitigate identified vulnerabilities
  • Track the usage of open source components and licenses used in the application
  • Provide continuous monitoring and reporting capability, to alert when new vulnerabilities are discovered.

Container Scanning

Container scanning security tools are designed to scan container images, to identify known vulnerabilities and potential security risks. They can typically:

  • Identify known vulnerabilities in the software packages and dependencies used by the container
  • Provide information about the severity of identified vulnerabilities
  • Provide guidance on how to remediate or mitigate identified vulnerabilities
  • Track the usage of open source components and licenses used in the container
  • Providing continuous monitoring and reporting capability, to alert when new vulnerabilities are discovered.

The Tools’ Impact on the Overall Risk Score

As a researcher I’ve learned that asking the right questions is sometimes more important than looking for the answers. Despite the overall tone of the blog, I am not questioning why we need security tools. Codific is actually a vendor of a security assurance programme management tool called SAMMY. I believe the main question to ask ourselves is “when do we need a given security tool?”. The answer is painfully obvious. First, we need to understand the discipline of secure software engineering. Then we can add the security tools to boost our productivity.

From experience I know that the reality in many organizations is the complete reverse. Companies introduce security tools hoping they will make their software systems more secure. Unfortunately, the blind investment in tools is unlikely to alleviate the problem.

Let me reiterate the problem by using 4 questions that we ask during threat modeling:

  • What are we working on (software system scope)? 
  • What can go wrong (threats and risks)?
  • What are we going to do about it (countermeasures)?
  • Did we do good enough job (effectiveness and efficiency)?

Too often companies jump to the security tools (i.e., the countermeasures) without having a good understanding of the scope, the threats and the risks. Hence the effectiveness and the efficiency of these countermeasures (i.e., security tools) in terms of the overall risk score is questionable at best.

Security Assurance Programme First, Security Tools Later

We need a systematic approach improved by tooling to solve the application security challenge. Not the other way around where we first bring in security tools and hope that it will solve the problem. The best way to introduce a security assurance programme is by leveraging the OWASP Software Assurance Maturity Model (SAMM). SAMM is a security assurance programme that consists of 15 security practices split into 5 business functions. I will drive the key take away message home by describing one of the main philosophies behind SAMM.

SAMM is a maturity model. Each of the 15 security practices consists of 3 maturity levels.

  • Level 1 requires an initial understanding and ad hoc provision of the given security practice.
  • Level 2 focuses on increasing efficiency and effectiveness.
  • Level 3 entails a comprehensive mastery.

Starting from Level 2 without covering Level 1 is rarely a good idea. This is also precisely what most organizations typically end up doing when they introduce tools without having an initial understanding of a given security practice.

OWASP SAMM model

Security Tools in Codific’s DevSecOps Pipeline

We leverage SAMM as an Application Security (AppSec) programme for all our applications. Based on the SAMM practices we have gradually selected and introduced a number of security tools. Over the past years we have tweaked quite some configuration settings in these tools to reduce the number of false positives and even add the detection of new types of weaknesses.

Software Composition Analysis with Debricked

One of the first security tools we’ve introduced was a software composition analysis (SCA) tool called Debricked. The initial reason we’ve introduced Debricked was not to find vulnerable third party packages, but to track the license risk. In the context of one of our success stories, EY has acquired the software application called Valitax that we had developed for one of our visionary customers. We had the legal obligation to analyze and track all license risks in the Valitax product.

The third party and supply chain always represented a low risk at our firm. We always strive towards including the latest releases for all third party dependencies and frameworks. That is a policy to minimize any technical debt within our products. Obviously, Debricked has helped us reduce the period of having a vulnerable third party library in production. Without Debricked we had to stay alert for any critical vulnerabilities, while now we simply get an immediate notification.

Static Application Security Testing: PHPStan and SonarCloud

In the context of secure coding guidelines we have explored various code quality tools. Our prime goal was to force all our engineers to follow the best practices in general. We have initially picked SonarCloud that can work with many languages. However the out-of-the-box version of SonarCloud was somewhat too rigid. Adding custom rules proved to be cumbersome and required a more complex build setup. Moreover, we were ideally looking to run some basic SAST rules as git hooks to speed up the feedback loop to the engineers. PHPStan proved to be an excellent tool to attach as a pre-commit git hook. We can easily add custom rules in PHPStan. As opposed to SonarCloud PHPStan never produces any false positives.

DAST: OWASP ZAP

We’ve recently also added OWASP ZAP to our CI/CD. We run ZAP against a copy of the production system (obviously, with test data). ZAP runs as a scheduled job in a headless mode within a docker container. It scans the shadow copy of the system for vulnerabilities. ZAP fails the CI/CD pipeline if the scan detects vulnerabilities above a certain threshold. Our CI/CD platform is integrated with Slack and failed pipelines will notify the product team. ZAP also produces a nice report with all findings, hence the product teams can view and correct the situation.

Manage your security posture with SAMMY.
Manage your security posture with SAMMY.

Conclusion

The application security challenge is an essential complexity all software vendors face. Security tooling promises to solve many of these challenges. The number of security tools is growing. There are in fact way more vendors than the size of the market can support. Too often organizations introduce security tools in an ad-hoc way as a mechanism to tackle the security challenge. As a result the tools are now becoming part of the problem. Nearly all typical DevSecOps tools generate lots of security warnings, many of which are either false positives or not exploitable. As a result security teams want less tools.

I believe a security assurance programme based on risks should drive the selection of the right security tools for your DevSecOps pipeline. Not the other way around. You should also demonstrate security improvements in terms of meaningful metrics rather than the sheer number of vulnerabilities found by the tools. Software Assurance Maturity Model (SAMM) is a simple, measurable, and technology / organization agnostic programme I recommend you to start with. Setting up your DevSecOps pipeline and picking the right tools for it will flow naturally once you adopt SAMM.

Reach out to us here