Updated: 18 January, 2024
14 January, 2023
Introduction to the security tool frenzy
“Invest in Outcomes, not Tools and Capabilities” – Paul Proctor, Gartner.
Security tools are overrated. They are even starting to become part of the problem. Companies are looking to improve their security postures by introducing security tools. Typical DevSecOps security tools (e.g., SCA, SAST, DAST) are great at finding vulnerabilities. One can immediately “see” the added value of these security tools by simply taking a free demo version for a spin. Hence, by bringing them in companies can demonstrate measurable improvements to their security postures. Or do they?
In my experience security posture improvements driven by tools has really shaky foundations. It’s like ensuring personal health relying solely on advanced medical equipment (MRI, heart rate monitors, blood test analyzers, etc). First of all, the tools often have quality issues and high false positive rates. Thus we end up obsessing over the vulnerability counts, rather than focusing on the actual security risks. Secondly, the time required to go over the findings and fix them is seriously underrated. One of our largest customers has once asked us to run our software images against their (state-of-the-art and expensive) security tools. There were 9120 findings with 80 having a severity score of 10 out of 10! Our red team was skeptical even one of those was exploitable. Finally, tools provide a false sense of security. Overall, this strategy has a minimal impact in terms of security risk reduction.
In this blogpost, I will discuss the problem in further detail. I will also present which tools Codific is using and how have we selected them. Don’t get me wrong, I love automation and tools are an integral part of that. However, as a security expert I am convinced that 80% of your security gains are unlikely to come from tooling. Tooling will help you boost productivity. Though you cannot solve the application security challenge solely based on tools.
Key takeaways
- Start by figuring out what is your risk profile and risk tolerance. From there conduct a SAMM assessment to figure out the gaps.
- Create processes and roles & responsibilities around those. Bring in the tools to streamline your processes and help your people.
- Security tooling (SAST, DAST, SCA) should be an integral part of your vulnerability management programme, not the programme itself.
- Codific has had some positive experience with SAST, DAST and SCA tools. However on a scale from 1 to 10 the added value of the tools is around 3.
- Adopt OWASP SAMM and focus on threat modeling first.
Codific’s journey in application security
Codific is a software security firm by design. Before starting the firm I was a researcher at the Imec-DistriNet lab of KU Leuven focusing on security and privacy engineering. I joined the club of the cool security guys in 2007 when together with 9 of my DistriNet colleagues we won the third place at an international capture the flag contest. Thus when I founded Codific, security was a non-negotiable software quality attribute. Up until a couple years back we didn’t have any security-specific tooling in place. Perhaps except for a few team red attack security tools (nikto, nmap just to name a few). Today we have a few tools in our DevSecOps pipeline. Nonetheless I strongly believe that their absolute overall added value to our security posture is relatively low. Our software assurance programme based on OWASP SAMM is what gives us most of our security superpowers.
Security tools are sometimes part of the problem
According to Mark Curphey (OWASP founder), a security tool crash is coming. Amongst the conditions that are preparing the perfect storm is that security teams want less tools. The sheer number of findings generated by the tools is the key motivation behind this. From a vendor perspective, it is the vulnerability findings that demonstrate the immediate added value of the tools. On the receiving end though, many of these findings are often not exploitable. As high as 90-95% of all Common Vulnerabilities and Exposures (CVE) identified in the National Vulnerability Database (NVD) are not exploitable in any given configuration. Hence, security teams get overwhelmed by the volume of vulnerabilities detected by modern scanners.
Let’s have a quick look a the most commonly security tool types deployed within a DevSecOps pipeline.
Static Application Security Testing (SAST)
SAST tools examine the source code to detect and report weaknesses that could lead to vulnerabilities. These tools can detect various types of vulnerabilities such as, SQL injections, Cross-site scripting (XSS), Insecure file operations, Buffer overflow, Unvalidated input, Hardcoded sensitive information, etc. In addition to security-related issues, SAST tools might also check for specific coding anti-patterns, security best practices and standards, and compliance to policies and regulations.
Software Composition Analysis
Software Composition Analysis (SCA) tools are used to scan the third-party libraries and dependencies used in an application to identify known vulnerabilities and potential security risks. These tools work by analyzing the application’s dependencies, identifying the versions of those dependencies, and comparing them against a database of known vulnerabilities. SCA tools can typically identify known vulnerabilities in the dependencies and libraries used by the application, track the usage of open source components and licenses used in the application, alert when new vulnerabilities are discovered, etc.
However, SCA findings are often based on CVSS scores to assess the impact of a vulnerability. It doesn’t say anything on the likelihood, hence risk. Moreover, vulnerabilities flagged by SCA tools are often not exploitable. Exploit Prediction Scoring System (EPSS) is starting to appear in SCA tools. EPSS measures how likely a particular vulnerability is to be exploited in the wild. However, the adoption of EPSS in SCA tooling is scarce.
Dynamic Application Security Testing (DAST)
DAST tools analyze the runtime behavior of an application to identify potential vulnerabilities. These tools can detect some vulnerabilities similar to SAST such as, SQL injections, Cross-site scripting (XSS), etc.
However DAST tools can detect vulnerabilities that are not possible to detect with SAST alone. In particular, DAST can detect vulnerabilities that arise from the interactions between different parts of the application. For instance, Insecure session management, Insecure cookie management, Insecure communication (e.g. HTTP header injection), Insecure storage, etc.
Container Scanning
Container scanning security tools are designed to scan container images, to identify known vulnerabilities and potential security risks. They can typically identify known vulnerabilities in the software packages and dependencies used by the container. This category of tools is notorious for false positives as well as generating so many vulnerabilities that the software teams are likely to simply start ignoring them.
The tools’ impact on the overall risk score
As a researcher I’ve learned that asking the right questions is sometimes more important than looking for the answers. Despite the overall tone of the blog, I am not questioning why we need security tools. Codific is actually a vendor of a security assurance programme management tool called SAMMY. I believe the main question to ask ourselves is “when do we need a given security tool?”. The answer is painfully obvious. First, we need to understand the discipline of secure software engineering. Then we can add the security tools to boost our productivity.
From experience I know that the reality in many organizations is the complete reverse. Companies introduce security tools hoping they will make their software systems more secure. There is an upside to that. Vulnerabilities generated by the tools are likely to raise awareness in all teams. However, blind investment in tools is unlikely to alleviate the problem.
Too often companies jump to the security tools as controls without having a good understanding of the scope, the threats and the risks. Hence the effectiveness and the efficiency of these controls in terms of the overall risk score is unclear.
Security assurance programme first, tools later
We need a systematic approach improved by tooling to solve the application security challenge. Not the other way around where we first bring in security tools and hope that it will solve the problem. The best way to introduce a security assurance programme is by leveraging the OWASP Software Assurance Maturity Model (SAMM). SAMM is a security assurance programme that consists of 15 security practices split into 5 business functions. I will drive the key take away message home by describing one of the main philosophies behind SAMM.
SAMM is a maturity model. Each of the 15 security practices consists of 3 maturity levels.
- Level 1 requires an initial understanding and ad hoc provision of the given security practice.
- Level 2 focuses on increasing efficiency and effectiveness.
- Level 3 entails a comprehensive mastery.
Starting from Level 2 without covering Level 1 is rarely a good idea. This is also precisely what most organizations typically end up doing when they introduce tools without having an initial understanding of a given security practice.
Security tools in Codific’s DevSecOps pipeline
We leverage SAMM as an Application Security (AppSec) programme for all our applications. Based on the SAMM practices we have gradually selected and introduced a number of security tools. Over the past years we have tweaked quite some configuration settings in these tools to reduce the number of false positives and even add the detection of new types of weaknesses.
Software Composition Analysis with Debricked
One of the first security tools we’ve introduced was a software composition analysis (SCA) tool called Debricked. The initial reason we’ve introduced Debricked was not to find vulnerable third party packages, but to track the license risk. In the context of one of our success stories, EY has acquired the software application called Valitax that we had developed for one of our visionary customers. We had the legal obligation to analyze and track all license risks in the Valitax product.
The third party and supply chain always represented a low risk at our firm. We always strive towards including the latest releases for all third party dependencies and frameworks. That is a policy to minimize any technical debt within our products. Obviously, Debricked has helped us reduce the period of having a vulnerable third party library in production. Without Debricked we had to stay alert for any critical vulnerabilities, while now we simply get an immediate notification.
Static Application Security Testing: PHPStan and SonarCloud
In the context of secure coding guidelines we have explored various code quality tools. Our prime goal was to force all our engineers to follow the best practices in general. We have initially picked SonarCloud that can work with many languages. However the out-of-the-box version of SonarCloud was somewhat too rigid. Adding custom rules proved to be cumbersome and required a more complex build setup. Moreover, we were ideally looking to run some basic SAST rules as git hooks to speed up the feedback loop to the engineers. PHPStan proved to be an excellent tool to attach as a pre-commit git hook. We can easily add custom rules in PHPStan. As opposed to SonarCloud PHPStan never produces any false positives.
As of 2024 we are looking for a replacement for SonarCloud. It hasn’t flagged a single true positive vulnerability in our codebase. We know from internal pen tests that there were issues.
DAST: OWASP ZAP
We’ve recently also added OWASP ZAP to our CI/CD. We run ZAP against a copy of the production system (obviously, with test data). ZAP runs as a scheduled job in a headless mode within a docker container. It scans the shadow copy of the system for vulnerabilities. ZAP fails the CI/CD pipeline if the scan detects vulnerabilities above a certain threshold. Our CI/CD platform is integrated with Slack and failed pipelines will notify the product team. ZAP also produces a nice report with all findings, hence the product teams can view and correct the situation.
Conclusion
The application security challenge is an essential complexity all software vendors face. Security tooling promises to solve many of these challenges. The number of security tools is growing. There are in fact way more vendors than the size of the market can support. Too often organizations introduce security tools in an ad-hoc way as a mechanism to tackle the security challenge. As a result the tools are now becoming part of the problem. Nearly all typical DevSecOps tools generate lots of security warnings, many of which are either false positives or not exploitable. As a result security teams want less tools.
I believe a security assurance programme based on risks should drive the selection of the right security tools for your DevSecOps pipeline. Not the other way around. You should also demonstrate security improvements in terms of meaningful metrics rather than the sheer number of vulnerabilities found by the tools. Software Assurance Maturity Model (SAMM) is a simple, measurable, and technology / organization agnostic programme I recommend you to start with. Setting up your DevSecOps pipeline and picking the right tools for it will flow naturally once you adopt SAMM.
