Updated: 29 February, 2024
28 February, 2024
Application security requires a systematic approach and requires dealing with software security throughout every stage of the software development lifecycle. However organizations typically struggle to create an effective improvement roadmap and end up in the rabbit hole of fixing security tool generated vulnerabilities. We believe that leveraging OWASP Application Security Verification Standard (ASVS) as a security requirements framework as well as a guide to unit and integration testing is by far the best pick in terms of ROI. Using requirements-driven testing and by turning security requirements into “just requirements” organizations can enable a common language shared by all stakeholders involved in the SDLC.
We have analyzed the complete ASVS to determine how much of it we could automate using various testing strategies. Our analysis indicates that 162 (~58%) of ASVS can be automatically verified using unit, integration, acceptance tests. We could increase the verifiability by another 10% by using SAST, DAST and SCA tooling.
We have also designed an empirical study where we have added 98 ASVS requirements to the sprint planning of a relatively large web application. Our team has followed a security test-driven development approach. A test engineer has written unit and integration tests for as many requirements as possible in 8 man-days. Our team has implemented test cases for 90 ASVS requirements. From this effort we have created 7 tickets for which the project did not comply with the standard.
Our study demonstrates that leveraging ASVS for requirements-driven testing can create a common theme across all stages of the software development lifecycle making security everyone’s responsibility.
Key takeaways:
- Start from ASVS and invest in security requirements that is the lingua franca between all stakeholders.
- Require your test engineers to create unit, integration and acceptance tests for as many security requirements as possible.
- Explicit security requirements provide a hands-on training and awareness for all stakeholders.
- Requirements driven testing removes the silos between sec and dev teams.
- To total cost for adopting requirements driven testing is negligible.
Introduction
In order to solve the application security challenges organizations need to adopt a systematic approach to dealing with security. In a nutshell that requires dealing with security throughout the complete software development lifecycle. A highly simplified version of OWASP SAMM would typically recommend at least the following activities throughout the software development lifecycle:
- Executive management defines and communicates to everyone the organization’s risk appetite and risk tolerance.
- Business analysts explicitly define security requirements when creating the high-level application specification.
- Architects employ security principles and patterns in order to create a (reusable) secure software architecture/design.
- Developers comply with various security policies and implement the security requirements.
- Test engineers verify amongst others the correct implementation of those security requirements.
- Operation engineers deploy the software system on top of hardened components and patch them regularly.
Unfortunately organizations (especially those with low security maturity) often get overwhelmed by the breadth of security activities and end up focusing on the lowest hanging fruits. Software teams typically leverage penetration testing and various security tools in order to find vulnerabilities and remediate them after the software system has been developed. Obviously, fixing security defects after the system has been developed is much more costly than dealing with it at the design level.
Security Requirements Frameworks
We believe that leveraging OWASP ASVS to enable requirements-driven testing is the best pick for a corporate security strategy. Requirements-driven testing will likely bring the best returns on investment for your AppSec programme. Your organization could actually create a corporate set of standards based on ASVS. Adding security requirements to the sprint planning fits naturally with the overall software development process. Dev teams will consider security as “just requirements” that are part of a nice and shiny feature. Test engineers will derive test cases for those security requirements and automate them just like for any other test scenario. As a result your dev and test teams will get their awareness and training in a hands-on fashion, which will have a more lasting impact than mandatory yearly training. Requirements are pervasive and will involve nearly every role involved in the SDLC. Security requirements will enable effective communication between sec and dev teams. Finally, adding this common thread will condition your teams for the higher maturity-level initiatives.
How much of ASVS can be automatically verified
In this blog, we present the case of leveraging OWASP ASVS as a collection of requirements that we add to the sprint planning. We subsequently include these requirements in automated unit, integration and acceptance tests. Our goal was to maximize the portion of ASVS that is poured into automated security regression test suites, thereby making security much more tangible for the sec, dev and test teams. Furthermore, we have also investigated how much of ASVS could be tested using SAST, DAST and SCA tooling. Note that we have looked at ASVS across all verification levels.
Our analysis concludes that roughly 58% of ASVS can be at least partially leveraged as “just requirements” and added to the sprint planning. 37% of ASVS can be verified using unit tests, 20% using integration tests and 1% using acceptance tests.
We can extend the ASVS automated verifiability by another 10% by leveraging SAST, DAST and SCA tooling. Note that some ASVS requirements are best tackled using multiple testing strategies. For instance, verifying output encoding should be done by using both unit tests as well as SAST to ensure no unsafe html methods are used. Our study concludes that about 14% of the requirements need more than 1 testing strategy.
Note that some ASVS requirements cannot be fully verified. We believe you can fully verify 61.2% and partially verify 6.8% of the requirements.
Creating unit and integration tests from ASVS for a large-scale web application
Aside from a theoretical analysis we have also done an empirical study for one of our SaaS products (a web application written in Symfony PHP framework). Our experimental design was slightly different from an ideal scenario. More specifically, our development team did not really focus explicitly on implementing any ASVS requirements. As it turned out though, our dev team had actually implemented many of them. So we have added 98 ASVS requirements to a dedicated sprint. We have requested one of our test engineers to write unit and integration tests for these requirements. Strictly speaking we have followed a security test-driven development.
Our team has implemented 90 (32.4%) of the ASVS requirements using unit and integration tests. A number of tests have actually failed indicating the requirements were not (correctly) implemented in the product. The test implementation effort was an 8 man-day time-boxed activity for one test engineer. Hence, we have eventually dropped 8 requirements due to time limitations. A senior security analyst has helped derive and refine the test cases. The total time-investment by the security analyst was about 2 man-days. To put this into perspective, we have spent about 4 man-years on the SaaS product.
Threats to validity for our experiment
Our study has a number of validity threats and we will discuss the top 3 of them. Firstly, our analyst and test engineer were already familiar with ASVS. We also have some ASVS-based unit and integration tests throughout other projects from which we could easily recycle them. Perhaps less relevant, but most of ASVS requirements were already implemented in the SaaS product making the testing somewhat easier. We believe that leveraging ASVS for requirements-driven testing is likely to require more effort in an organization that is not familiar with the standard. However, such organizations will likely have increasing returns on investment.
It is possible that we may have misinterpreted some ASVS requirements and fell prey to the “cargo cult” ending up with shallow implementations.
Finally, some of the automated tests only provide a partial implementation of their respective ASVS requirements, especially where SAST/DAST/SCA tools are to be leveraged.
Experimental data
We have currently published our analysis in a Google Spreadsheet. We plan to publish the actual test cases as well.
Conclusion
To effectively tackle the application security challenge organizations need to deal with security throughout the entire software development lifecycle. However organizations often struggle to find the best strategy to bootstrap that effort. We believe that leveraging ASVS as a guide to requirements-driven testing is the most effective approach to ensure everyone involved in the SDLC is responsible for security. Business analysts will leverage ASVS to create the application specification. Architects will have to deal with these in a very concrete way. Developers will have to implement them as “just requirements”. Finally, test engineers will have to verify them using various testing strategies. Hence, creating security requirements is a non-invasive approach to make security everyone’s responsibility.
Having explicit security requirements in the SDLC creates a common thread that is everyone’s responsibility. ASVS-driven security requirements create awareness in a natural way.
