12 June, 2024
OWASP Software Assurance Maturity Model (SAMM) is becoming an industry standard application security program framework. That is hardly a surprise as SAMM provides a measurement-based approach to improving product security. SAMM is a very mature framework dating back to 2009. In a nutshell, SAMM provides you with the necessary tools to measure your software maturity on a scale from 0 to 3.
However, organizations often struggle to figure out where to go after their initial SAMM assessment. At the two opposite extremes we have seen teams that try to go to the maximum score as fast as possible. Other teams struggle to figure out which activity to pick for improvement. Up until now SAMM provided some guidance. “Never try to boil the ocean” and “start from risk”. Nonetheless, getting some real-world data in terms of how other organizations are doing is by far one of the hottest topics around the project.
Today (12th of June 2024), Brian Glas who leads the Benchmarking project within the SAMM core team has shared the first data set. If you ask me, the “SAMM Top 10” is likely to become the hottest meaningful Top 10 effort within the OWASP community.
Benchmark demographics
The Industry Benchmark OWASP SAMM score is calculated based on assessment results that are submitted to the SAMM Benchmarking project. So far the number of submissions is relatively low (25). In the coming weeks and months the data will be updated and we will update this page.
Looking at the metadata behind the actual results there are several interesting observations to note.
The majority of submissions are from large multinational companies
The dataset largely represents submissions by large and global companies. There is a sizeable representation of small companies in the dataset. However the percentage of medium companies is very low. We would hope that more medium-sized companies will send their results to the Benchmarking project. The way the size question is formulated here is that “small” is an organization with less than 100 developers and “large” has more than 1000 developers.
Third-party assessments
Most of the submissions are based on third-party assessments. That is great news as third-party assessments are typically more precise and objective. In this podcast, SAMM veterans including Aram Hovsepyan, Brian Glas, Maxim Baele and Rob van der Veer shed some light why self-assessments could negatively impact the quality of data.
The average Industry Benchmark OWASP SAMM score
The average overall OWASP SAMM score is 1.43 out of 3. This is based on the data in the official OWASP SAMM dataset as it stood on June the 12th, 2024.
Industry benchmark score for Governance
The average Industry Benchmark OWASP SAMM score for the Governance business function is 1.33 out of 3. Combined with the fact that the majority of the companies in the data set are large corporations this seems somewhat surprising. It is a known fact that large organizations invest heavily in governance and we would expect the score to be somewhat higher. On the other hand, it is possible that the small organizations in the dataset average out. It would help if we could have more data and aggregate scores per company size.
Industry benchmark score for Design
The average OWASP SAMM score for the Design business functions is 1.45. Perhaps a product of “shifting left”.
Industry benchmark score for Implementation
Implementation is the second highest scoring business function with an average score of 1.48. The whole DevOps culture obviously helps here as many companies are heavily investing in CI/CD automation, which is one of the core topics within the Implementation business function of SAMM.
Industry benchmark score for Verification
Verification is the lowest scoring business function with an average score of 1.14. We don’t really see any surprises here despite the fact that nearly all organizations are heavily focused on security testing tools and penetration testing. However maturity for the verification business function is obviously a lot more than that. We would also argue that investment in design is a waste without the verification aspect of it.
Industry benchmark score for Operations
Finally, Operations is the highest scoring business function in the benchmark with an average score of 1.73. Historically, many organizations are investing heavily in application security areas that constitute operations, such as, incident management, patching and updating, configuration hardening.
The highest scoring security activities across the industry
The highest scoring security activities in SAMM are as follows:
- Incident Management: 1.96
- Security Requirements: 1.72
- Environment Management: 1.71
- Secure Deployment: 1.59
- Secure Architecture: 1.57
It is somewhat surprising to see Security Requirements and Secure Architecture so high in the rankings. “Shifting left” is a clear trend in the dataset, but it makes us wonder whether organizations are doing the right thing. Security Requirements should ideally permeate through the rest of the software development lifecycle in implementation and verification. Hence a high score on Security Requirements should in theory also result in a high score in requirements-driven testing. Not only we do not see that, but also Requirements-Driven Testing is the lowest scoring activity. Our suspicion is confirmed when we look at the Secure Architecture that is amongst the top scorers, with Architecture Assessment and Threat Assessment being amongst the lowest scoring activities.
The lowest scoring security activities across the industry
The lowest scoring activities across the industry benchmark are as follows:
- Requirement-Driven Testing: 1.05
- Architecture Assessment: 1.06
- Threat Assessment: 1.07
- Strategy and Metrics 1.20
- Security Testing: 1.31
In the previous section, we have already discussed the surprises with Requirements-Driven Testing, Architecture Assessment and Threat Assessment. Strategy and Metrics is surprising in theory. Indeed, SAMM advocates starting from the strategy and introducing metrics to figure out whether the improvements are actually meaningful. On the other hand, we have adopted SAMM at Codific about 2.5 years ago. We are pretty clear on strategy, but we haven’t yet figured out the metrics part of the story. We believe more guidance and research is necessary around this area.
Contribute to the OWASP SAMM Benchmark.
The publication of benchmark data has just begun. This dataset is based on 25 quality datasets, but we need much more to be able to provide industry specific breakdowns. The goal is 100 datasets.
Therefore we encourage everyone to contribute.
How can I contribute to the OWASP SAMM Benchmark project?
Link to the folder to contribute data
Stay tuned for more OWASP SAMM benchmark data! This page will update. (last update June 13th)
