20 March, 2024
Application security is a paramount concern for organizations that develop software. However systematically managing AppSec across diverse development teams in a measurable way remains a challenge. This blog outlines Zebra Technologies’ journey in adopting the OWASP Software Assurance Maturity Model (SAMM) as our guiding framework for measuring and improving application security practices. Zebra is a Fortune 500 company with 35 different product and IT teams developing and maintaining secure software applications and systems. Though you may not always see them, Zebra is everywhere. Whether it’s on that recently delivered package, or the water bottle you just scanned at the supermarket – even in the shoulder pads of NFL players, Zebra is there.
Key Takeaways
- Despite initial scepticism and the inherent challenges of integrating SAMM, particularly with embedded and delivered software teams, the implementation led to significant improvements.
- The introduction of SAMM facilitated a risk-driven, measurable approach to security.
- SAMM provided a clear framework for comparison across business units and promoting a shared platform for discussing security concerns. Moreover, the gamification of SAMM scores spurred healthy competition among units, though it raised questions about the focus on risk-based improvements versus score chasing.
- Ultimately, the correlation between SAMM scores and other quality metrics affirmed the value of a SAMM-driven approach. We have seen a moderate (-0.5) inverse correlation between SAMM scores and metrics produced by the Nucleus tool, which Zebra uses across all teams.
Overall, SAMM demonstrated tangible enhancements in application security and broader software development lifecycle processes at Zebra Technologies.
Introduction
Application security is the number one concern at any organization that develops software. While recognizing the problem is the first step, dealing with application security across every development team is extremely challenging. More importantly, there are no role models to learn from. Furthermore, despite being part of the solution, security tools might end up making the problems worse.
Looking at the problem in a top-down fashion it is obvious that the right solution should be risk-driven and measurable. Without risk there is no need for security! If you don’t measure you can’t improve! That is precisely what application security programs offer as part of the solution. However, surprisingly there are not too many AppSec programs to pick from. The various certification standards (e.g. ISO27001, FedRamp, SOC2) are all risk-focused, but provide no real measurability. Hence, NIST SSDF, BSIMM and OWASP SAMM were the only three frameworks that made our shortlist. SSDF is somewhat unclear when it comes to the measurability aspect. BSIMM is hard to comprehend, and privately owned. SAMM was clear, concise and easy to measure. Hence, in 2021 we have introduced OWASP SAMM as an application security management programme at Zebra Technologies.
While it was not a smooth journey, in the past three years Zebra has seen remarkable improvements throughout all their teams. First of all, SAMM provided an objective framework to compare the different business units to each other. Zebra has seen an immensely positive impact on awareness. Finally, SAMM scores across different business units correlated with the risk scores produced by the Nucleus Application Security Posture Management (ASPM) tool. Hence, we strongly believe that SAMM scores can be leveraged as predictors of general maturity in the SDLC.
SAMM Scopes per Business Unit
Zebra Technologies has introduced SAMM as a framework to measure and improve the application security posture across all business units (BU). They have created SAMM scopes at the level of the BU. Hence, every BU got a single set of SAMM scores after each assessment.
Zebra already had weekly security forums with all BU leads and SAMM became part of the agenda topics during these meetings. There was quite some pushback when SAMM was introduced. Many BU leads were initially sceptical how SAMM would be different from other security tools. Interpreting certain SAMM activities and quality criteria was hard. That was especially true for the embedded and delivered software teams. SAMM seems somewhat more focused on web and mobile applications.
One last challenge inherent to SAMM was the fact that governance and operations are business functions that cross the boundaries of a business unit (i.e., scopes in SAMM). These are typically activities that are handled at the level of the organization. Hence, Zebra had to introduce a more coordinated approach for communicating and discussing these issues across the entire organization. However, discussing concepts like strategy, metrics, risk and training provides a clear rationale to the “why” question. These discussions have created the necessary common language and the buy in from the different teams. We believe that thanks to this unique characteristic of SAMM Zebra has seen quite a nice transition from introducing SAMM towards a clear understanding of its goals.
Security Gamification
An (un)expected side-effect from introducing SAMM was its gamification aspect. All BU SAMM scores were published on a corporate dashboard. The executive board had an eye on that board as well. This has improved the awareness aspect of both the programme as well as the security practices. Furthermore, introducing SAMM dashboards has resulted in unofficial inter-BU competition to try to get a higher score. To ensure objectivity Zebra has involved Codific as an external SAMM practitioner to validate the scores.
However, gamification is a controversial topic within OWASP SAMM. One of the main SAMM philosophies is improvements based on risk. Focusing on the scores might be a waste of resources. This is why Zebra has eventually moved away from an absolute SAMM score towards the so-called “percentage to target” metric discussed in the next section.
Corporate Benchmarking
Security is all about risk management. However SAMM does not offer (for now) any benchmarks of what the eventual target scores should be. Initially, at the corporate level Zebra has introduced a minimum score per SAMM security activity that every BU had to achieve. Some BUs had to get a higher score given their inherent risk. They reported the absolute SAMM scores as well as the gap to the minimum on a quarterly basis to the board.
However, most of the teams were ambitious and aspired to get an average score of 3, which is neither practical nor recommended. Going above and beyond in application security is counterproductive. For instance, a software system that runs on-prem and does not process any sensitive data inherently has a different risk appetite than a cloud-based web application that processes patient records. Hence, Zebra has switched the reporting to mainly focus on the gap to the minimum set of scores dictated by the corporate SDLC policy.
Yet another issue with the gap to minimum was that teams would typically look for the easiest SAMM activities to improve. However, prioritization should be driven by risk. Thus Zebra has refined the definition of the gap to minimum. They have introduced a metric called percentage to target that calculates the average percentage of the gap per SAMM question rather than the total. In simple terms, the new metric does not provide any bonus points for overachieving. This has largely solved the problems and created a single score from 0 to 100 that is easy to communicate.
Correlation with Risk and Other Metrics
In addition to SAMM Zebra also report a number of other metrics to the executive board on a quarterly basis. One of these metrics is a risk score calculated by an Application Security Posture Management (ASPM) tool. The tool calculates these scores by analyzing the security configuration, vulnerabilities, and threats within an application’s environment to estimate the risk of a security incident.
We have analyzed the risk scores produced by an ASPM tool and SAMM scores. We have determined that there is a consistent moderate inverse correlation between the two metrics. In other words, high SAMM scores correlate with low risk scores in ASPM tools. Hence, we believe that a SAMM-driven approach to application security is likely to lead to improvements in other quality attributes for the software development lifecycle. Inversely, we have noticed that mature teams following best practices in software development typically scored much higher on SAMM. Hence, organizations can leverage SAMM as a predictor of general maturity.
Conclusion
About 3 years ago Zebra has simultaneously introduced OWASP SAMM to all its business units. Zebra has created a dashboard to show the BU SAMM scores and report them to the executive board. Despite a number of obstacles SAMM has lived up to its promises. It has enabled a measurable approach to application security. Awareness throughout Zebra teams grew substantially. As expected, higher SAMM scores correlated with improvements throughout other measurements. SAMM is like a pair of glasses for visualizing AppSec processes, simplifying the task of identifying gaps and prioritizing enhancements.
