How to do SAMM assessments: Everything you need to know

Starting a OWASP SAMM assessment journey can feel overwhelming. SAMM, short for Software Assurance Maturity Model, is the process of guiding organisations towards understanding their current security maturity within a defined scope, whether its a team, business unit, or the entire organisation.

At its essence, SAMM assessment involves interpreting the layers of security practices through a structured questionnaire. This questionnaire is composed of 90 multiple-choice questions and a set of quality criteria defining the benchmark for each query. However, mastering the art of conducting a precise SAMM assessment presents its own set of challenges. Determining the right personnel to lead the assessment, ensuring objective scoring, and what is necessary to prepare in advance from both interviewees and interviewers are just a few hurdles organisations encounter along the way. 

The importance of assessing security maturity cannot be overstated. Beyond serving as a foundational step towards enhancement, it proves invaluable in scenarios like mergers and acquisitions, where understanding the security posture of an organisation becomes critical.

Insights from industry experts

On our podcast, our CEO Aram hosted a panel of leading industry experts to gain insights into how to properly do SAMM assessment. Joining Aram were Rob van der Veer, Brian Glas and Maxim Baele, all esteemed members of the core team behind OWASP SAMM. Together, they explored the nuances of SAMM assessment, shedding light on the process, practical implementation strategies, best practices, and insider tips to streamline the process. 

Listen to the podcast:

Below, CEO Aram Hovsepyan will highlight the pros and cons of doing self-assessments vs inviting an (independent) expert assessment team. Then we provide some insights on how to conduct effective SAMM assessments. Describing how to plan and prepare for the interviews. Then, focusing on the communication skills that are required for conducting an effective interview.

Self-assessments vs independent expert assessments

Highlighted in the table are the main difference between a self-assessment and an expert assessment. There are 2 possible modes when running an assessment. Each has its own pros and cons. Ideally, for an organization it is a good idea to have a mix of self-assessments and expert assessments. The precise combination of these two falls outside the scope of this blog.

Self-assessments

These are quicker and cheaper to arrange. Self-assessments typically do not require interviews because the assessor within the team is already familiar with the security realities. A self-assessment is also one of the easiest ways for the team to get familiar with SAMM in a more hands-on manner.

Unfortunately, there are also many potential downsides for self-assessments. First of all, self-assessments are typically too positive due to inherent biases. Furthermore, misinterpretations and lack of expertise when it comes to understanding the SAMM model could further reduce the precision of the assessment. In the extreme cases the misinterpretation may lead to shallow implementations or even “cargo cults” with blind imitation of the security practices without understanding the underlying principles or reasons. For instance, in Education and Guidance practice there is a security activity that requires developers to follow a training. If everyone has a mandatory training of 1 hour per year, the assessor may be inclined to tick off all the boxes for this activity.

Another source of self-assessment issues stems from the quality criteria in SAMM activities. These are mandatory and provide additional guidance on the definitions of done for each activity. However there are scenarios where certain criteria might be subject to interpretation. For certain cases they might even not apply (e.g., IoT devices that do not have over-the-air updates eventually requiring someone to put the security credentials).

Expert assessments

Expert assessments are unlikely to suffer from the drawbacks of the self-assessments. They are by design more objective. Expert assessors typically do not have any incentives to exaggerate or downplay the security realities. Expert assessments are going to require more time and budget. The interviews only will typically take 6 to 9 hours from the stakeholders.

Independent vs internal expert assessments

Expert assessments could be conducted by a different team within the same organization or by an external team of consultants. There are some nuances for the two options each having their pros and cons. 

The internal team within the same organization is likely to have more context and knowledge of the organizational practices and activities. The Governance and Operations business functions are typically handled at the organizational level. There are often additional constraints, processes and guidelines at the level of the team. However an internal team will be a lot quicker to familiarize themselves with this information. The internal team is also more likely to know about risks throughout the organization and end up providing a more suitable improvement roadmap. On the other hand, the internal team might have some pressures (e.g., from the board) to downplay certain issues. Finally, the internal team might also get pressure from the team they are assessing depending on the organizational hierarchy.

The external assessment team will be truly objective and unlikely to downplay any observations. However they will be less familiar with the organizational realities and especially the risk appetite of the board. This is not a hurdle that is impossible to overcome though. Especially for running an assessment in several teams within the same organization the external team is likely to excel.

Whether you run an assessment as an external or internal expert team there are still a number of best practices when conducting the assessment interviews. In the remainder of this blog we highlight some of them. For a more complete description we refer to our blog in the OWASP SAMM web page.

Expert assessment interviews

In this final part of this blog we will focus on two key aspects of conducting effective SAMM assessments. Firstly, we will describe the practicalities of planning and conducting an interview. Then, we will focus on the soft skills and the communication aspect of the interview.

Assessment interview essentials

“Failing to plan is planning to fail.”

Interview format and planning

Everyone has their own preferred format for conducting the assessment interviews. The interview format may also depend on the expected security maturity within the assessed team. More mature teams are likely to need more time to go through all their security practices. My experience suggests that you should account for at least 6 to 9 hours of interviews. For teams that are expected to have a lower score (e.g., 1 or less) the assessment typically would take 3-4 hours.

Here is a sample interview format and selection of topics that we prefer to follow with Brian. Brian is our colleague at Codific, but he is also the OWASP SAMM core team member and OWASP Top 10 and SAMM Benchmark project lead. 

In terms of planning it is a good idea to keep the interview sessions at least a day or two apart. This gives some time to reflect on what was discussed during the previous sessions and perhaps even gather some additional documentation. We typically schedule 2 to 3 interviews per week.

Preparation

It is a good idea to look at the model before the interview just to keep things fresh in your memory. You need to have a fairly good understanding of SAMM details. To be honest, I don’t think memorising the full SAMM structure by heart is an overkill. In any case make sure you don’t have to fumble for the model details instead of devoting your full attention to the interview.

More importantly, make sure to familiarize yourself with the organization you will be assessing. Brian and myself always ask for any security-related documents, such as organizational policies and standards, process-related documents and artefacts from completed activities. Not everyone on the core team is in favour of this approach though. The downside of asking these in advance is that the organization might create these artifacts for the sole purpose of the assessment that beats their purpose.

Kick-off meeting

We believe the ideal kick-off meeting is organised by the SAMM champion within the organization. The SAMM champion could be the CISO or the AppSec director. Asking him or her to set the context of the assessment is a great starting point. When introducing yourself make it more personal and invite others to do the same. Make sure to listen and remember the names and stories of everyone in the conversation.

The kick-off meeting is also a great moment to plan the sessions in everyone’s agenda. Describe the interview structure and what is to come during the sessions. More importantly though, stress the fact that the assessment is NOT going to be an exam or an audit. It is of paramount importance that the SAMM champion within the organization also clearly communicates this.

What are the key skills needed to run an effective SAMM assessment

“Strong communication skills can help you build trust, credibility, and influence.”

It is not an exam nor an audit

I can’t stress this enough, but SAMM assessments are not about the score. The score is just a starting point in the journey. If there is no journey, getting a score is meaningless. Having a maximum average score of 3.00 is virtually impossible. Your focus should be to understand the risks, the current controls and creating the most effective improvement roadmap in terms of ROI. Organizations should get “just enough security”, not a SWAT team around a bank vault protecting the lunch money. Obviously, if you are processing extremely sensitive data (e.g., patient records) you might need the metaphorical SWAT team and the vault. However you still need to get to the point where you risk are reduced to an acceptable degree. 

In this context, trying to cheat and paint a more rosy view of the security realities would be the worst possible outcome for the organization and the team. By clearly conveying to everyone that the assessment is not an audit you will hopefully help everyone relax and tell the real story behind the security activities in their team. However feel free to ask for artefacts early in the interview process. This will ensure that interviewees are less likely to exaggerate.

Keep a natural conversation

Avoid giving the impression of an interrogation to the interviewees. To achieve this turn the interview into a natural conversation. This means that you will have to improvise at times and switch the order of the questions you have prepared. If the interviewees digress a little that is quite okay. However try to steer the conversation in a gentle manner.

To make sure you keep a natural conversation flow try to ask the broad and open-ended questions. Here are some examples:

• Don’t ask whether the team leverages STRIDE or LINDDUN methodologies. Ask how the team threat models and if they say they don’t, ask them how do they consider the likelihood and impact of bad things happening. 
• Ask the interviewees to describe their build process.
• Don’t ask whether they leverage Amazon KMS. Ask them how they deal with production secrets. Ask how they envision improving the secret management process.

Be empathic

Avoid judgement and comparison

Even if people understand that the assessment is not an exam, any hints of judgement will put them in a defensive mode. That is probably the worst thing to happen during the interview process. Avoid comparing their experiences to your own ones (unless you have experienced the same).

Put yourself in their shoes

From an emotional perspective as an interviewer you need to consider the daily realities of the team. They are building software systems and security was most likely not even a thing a while ago. I have seen a lot of teams who are overworked and stressed due to the sheer amount of functional work they have to deliver. Try to relate to the stakeholders on the emotional level and be supportive. As a security expert your expectations are probably high and you would like to see everyone getting closer to maturity level 3. However, depending on their risk appetite for certain practices it is possible that the team doesn’t really need to be at level 3.

Listen, rephrase and encourage

Engage in deep listening. Pay attention not only to words, but also to nonverbal cues like body language, facial experssions and tone of voice. Encourage the responses with enthusiasm. Paraphrasing is a great idea as well. Avoid discussing how things could be improved unless they explicitly ask for it.

Conclusion

OWASP SAMM is a valuable framework for organizations seeking to improve their application security posture. By conducting a realistic assessment of their current state, organizations can identify areas for improvement and develop a targeted roadmap for remediation. While self-assessments can be conducted, engaging an independent expert assessment team can provide additional benefits, such as a more objective perspective and specialized expertise. The two key aspects for an effective SAMM assessment are preparation and powerful communication skills.