How to conduct effective SAMM assessments

Updated: 1 November, 2024

28 December, 2023

For the past 20 years web applications have always been the number one action vector for incidents and breaches (DBIR Verizon 2023).

OWASP SAMM offers a systematic approach to tackling the challenges in application security. Amongst the key philosophies in SAMM is the idea of assessing the current situation and creating an incremental roadmap of improvements driven by risk. Creating and implementing an effective roadmap is a challenging task on its own. However to get to that starting point you will need to conduct a realistic assessment of the organization’s security practices. An imprecise assessment might lead to a suboptimal allocation of resources in the improvement phase.

In this article, we provide some guidelines of how to run assessments. This blog is my personal reflection on the SAMM assessment process. It is however based on the combined experience of four OWASP SAMM core team members, namely myself, Brian Glas, Rob van der Veer and Maxim Baelen that was published on the OWASP SAMM page.

In the first part of this article, I will briefly highlight the pros and cons of doing self-assessments vs inviting an (independent) expert assessment team. Then I provide some insights on how to conduct effective SAMM assessments. I first describe how to plan and prepare for the interviews. Then, I focus on the communication skills that are required for conducting an effective interview.

Key takeaways:

• Self-assessments are easier and cheaper to run. However they are inherently biased and might lead to results being too optimistic. Hence, we recommend conducting expert assessments.
• Expert assessor could be an external team (e.g. one of the SAMM practitioners) or an internal SAMM expert assessor. Expert assessments will have less biases and end up in a more precise assessment.
• When conducting an external assessment prepare and create an assessment plan. Select the right stakeholders who are likely to know most of the organizational realities.
• Set up several interviews with the key stakeholders, ideally a day or two apart.
• Allow a more natural flow of the conversation. Avoid questionnaire-like interviews. Ask more general questions (e.g., tell us about your build process). Then direct the flow of the interview towards the details (e.g., what about application secrets).
• Avoid negativity or judgement. Never tell the interviewees how they should be doing certain things (unless they explicitly ask for best practices). Allow the stakeholders to talk freely. Be supportive, courteous, friendly, respectful and humble.
• Focus on the endgoal, namely improving the organization’s security posture. The current assessment is just a starting point.

Why are security assessments challenging?

Security assessments in general and SAMM assessments in particular are not simple. Amongst the key reasons for assessments being difficult are:

• Stakeholders involved in the assessment might feel they are being audited, which will put them in a defensive mindset. Hence, they might (sometimes even subconsciously) present the security realities through more rosy glasses.
• Security professionals sometimes have an inherent conflict with the dev teams. Hence, they might present the facts through a more pessimistic viewpoint.
• Self-assessments are likely to be more subjective. However self-assessors are more familiar with the organizational realities. Furthermore self-assessments are cheaper to prepare, to conduct and they result in a positive learning effect.
• Expert assessments are likely to be more objective. However they might end up being based on a less thorough understanding of the organization’s realities.

Let us first discuss the main differences between a self-assessment and an expert assessment.

Self-assessments vs independent expert assessments

There are 2 possible modes when running an assessment. Each has its own pros and cons. Ideally, for an organization it is a good idea to have a mix of self-assessments and expert assessments. The precise combination of these two falls outside the scope of this blog.

Self-assessments

These are quicker and cheaper to arrange. Self-assessments typically do not require interviews because the assessor within the team is already familiar with the security realities. A self-assessment is also one of the easiest ways for the team to get familiar with SAMM in a more hands-on manner.

Unfortunately, there are also many potential downsides for self-assessments. First of all, self-assessments are typically too positive due to inherent biases. Furthermore, misinterpretations and lack of expertise when it comes to understanding the SAMM model could further reduce the precision of the assessment. In the extreme cases the misinterpretation may lead to shallow implementations or even “cargo cults” with blind imitation of the security practices without understanding the underlying principles or reasons. For instance, in Education and Guidance practice there is a security activity that requires developers to follow a training. If everyone has a mandatory training of 1 hour per year, the assessor may be inclined to tick off all the boxes for this activity.

Another source of self-assessment issues stems from the quality criteria in SAMM activities. These are mandatory and provide additional guidance on the definitions of done for each activity. However there are scenarios where certain criteria might be subject to interpretation. For certain cases they might even not apply (e.g., IoT devices that do not have over-the-air updates eventually requiring someone to put the security credentials).

Expert assessments

Expert assessments are unlikely to suffer from the drawbacks of the self-assessments. They are by design more objective. Expert assessors typically do not have any incentives to exaggerate or downplay the security realities. Expert assessments are going to require more time and budget. The interviews only will typically take 6 to 9 hours from the stakeholders.

Independent vs internal expert assessments

Expert assessments could be conducted by a different team within the same organization or by an external team of consultants. There are some nuances for the two options each having their pros and cons. 

The internal team within the same organization is likely to have more context and knowledge of the organizational practices and activities. The Governance and Operations business functions are typically handled at the organizational level. There are often additional constraints, processes and guidelines at the level of the team. However an internal team will be a lot quicker to familiarize themselves with this information. The internal team is also more likely to know about risks throughout the organization and end up providing a more suitable improvement roadmap. On the other hand, the internal team might have some pressures (e.g., from the board) to downplay certain issues. Finally, the internal team might also get pressure from the team they are assessing depending on the organizational hierarchy.

The external assessment team will be truly objective and unlikely to downplay any observations. However they will be less familiar with the organizational realities and especially the risk appetite of the board. This is not a hurdle that is impossible to overcome though. Especially for running an assessment in several teams within the same organization the external team is likely to excel.

Whether you run an assessment as an external or internal expert team there are still a number of best practices when conducting the assessment interviews. In the remainder of this blog we highlight some of them. For a more complete description we refer to our blog in the OWASP SAMM web page.

Expert assessment interviews

In this final part of this blog we will focus on two key aspects of conducting effective SAMM assessments. Firstly, we will describe the practicalities of planning and conducting an interview. Then, we will focus on the soft skills and the communication aspect of the interview.

Assessment interview essentials

“Failing to plan is planning to fail.”

Interview format and planning

Everyone has their own preferred format for conducting the assessment interviews. The interview format may also depend on the expected security maturity within the assessed team. More mature teams are likely to need more time to go through all their security practices. My experience suggests that you should account for at least 6 to 9 hours of interviews. For teams that are expected to have a lower score (e.g., 1 or less) the assessment typically would take 3-4 hours.

Here is a sample interview format and selection of topics that we prefer to follow with Brian. Brian is our colleague at Codific, but he is also the OWASP SAMM core team member and OWASP Top 10 and SAMM Benchmark project lead. 

In terms of planning it is a good idea to keep the interview sessions at least a day or two apart. This gives some time to reflect on what was discussed during the previous sessions and perhaps even gather some additional documentation. We typically schedule 2 to 3 interviews per week.

Preparation

It is a good idea to look at the model before the interview just to keep things fresh in your memory. You need to have a fairly good understanding of SAMM details. To be honest, I don’t think memorising the full SAMM structure by heart is an overkill. In any case make sure you don’t have to fumble for the model details instead of devoting your full attention to the interview.

More importantly, make sure to familiarize yourself with the organization you will be assessing. Brian and myself always ask for any security-related documents, such as organizational policies and standards, process-related documents and artefacts from completed activities. Not everyone on the core team is in favour of this approach though. The downside of asking these in advance is that the organization might create these artifacts for the sole purpose of the assessment that beats their purpose.

Kick-off meeting

We believe the ideal kick-off meeting is organised by the SAMM champion within the organization. The SAMM champion could be the CISO or the AppSec director. Asking him or her to set the context of the assessment is a great starting point. When introducing yourself make it more personal and invite others to do the same. Make sure to listen and remember the names and stories of everyone in the conversation.

The kick-off meeting is also a great moment to plan the sessions in everyone’s agenda. Describe the interview structure and what is to come during the sessions. More importantly though, stress the fact that the assessment is NOT going to be an exam or an audit. It is of paramount importance that the SAMM champion within the organization also clearly communicates this.

What are the key skills needed to run an effective SAMM assessment

“Strong communication skills can help you build trust, credibility, and influence.”

It is not an exam nor an audit

I can’t stress this enough, but SAMM assessments are not about the score. The score is just a starting point in the journey. If there is no journey, getting a score is meaningless. Having a maximum average score of 3.00 is virtually impossible. Your focus should be to understand the risks, the current controls and creating the most effective improvement roadmap in terms of ROI. Organizations should get “just enough security”, not a SWAT team around a bank vault protecting the lunch money. Obviously, if you are processing extremely sensitive data (e.g., patient records) you might need the metaphorical SWAT team and the vault. However you still need to get to the point where you risk are reduced to an acceptable degree. 

In this context, trying to cheat and paint a more rosy view of the security realities would be the worst possible outcome for the organization and the team. By clearly conveying to everyone that the assessment is not an audit you will hopefully help everyone relax and tell the real story behind the security activities in their team. However feel free to ask for artefacts early in the interview process. This will ensure that interviewees are less likely to exaggerate.

Keep a natural conversation

Avoid giving the impression of an interrogation to the interviewees. To achieve this turn the interview into a natural conversation. This means that you will have to improvise at times and switch the order of the questions you have prepared. If the interviewees digress a little that is quite okay. However try to steer the conversation in a gentle manner.

To make sure you keep a natural conversation flow try to ask the broad and open-ended questions. Here are some examples:

• Don’t ask whether the team leverages STRIDE or LINDDUN methodologies. Ask how the team threat models and if they say they don’t, ask them how do they consider the likelihood and impact of bad things happening. 
• Ask the interviewees to describe their build process.
• Don’t ask whether they leverage Amazon KMS. Ask them how they deal with production secrets. Ask how they envision improving the secret management process.

Be empathic

Avoid judgement and comparison

Even if people understand that the assessment is not an exam, any hints of judgement will put them in a defensive mode. That is probably the worst thing to happen during the interview process. Avoid comparing their experiences to your own ones (unless you have experienced the same).

Put yourself in their shoes

From an emotional perspective as an interviewer you need to consider the daily realities of the team. They are building software systems and security was most likely not even a thing a while ago. I have seen a lot of teams who are overworked and stressed due to the sheer amount of functional work they have to deliver. Try to relate to the stakeholders on the emotional level and be supportive. As a security expert your expectations are probably high and you would like to see everyone getting closer to maturity level 3. However, depending on their risk appetite for certain practices it is possible that the team doesn’t really need to be at level 3.

Listen, rephrase and encourage

Engage in deep listening. Pay attention not only to words, but also to nonverbal cues like body language, facial experssions and tone of voice. Encourage the responses with enthusiasm. Paraphrasing is a great idea as well. Avoid discussing how things could be improved unless they explicitly ask for it.

Conclusion

OWASP SAMM is a valuable framework for organizations seeking to improve their application security posture. By conducting a realistic assessment of their current state, organizations can identify areas for improvement and develop a targeted roadmap for remediation. While self-assessments can be conducted, engaging an independent expert assessment team can provide additional benefits, such as a more objective perspective and specialized expertise. The two key aspects for an effective SAMM assessment are preparation and powerful communication skills.

Additional Resources

If you want to learn more about OWASP SAMM you can take the free SAMM training.

No matter which assessment mode you pick, SAMMY is the most mature OWASP SAMM supporting tool that will help you in your security journey.