How to integrate OWASP ZAP in Gitlab CI/CD pipeline.

29 December, 2022

In this article you will learn how to automate your secure deployment by integrating OWASP ZAP in your GitLab CI/CD pipeline. Doing so for GitHub is largely similar with GitHub’s actions.

Cybersecurity is becoming a priority on a worldwide scale. Both the US (NIST SSDF and Memorandum M-22-18) and the EU (NIS2) have recently introduced legislations to aim at high common level of cybersecurity. Meanwhile, the application security state-of-the-practice in large and small firms is far from ideal (to put it mildly). The attack sophistication and absolute numbers are increasing. The community is playing a whack-a-mole game hoping tooling will solve everything. The security teams on the other hand have already realized we are headed towards a train crash. Security teams want less and not more tools as tooling pushes them down the rabbit hole with thousands of alerts. Paradoxically, we have known the systematic solution to the security challenge. We (the actual security experts) name it a secure software development lifecycle or an AppSec programme.

There are many AppSec programmes out there (NIST SSDF, Microsoft SDLC, BSIMM, etc) with OWASP SAMM being by far the simplest one to start with. The cornerstones of any AppSec programme are People, Process, Knowledge, Tools, Training and Risk. Thus, tooling is just one wheel in the whole machine. To make tooling efficient we should start from the process and fit the tools in the puzzle, rather than trying to reverse engineer the puzzle from the tools. 

In this blog, we present a small part of our AppSec programme, namely secure deploy. We provide a high-level overview of how the process is structured and how we’ve intergrated the tool in our process.

What is the Secure Deployment practice

One of the final stages in delivering secure software is ensuring the security and integrity of developed applications are not compromised during deployment. It focuses on removing manual error by automating the deployment process as much as possible, and making its success contingent upon the outcomes of integrated security verification checks. Secure Deployment also fosters Separation of Duties by making adequately trained, non-developers responsible for deployment.

It also goes beyond the mechanics of deployment. Secure Deployment focuses on protecting the privacy and integrity of sensitive data, such as passwords, tokens, and other secrets, required for applications to operate in production environments. In its simplest form, suitable production secrets are moved from repositories and configuration files into adequately managed digital vaults. In more advanced forms, secrets are dynamically generated at deployment time and routine processes detect and mitigate the presence of any unprotected secrets in the environment.

How do we deploy securely

We dynamically extract all secrets, keys, password for the given application from a digital vault at the time of deployment. Before actually deploying, we run security and functional tests in a pipeline based on ASVS. If any of those tests fails, the deployment process gets halted and the CI/CD sends a message to the team on Slack. As part of the security test suite we run SAST and DAST tool scans. Our DAST “weapon of choice” is OWASP Zed Attack Proxy (ZAP for short).

What is ZAP?

ZAP is free, highly customizable and flexible open source web application tool with a great community behind it. Before getting into how we integrated it into our GitLab deployment pipeline, we highly recommend you to watch the ZAP introduction videos.

In simple terms, ZAP is a toolbox you can leverage to search for vulnerabilities in your web application both manually and in an automated fashion. It’s functionality is abundant, but in this blog we will be focusing on running a basic automated scan and integrating it into a GitLab pipeline.

Overall setup

We run ZAP against a shadow copy of the live deployed system (obviously, with test data). You could also leverage the Dev or Acc instances as a shadow copy. ZAP runs as a scheduled job in a headless mode within a docker container. It scans the shadow copy of the system for vulnerabilities. ZAP is configured to fail the CI/CD pipeline if the scan detects vulnerabilities above a certain threshold. Our CI/CD platform is integrated with Slack and failed pipelines will notify the product team. ZAP also produces a nice report with all findings, hence the product teams can view and correct the situation.

The OWASP ZAP Automation Framework

ZAP offers several ways of automating and different ways to scan. The currently recommended way is through ZAP Automation Framework. We use a “baseline” scan on a nightly schedule. This scan is perfect for running daily because it is  fast and passive. Passive means that it does not perform any attacks, such as trying out input fields for SQL Injection. Let’s see how we can integrate it into our Gitlab pipeline.

Integrating the OWASP ZAP Automation Framework Baseline scan into your Gitlab pipeline

    1. Setup the Context

      We’ll be using the vulnerable Broken Crystals Application to get things rolling. You can obviously use your own application as well. We’ve set up a user with an email ‘admin@admin.com’ and a password: ‘admin’ so we’ll use that to setup the ZAP context. Double click on the ‘Default Context’  from the context, chose the ‘Authentication’ menu and fill the settings as show below.

      After that chose the ‘Users’ menu and click on the ‘Add’ button to add our user details and fill our ‘admin@admin.com’ user details.

      Integrate OWASP ZAP and Gitlab. Step 2
      ZAP Context add user
    2. Run the First Automation Plan

      Now that we have our context set up, let’s get cracking on the Automation Plan. We won’t be focusing on how to run an automated scan via the GUI, but there’s a great in depth explanation on how to do that here. Just make sure you have the following settings set up for your ‘Spider’ job:

      Integrate OWASP ZAP and Gitlab. Step 3
      ZAP Automation Framework Spider job config
      Integrate OWASP ZAP and Gitlab. Step 4
      ZAP Automation Framework Spider job advanced config

      And for your ‘Spider Ajax’ job:

      Integrate OWASP ZAP and Gitlab. Step 5
      ZAP Automation Framework Ajax Spider job config

      Integrate OWASP ZAP and Gitlab. Step 6
      ZAP Automation Framework Ajax Spider job advanced config
    3. Export the Automation Plan

      After running it successfully, we need to export our automation framework configuration file. This can be done by clicking on the “Save Plan As…” option as shown below, which will save your automation plan as a yaml file.

      Integrate OWASP ZAP and Gitlab. Step 7
      ZAP Automation Framework export plan button

      You can read more about what the produced configuration file contains here.

    4. Testing the Automation Plan with ZAP’s Command Line Tool

      The next step is to pull the ZAP docker image:

      docker pull owasp/zap2docker-stable
      

      and after that all we need to do is go to the directory where we downloaded our automation.yaml configuration file and run the command:

      docker run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable zap.sh -cmd -autorun wrk/borken_crystals.yaml
      

      The result should be something along the lines of:

      Job spider finished
      Job spiderAjax started
      Job spiderAjax found 0 URLs
      Job spiderAjax test of type stats failed: At least 100 URLs found [0 < 100]
      Job spiderAjax finished
      Job passiveScan-wait started
      Job passiveScan-wait finished
      Job report started
      Job report failed to generate report: Cannot create directory '/Users/codific/2022-12-29-ZAP-Report-brokencrystals.com'.
      Job report finished
      Automation plan failures:
      Job report failed to generate report: Cannot create directory '/Users/codific/2022-12-29-ZAP-Report-brokencrystals.com'
    5. Adding OWASP ZAP Automation Plan to GitLab’s CI/CD

      Before we move onto adding the script to the CI/CD, you should replace the occurrences of some configurable strings in your broken_crystals.yaml file:

      env:
      ...
        contexts:
      ...
          users:
            - name: "admin"
              credentials:
                password: "PASSWORD"
                username: "USERNAME"
      

      And replace the report task config, found at the end of the file, with the following:

      ...
      - parameters:
          template: "traditional-html"
          reportDir: "CI_PROJECT_DIR"
          reportFile: "ZAP_REPORT"
          reportTitle: "ZAP Baseline Scan Report"
          reportDescription: ""
          displayReport: false
        risks:
          - "info"
          - "low"
          - "medium"
          - "high"
        confidences:
          - "falsepositive"
          - "low"
          - "medium"
          - "high"
          - "confirmed"
        sections:
          - "instancecount"
          - "alertdetails"
          - "alertcount"
        name: "report"
        type: "report"
      - parameters:
          template: "traditional-md"
          reportDir: "CI_PROJECT_DIR"
          reportFile: "ZAP_ALERT_REPORT"
          reportTitle: "ZAP Scanning Report"
          reportDescription: "This is report that should contain only alerts that should\
            \ fail the pipeline and be revisited by a team member"
          displayReport: false
        risks:
        - "low"
        - "medium"
        - "high"
        confidences:
        - "low"
        - "medium"
        - "high"   
        - "confirmed"
        sections:
        - "alertdetails"
        - "alertcount"
        name: "alert-report"
        type: "report"
      

      This allows us to move the credentials to secure gitlab variables and it also generates an HTML and a Markdown report which we will use to determine whether the scan completed successfully (didn’t find any scary errors). Move the configuration file inside your repository in  a new directory called ‘zap’.

      Next, create a script in your project root directory called zap_automation_baseline_scan.sh with the following content:

      #!/bin/bash
      
      sed -i "s/PASSWORD/$PASSWORD/g" zap/broken_crystals.yaml
      sed -i "s/USERNAME/$USERNAME/g" zap/broken_crystals.yaml
      sed -i "s/ZAP_REPORT/$ZAP_REPORT/g" zap/broken_crystals.yaml
      
      # NOTE:
      # We need this because the variable contains / in it.
      # More info here https://unix.stackexchange.com/questions/255789/is-there-a-way-to-prevent-sed-from-interpreting-the-replacement-string/255869#255869
      ESCAPED_CI_PROJECT_DIR=$(sed -e 's/[&\\/]/\\&/g; s/$/\\/' -e '$s/\\$//' <<<"$CI_PROJECT_DIR")
      sed -i "s/CI_PROJECT_DIR/$ESCAPED_CI_PROJECT_DIR/g" zap/broken_crystals.yaml
      
      sed -i "s/ZAP_ALERT_REPORT/$ZAP_ALERT_REPORT/g" zap/broken_crystals.yaml
      
      /zap/zap.sh -cmd -autorun $CI_PROJECT_DIR/zap/broken_crystals.yaml
      
      returnCode=0
      if grep -q "Instances" alert-report.md; then
        head -n 20 $ZAP_ALERT_REPORT.md
        echo "DAST RESULT: There are some vulnerabilities that ZAP has found (those visible here may not be the only ones). See the detailed report for more information." 
        returnCode=1
      fi
      
      exit $returnCode
      

      After that add the following to your .gitlab-ci.yaml file:

      zap-automation-baseline-scan:
        stage: test
        image:
          name: owasp/zap2docker-stable:latest
          entrypoint: [""]
        variables:
          CI_PROJECT_DIR: path_to_project_dir
          ZAP_ALERT_REPORT: alert-report
          ZAP_REPORT: baselinescan
        script:
          - bash zap_automation_baseline_scan.sh
        only:
          refs:
          - branches
        artifacts:
          when: always
          expire_in: 1 week
          paths:
            - ${CI_PROJECT_DIR}/${ZAP_REPORT}.html
            - ${CI_PROJECT_DIR}/${ZAP_ALERT_REPORT}.md
      

      And voila! Running this gitlab job will cause it to fail if our script has found the ‘Instances’ heading in the Markdown report file generated by the scan. We also have a prettier HTML version of the report which our colleagues could review for more information on what exactly the issues were.

Integrating the OWASP ZAP Full Scan into a GitLab Pipeline

Alongside the “baseline scan”, which we run daily, we also use a “full scan” which is aggressive and slow. That’s why we run it on a weekly basis. The setup is similar.

Create a zap_full_scan.sh script with the following content:

#!/bin/bash
# NOTE:
# We need this because the variable contains / in it.
# More info here https://unix.stackexchange.com/questions/255789/is-there-a-way-to-prevent-sed-from-interpreting-the-replacement-string/255869#255869
ESCAPED_CI_PROJECT_DIR=$(sed -e 's/[&\\/]/\\&/g; s/$/\\/' -e '$s/\\$//' <<<"$CI_PROJECT_DIR")
sed -i "s/CI_PROJECT_DIR/$ESCAPED_CI_PROJECT_DIR/g" zap/zap_config.conf

ESCAPED_TARGET=$(sed -e 's/[&\\/]/\\&/g; s/$/\\/' -e '$s/\\$//' <<<"$TARGET")
sed -i "s/TARGET/$ESCAPED_TARGET/g" zap/context.context

/zap/zap-full-scan.py -d \
                      -t https://brokencrystals.com \
                      -r $CI_PROJECT_DIR/$ZAP_REPORT.html \
                      -w $CI_PROJECT_DIR/$ZAP_ALERT_REPORT.md \
                      -n $CI_PROJECT_DIR/zap/context.context \
                      -U $USERNAME

returnCode=0
if grep -q "Instances" $CI_PROJECT_DIR/$ZAP_ALERT_REPORT.md; then
  head -n 20 $CI_PROJECT_DIR/$ZAP_ALERT_REPORT.md
  echo "DAST RESULT: There are some vulnerabilities that ZAP has found (those visible here may not be the only ones). See the detailed report for more  information."
  returnCode=1
fi

exit $returnCode

And add the following to your .gitlab-ci.yaml file:

zap-full-scan:
  stage: test
  image: 
    name: owasp/zap2docker-stable:latest
    entrypoint: [""]
  variables:
    CI_PROJECT_DIR: path_to_project_dir
    ZAP_ALERT_REPORT: alert_report
    ZAP_REPORT: full_scan
  script:
    - bash zap_full_scan.sh
  only:
    refs:
      - branches
  artifacts:
    when: always
    expire_in: 1 week
    paths:
      - ${CI_PROJECT_DIR}/${ZAP_REPORT}.html
      - ${CI_PROJECT_DIR}/${ZAP_ALERT_REPORT}.md

We don’t recommend using the full scan towards production applications as it may cause corrupted database, stored XSS, etc. At the end of the scan the report that is generated is similar to the one generated by the “baseline” scan, with the only difference that it might have found different vulnerabilites due to the nature of the scan.

Conclusion

In this blog we have presented a partial view of our secure deployment solution. Before we deploy the actual application on the server we run a dynamic application security test using OWASP ZAP. We have presented in details how we have integrated ZAP into our GitLab CI/CD pipeline. Note that you can easily adapt the scripts to integrate it into e.g., GitHub’s CI/CD.

What does the Codific team build with OWASP ZAP and Gitlab?

Codific is a team of security software engineers that leverage privacy by design principles to build secure cloud solutions. We build applications in different verticals such as HR-tech, Ed-Tech and Med-Tech. Secure collaboration and secure sharing are at the core of our solutions.

Videolab is used by top universities, academies and hospitals to put the care in healthcare. Communication skills, empathy and other soft skills are trained by sharing patient interview recordings for feedback.

SARA is used by top HR-Consultants to deliver team assessments, psychometric tests, 360 degree feedback, cultural analysis and other analytical HR tools.

SAMMY Is a Software Assurance Maturity Model management tool. It enables companies to formulate and implement a security assurance program tuned to the risks they are facing. That way other companies can help us build a simple and safe digital future. Obviously our AppSec program and SAMMY itself is built on top of it.

We believe in collaboration and open innovation, we would love to hear about your projects an see how we can contribute in developing secure software and privacy by design architecture. Contact us.

Reach out to us here