Updated: 18 January, 2024
5 September, 2022
Security verification is about validating that a system or application adheres to predefined security requirements or standards. There are many types of security verification activities ranging from penetration testing to vulnerability scans by automated tools. One of the less known types of security verification is the stress testing. Stress testing is part of what Software Assurance Maturity Model (SAMM) calls Misuse/Abuse testing. Misuse/Abuse testing aims to detect unexpected design flaws and implementation bugs by focusing on the so-called negative tests trying to “break” something in the system. Wait, but isn’t stress testing mainly related to testing the scalability of the system? Well, it is! However availability is one of the core security qualities, alongside confidentiality and integrity. Hence, stress testing is an inherent part of security verification testing.
In this blog we present our insights in implementing stress testing at Codific. We also discuss the required stress test tooling and its integration in the CI/CD pipeline. Finally, we provide some tips and tricks that are likely to be the hardest part of creating the stress tests.
Key takeaways
- Start creating stress tests for your web applications and APIs. Stress tests are straightforward and inexpensive.
- Start from pure load testing and move towards more intricate abuse cases. In case you leverage auto-scaling focus solely on abuse cases.
- Integrate your stress tests in your CI/CD pipelines.
- You can run the stress tests either against an identical replica of the production environment or spin-off a stress test environment on the same machine during off-peak periods.
- Leverage JMeter, which is a mature open-source project to set up your stress tests.
Extract from the OWASP SAMM Fundamentals course.
How to Design a Meaningful Stress Test
Designing stress tests is a relatively straightforward task. You should determine the types of stress cases that the system may encounter in real-world scenarios. This can include simulating high user traffic, spikes in user activity, or long-running processes. Then you create workload patterns that simulate these scenarios for the system. This can involve setting up test data that reflects different usage patterns or creating scripts to automate simulated user interactions.
There are (at least) 2 categories of scenarios you can design for your software system.
Positive Stress Test Scenarios
The positive stress tests focus on the legitimate flows for your software system. These are the more scalability-testing type of tests. Consider the following chain of events in the context of an online banking application:
- The user authenticates to the online banking system.
- The user checks his bank account.
- The user creates 50 transfers.
- The user exports the history of the past month to Excel.
Despite of the large number of transfers this is a legitimate flow. Hence, a positive stress test would focus on running many concurrent requests mimicking this legitimate flow.
Negative Stress Test Scenarios
The negative stress tests focus on resource starvation or exhaustion. Consider a slightly different scenario for the banking application:
- The user authenticates to the online banking system.
- The user exports the history of the past month to Excel and does this 50 times.
Assuming that an Excel export is a much more resource intensive operation we consider this a possible attack flow, hence a negative stress test scenario.
Stress Testing APIs
Note that throughout this blog we mainly focus on web applications. API endpoint stress testing is not much different. However API endpoints are typically stateless, hence a concept of a flow is meaningless. Nonetheless based on data patterns you would still have a positive and negative stress test cases. Consider an API endpoint that imports users into the system. A negative stress test could try to abuse this endpoint by importing either a very large number of users or by importing users with unrealistically long names, addresses or emails.
Extract from the OWASP SAMM Fundamentals course.
Which Tools Can I Use to Implement Stress Testing
There are quite some stress testing tools available out there, e.g., ab, siege, Apache jMeter. We strongly believe that Apache JMeter is the best pick given its simplicity and flexibility. JMeter is also very easy to integrate in your CI/CD pipeline. Hence you can easily run your stress tests nightly to make sure new updates to the system didn’t reduce the resilience of your software systems.
The tooling setup is our of scope for this blog, however there are some tips and tricks for JMeter that might save you quite some time. We had to find them out the hard way.
Authentication in Stress Test Scenarios
The easiest way to create and run your stress tests is to hard code your authentication information in JMeter. It is quite obvious you will not run the stress tests against a production copy of your system. However assuming you have multi-factor authentication (MFA) in place, it still remains a challenge on how to log in the MFA users. Here are 2 alternatives when it comes to time-based one-time password (TOTP) based MFA.
- Use a script that generates the MFA token based on a hard-coded secret. Integrating that script in JMeter should be relatively straightforward.
- Adjust your application to auto-fill MFA in non-production environments. The MFA code is actually generated by the web application and sent in the response on the same page where the user has to provide his MFA. We would highly recommend writing unit tests to make sure that this never happens in a production environment though.
We prefer the second alternative as we had to implement this anyway for Apple App Store reviewers who apparently have no idea TOTP MFA is.
Running the Tool
You can never run the stress tester on the same machine where your software system runs. So developers have to keep this in mind when setting up everything necessary or working on improvements. However something we also figured out is that internet connectivity might have some impact on the test results. Hence, during the stress tests do make sure that JMeter runs on a machine that is physically close to the server on which the stress test copy of your system is running.
Stress Test Environment Setup
One of the most essential aspects of stress tests is the environment setup against which you will be stress testing.
Against Which Servers Shall I Run the Stress Tests
As we’ve mentioned previously the whole purpose of stress testing is to make sure your system remains resilient during high / unexpected / abuse loads. Your system runs on the production server though and running the stress tests against the production environment is probably not a good idea. There are many alternative solutions to this problem, but we recommend one of the following two approaches.
Running the stress tests against an exact replica of the server
You can create an identical copy of your production server and run your stress tests against that copy. A copy means same underlying (virtualized) hardware, same functional and security configuration. Unfortunately this approach has one great downside, namely the cost. However if your Disaster Recovery Plans include a hot replica on an alternate site that is a prime candidate for your stress tests in fact.
Running the stress tests against the actual server
The second alternative is running the stress tests against the actual server. Once again, we strongly discourage doing this against the production environment. However you could create a stress test environment on the same (virtualized) server. Assuming that your system has off-peak hours (e.g., during the night) this is a viable alternative that has little impact on the pricing. The main downside is that your tests might cause a temporary downtime to your production environment during the testing moments.
At Codific for most of our stress tests we’ve opted for the second strategy. Here are some further tips and issues to keep in mind when using this second approach:
- Make sure your stress test environment is not publicly available to everyone on the internet.
- Separate completely your stress test data from the production data.
- Modern web servers are great at recovering from high-load events quickly. Hence your stress tests don’t have to take hours. In case that is necessary than you cannot run them against the production server.
- Your stress tests might completely lock up your server. Assuming the off-peak hours are during the night you might only figure this out in the morning.
Web and Database Server Configuration Settings
Aside from your software architecture, design and code efficiency, there are hundreds of variables that contribute to the scalability and the resilience of your software system. There are some important web and database server configuration settings though that could render your stress tests unsuccessful despite having an optimized architecture. Here are the most essential things to keep in mind:
- Make sure to know your web server’s concurrent connections limit. If this number is reached your web server will immediately return a 503 HTTP code (server overloaded). This is actually a very desirable property of your web server as having no limits in terms of accepting connections would completely lock up your server. You would rather have some percentage of users getting a 503 code rather than a server that simply doesn’t react anymore to any of its users.
- Keep your timeouts as low as possible. In case of an overload it is a good idea that your requests would automatically fail after a certain amount of time hereby freeing up resources (hence, making sure that your server remains operational).
- Make sure to know the maximum number of simultaneous connections for your database server.
- Containers and container orchestration engines might also add up to the configuration complexities.
What Are the Meaningful Metrics for My Stress Tests
“You don’t manage what you can’t measure”. Hence you must measure the stress test results and introduce KPIs to make sure your tests are in fact in line with your resilience strategy. You can be very fancy about it, but here are a number of pragmatic metrics and KPIs we are measuring.
- Error rate. KPI: maximum 1% of the requests may fail.
- Mean response time. KPI: 5 seconds.
- Maximum response time. KPI: 20 seconds.
Obviously you can create different KPIs for different endpoints / sections of your software system.
Adding Stress Testing to my DevSecOps Build Pipeline
Updates to your software system might have an impact on its resilience. To tackle this issue, first of all your should periodically revisit the design of your stress tests to integrate new flows. Furthermore, you should run your stress tests on a regular basis. Fortunately, this is straightforward as you can easily add your stress test to the CI/CD pipeline and run it on a nightly schedule. Here is the configuration file for Gitlab CI/CD that runs the stress test. You can create the stress test recipe using JMeter’s UI and save it in a .jmx file. All you need to do then is to pass this file to JMeter’s container.
After the stress tests we have added a custom bash script (jmeter stress test script) that will process the stress test results and fail the CI/CD pipeline if necessary.
Summary
In this blog we have presented how Codific implements maturity level 3 of for SAMM’s Misuse/Abuse testing practice.
What do we build with SAMM and SAMMY?
Codific is a team of security software engineers that leverage privacy by design principles to build secure cloud solutions. We build applications in different verticals such as HR-tech, Ed-Tech and Med-Tech. Secure collaboration and secure sharing are at the core of our solutions.
Videolab is used by top universities, academies and hospitals to put the care in healthcare. Communication skills, empathy and other soft skills are trained by sharing patient interviews recordings for feedback.
Top HR-Consultants leverage SARA to deliver team assessments, psychometric tests, 360 degree feedback, cultural analysis and other analytical HR tools.
SAMMY Is a Software Assurance Maturity Model management tool. It enables companies formulate and implement a security assurance program tuned to the risks they are facing. That way other companies can help us build a simple and secure digital future. Obviously our AppSec program and SAMMY itself is built on top of it.
We believe in collaboration and open innovation, we would love to hear about your projects an see how we can contribute in developing secure software and privacy by design architecture. Contact us.
