Introduction to CI/CD Performance Testing: Common Problems and Key Solutions

Trang Bui - Apr 6, 2018

Continuous integration/continuous delivery (CI/CD) is being rapidly adopted by many organizations to improve their time to market. According to the World Quality Report 2017-2018, 88 percent of companies are using or experimenting with DevOps principles. Since the initial stage for DevOps is the CI/CD pipeline, it is critical to have testing woven into every build cycle. The concept of shifting left while integrating performance testing in the continuous delivery model is an important goal for testers.

Here, we’ll identify some typical challenges development and QA teams encounter when setting up performance testing within a CI/CD pipeline, as well as the solutions that can be used to resolve these issues.

Today’s Testing Environment

Applications are becoming increasingly complex, with a lot of dependencies and integrations with third-party platforms. This makes building a production-like environment very involved – and expensive.

A recommended goal for testing in CI/CD is to verify the performance trends of the application following changes to a simpler configuration. This allows testers to evaluate different versions compared to a baseline result on an identical configuration. This approach also means testers can use simplified containers as opposed to building a full, realistic environment.

Time Required for Testing

Executing performance testing is a time-consuming task due to the high volume of data prepared and long execution for resource decay. With this in mind, data should be prepared in advance and reused wherever possible.

Shorter duration test cases for new functional changes can also be helpful in reducing testing time, and should be executed in CI/CD pipelines first. Longer duration end-to-end performance tests and load/stress tests should be done in a concurrent, yet separate cycle of testing.

In addition, testing teams should also look to adopt practices including:

• Early involvement of testers: Performance/load testers should be involved in the project as early as possible, allowing them to work with other team members to apply inputs of their testing environments and strategy. Those tasks should be defined according to stories planned within each sprint.
• Expanded acceptance criteria for user stories: There should also be performance acceptance criteria for each story, so testers can create and execute performance tests based on those specifications.
• Short feedback loops: The goal of CI/CD pipelines is to encourage short feedback loops, so this approach should be taken with performance tests for new components as well. Small load testing to measure performance or to monitor a trend compared with historical runs is necessary. In addition, it can be helpful to use parallel test suites for even faster feedback.
• Release-ready testing: End-to-end scenarios with high volumes of data and transaction loads are still needed during the production phase to ensure a quality product. These types of time-consuming tests should be executed nightly or after sprints are completed, independent of CI/CD pipelines.

The current application testing environment is complex, but using CI/CD can help streamline testing processes.

Recommended Tools for Performance Testing

It’s important that testing teams have the right tools in place to support integrated performance testing. Some tools we’ve found most successful include:

• Apache Jmeter
• Smartmeter.io
• Webload

Check out this resource from Software Testing Help for more reviews of available tools.

In addition, there are also certain plugins that can be used with Jenkins to enable performance result analysis, such as:

• Plot
• Performance Publisher
• Blazemeter
• Taurus

Stackify also recommends these DevOps tools.

In the current competitive world of complex, integrated applications and websites, performance must be evaluated and monitored. With the basic fundamental approach to testing earlier and testing often, it’s possible to avoid the pitfalls many users face when adopting CI/CD for evaluation application performance.