Integrate Cloud Load Testing into Your Release Pipeline with VSTS

//Integrate Cloud Load Testing into Your Release Pipeline with VSTS

How fast does your website respond under typical user load? What would performance look like with a sudden 75% increase in traffic? Or 300%? Is the code you’re about to deploy going to improve or degrade performance of the website? What kind of difference will scaling out or up make to your website’s performance? If you don’t know the answer to these questions, you could be missing out on a key opportunity to improve customer satisfaction. So how do we monitor performance and test for scalability?

Enter Load Testing

Load testing is a consideration often overlooked until users report issues with performance during peak times. With no objective measure of performance, Ops might simply begin throwing additional resources at the problem. The first step is typically scaling services up to plans with more processors, RAM, or faster storage. Where that fails, services may be scaled out, adding additional instances of a service to handle the increased capacity. Services continue to be scaled up and out until the complaints stop coming in, again with no real objective measurement of performance. Load testing takes away this uncertainty by allowing you to stress your resources with a specified number of users and quantify request performance.

Using Cloud Load Tests in VSTS, teams can simulate their typical user load and record a baseline of performance. Performance numbers go from subjective measures of “seems fast” and “responsive” to objective measurements of how many milliseconds on average web requests take. Cloud Load Test results also provide an aggregated look at how quick specific calls are, allowing teams to identify the slowest calls in their application which may be ripe for optimization.

Capacity Planning

With a baseline established, additional load tests can be executed to simulate varying levels of user demand to see how many users the current architecture can support before response time begins to suffer. With Azure’s ability to quickly scale up and out for small periods of time, you can easily and inexpensively test what performance would look like bumping up from an S1 to an S3 or from 1 instance to 2 instances. When the load tests are complete, services can be scaled back to their original parameters to minimize cost.

Load tests will not only give you more a much better idea of how many users you can support now, but what your best options are if load does increase dramatically in the future. This is especially useful for planned increases in traffic due to new ad campaigns that are expected to drive a large amount of traffic to your site as soon as it drops. Knowing exactly how much you need to scale your site beforehand means less chance of users having a poor experience on your site as they rush to check it out.

Load Testing as part of your Release Pipeline

All of what we’ve covered so far is standard capacity planning best practices. However, load testing has typically been a very complicated process to setup, execute, and analyze. With Cloud Load Testing, the initial cost of setting up load tests has been greatly reduced now that recording tests is so simple and there is no hardware/software to install and maintain for load test agents. Cost is also trivial using VSTS, with a load test often costing less than $1 per run. Now that load testing so inexpensive and easy to execute, it makes sense to execute your load tests regularly during your release cycle.

Executing load tests once per iteration or once every few days allows you to quickly identify performance issues that might be related to new features and hotfixes deployed. The more often load tests are run, the easier it is to narrow down the list of commits to your code that are likely culprits. Load tests also allow you to quantify whether improvements that were made to increase performance delivered the expected results by comparing to previous load tests.

For those looking to take this a step further, VSTS build and release tasks already exist to automate execution of load tests against your site. All releases can be automatically load tested at any point during your release pipeline. Code that slows your site down can be caught before it’s deployed to Prod and affecting customers.

VSTS provides a great interface to compare load test results to each other and, when coupled with Application Insights, can let developers see exactly where a problem occurs before customers will ever notice.


A little about us: Nebbia Technology is a software company that specializes in Azure-based solutions and DevOps best practices. As a Microsoft Gold partner in both Cloud and Application Development, with every single developer and consultant Microsoft-certified, we are the DevOps experts to help your team get the most out of your tools and software development process.

By |2017-05-04T05:17:28+00:00May 3rd, 2017|

About the Author:

Mikey is a Software Architect at Nebbia Technology with over 18 years of experience with Microsoft development and server technologies. He has worked with the .NET Framework stack and TFS/VSTS DevOps and Agile practices for the past 12 years and has a particular soft spot for obscure database technologies.