Performance Testing Applications In The Cloud Era

One of Calvin and Hobbes cartoons features Calvin’s family driving over a bridge and Calvin asking his Dad how they decide on the bridge’s load limit. His Dad tells him they drive bigger and bigger trucks over the bridge until the bridge collapses. They then rebuild the bridge and the weight of the last truck to make it over safely is the weight limit.

Source: https://imgur.com/gallery/q1vI70y

Of course, that’s not how bridges are tested in the real world (though it would be fun to watch), but it is pretty much how software has traditionally been tested for load-bearing and performance. You simply drive traffic at your app and measure response times until it jams up and stops responding or comes crashing down. But cloud computing has changed the paradigm in several ways, according to software testing company A1QA.

How the Cloud Has Changed Performance Testing

For one thing, it is now much easier to scale up. In the past, performance testing was usually done on a specific hardware rig and resolving performance problems often meant making investment decisions to purchase ever bigger servers.

However, in today’s world of cloud computing, the hardware is no longer as strict a limiting parameter as it was. Auto-scaling means that you can easily scale up and pay for just the resources that you use.

Because you are no longer wedded to one specific set of hardware, you should ask yourself 3 questions before starting performance testing:

Do I even need to performance test?
If yes, how do I do that for the cloud?
What do I focus on?

Let’s try to answer these questions.

Do I need to test performance?

It depends to an extent on the type of software and whether you are migrating to the cloud or rolling out a cloud-ready implementation.

If you are doing a “lift-and-shift” of an existing application, then you can take some confidence from previous performance testing and existing information about its performance, and the existing hardware should give you a good idea of your target infrastructure scale. In this case, you could consider restricting your performance testing to validation that it meets existing performance benchmarks on your new cloud setup before going live.

After an app has been lifted to the cloud like this, it is possible to tweak the app itself or the hosting solution (a process commonly referred to as re-platforming) to take advantage of cloud-specific features to optimize performance, or to reduce costs without adversely affecting performance.

If you are testing a cloud-native application, then you should consider performing the usual types of performance tests in a typical cloud environment before going live. In fact, it probably goes without saying that before you even start to consider testing performance, you should have been designing and developing to optimize performance!

Let’s look at the various types of performance tests to consider and the questions that they should enable you to answer ranked in a tentative order of importance.

Load Test

How does the system perform under expected normal load? You should be interested in measuring the latency, throughput, and capacity of your system. You should be looking to calculate your capacity per resource, which ties in with scalability testing

Scalability Test

How does capacity increase as resources increase? Is it a straight-line relationship? Hopefully yes, as it makes your cost calculations easier. What if initial loads are less than expected, can you scale back easily so you’re not paying for unused capacity?

Stress Test

How does your system behave under heavier than average load? What if it goes viral and volume increases to ten or 100 times normal? Does it crash under the weight of ever-fuller shopping baskets?

Availability & Resilience Test

If the system does crash, how quickly can it be brought back online? Can downtime be avoided in various scenarios? How about when performing maintenance?

Configuration Test

As mentioned above, the configuration should be tweaked continuously, also known as re-platforming, to strive toward optimal performance within your cost limits.

Browser Performance Test

Although there seems to be ever more homogenization between the big 3 browsers, if your app has serious performance problems on any one of them, if for example your landing page takes an extra second or two to load, this could lead to a significant number of lost opportunities, as people simply won’t hang around.

Which brings us to the second point.

How to do performance testing in the cloud?

If it is a private cloud solution then you have access to the underlying hardware and you can see all the details of system response times, bottlenecks, etc.

If, on the other hand, you are looking at a public cloud solution, then another company is hosting your solution and you no longer have access to the servers. While this may not matter for something like a browser performance test, for your stress tests and load tests you will have to consider cloud performance monitoring tools.

The good news is cloud providers also provide performance monitoring tools that allow you to set alarms, visualize metrics, create dashboards and logs, etc. The bad news is they cost money. So this is another element to your cost equation to be considered when doing your calculations. Note there is a trade-off here – the more detail and information you have to inform your decision-making, the more it will cost you. This topic probably deserves a whole future article itself.

What do I focus on?

As to the third question above, we would suggest the simple answer is cost. Whereas in the pre-cloud era, the limiting constraint was the available hardware, the cloud presents the opportunity to significantly reduce your costs and yet maintain performance, so cost rather than hardware becomes the limiting constraint.

Therefore setting a maximum benchmark cost for your hosting solution (plus monitoring and related costs) and testing to optimize performance within this limit gives your performance testing more meaning.

On a final note

Performance testing in the cloud may not be as much as fun as watching increasingly bigger trucks driving over a bridge, but if it increases your bottom line, you can take a truckload of satisfaction in a job well done.

Performance testing applications in the cloud era

How the Cloud Has Changed Performance Testing