Performance, Scalability and Availability in IT

Trends in Load and Performance Testing

Posted by J on February 26, 2009

Everyone realizes the value of load and performance testing. What most people find difficult to understand is why it is so expensive, why it takes so long and why it is such a black art. Exciting new trends in this space promise a way out of the malaise; more on that below, but first a little elaboration on the limits of load and performance testing.

  1. We want the system under test to be as close to the production setup as possible.  The more different it is, the more factors you have to correct for, and the less confidence you will have in the results.  A system sized the same as production (and populated with a similar amount of data) is often a budget-buster, especially because it is used for a small part of the system life cycle and is idle for the remainder of the time.
  2. If a system sized similarly to production is not used, or not populated similar to production, interpreting the results by extrapolating them from measurements is itself subject to a degree of guesswork. If you are using the results to convey bad news, you have some convincing to do. If you are using the results to convey good news, you still have some convincing to do.
  3. The stress testing tools are expensive because they are designed to cover a range of technologies, from windows applications to batch scripts to web applications.
  4. Since  scripts need to be written in a specialized language, writing scripts for stress testing is a specialized skill.  Writing such scripts before the application is available is even more of a challenge.
  5. It is common practice to use scripts running within the company’s infrastructure to test web applications.  This makes testing more repeatable but it ignores the vissisitudes of connecting through the internet to get to the company’s servers.

There are a few encouraging trends on the horizon.  The new trends have their own limitations but at least there is hope.

  • Amazon’s new offering, Elastic Compute Cloud (EC2), lets you obtain any number of servers at a moment’s notice and you pay for what you use.  You create a machine image and upload it to Amazon (I know this is not as easy as it sounds). If you need 5 servers running that image, it’s a bunch of web services and the servers are yours in 10 minutes.  You run your tests and give up the machines so you don’t pay for them.  It is finally cost effective to obtain identically-sized servers. Rightscale will help you manage your usage of EC2.
  • LoadStorm and BrowserMob are two recent startups who provide the capability to create and execute scripts to stress the servers.  Both do their work external to the company’s infrastructure.  I have not had the opportunity to use their products. 

If you know of others, please let us know.


One Response to “Trends in Load and Performance Testing”

  1. Well done! Your thoughts are insightful. Thank you for sharing.

    We have found that configuring servers in the cloud is a bit more difficult than we anticipated. It has taken us several months to figure out the subtleties of dynamically allocating load generators (new instances of EC2). We are hoping that developers and testers will prefer to use our service because it abstracts the load generator layer from them and saves them quite a bit of time. Also, we do not have a scripting language to learn. LoadStorm may not have the most features, but it is the easiest way to quickly ramp up thousands of virtual users to load test a web application.

    Thanks again for sharing. Good luck.


Sorry, the comment form is closed at this time.

%d bloggers like this: