Trends in Load and Performance Testing

Everyone realizes the value of load and performance testing. What most people find difficult to understand is why it is so expensive, why it takes so long and why it is such a black art. Exciting new trends in this space promise a way out of the malaise; more on that below, but first a little elaboration on the limits of load and performance testing.

1. We want the system under test to be as close to the production setup as possible.  The more different it is, the more factors you have to correct for, and the less confidence you will have in the results.  A system sized the same as production (and populated with a similar amount of data) is often a budget-buster, especially because it is used for a small part of the system life cycle and is idle for the remainder of the time.
2. If a system sized similarly to production is not used, or not populated similar to production, interpreting the results by extrapolating them from measurements is itself subject to a degree of guesswork. If you are using the results to convey bad news, you have some convincing to do. If you are using the results to convey good news, you still have some convincing to do.
3. The stress testing tools are expensive because they are designed to cover a range of technologies, from windows applications to batch scripts to web applications.
4. Since  scripts need to be written in a specialized language, writing scripts for stress testing is a specialized skill.  Writing such scripts before the application is available is even more of a challenge.
5. It is common practice to use scripts running within the company’s infrastructure to test web applications.  This makes testing more repeatable but it ignores the vissisitudes of connecting through the internet to get to the company’s servers.

There are a few encouraging trends on the horizon.  

Performance and Scalability Survey results

The results of the survey are available at the Quantitecture Web Site.

Comments on this blog are closed but you may contact us through the company web site.

Thank you.

The Accuracy of Performance Data

To what extent must measurements or modeling results hew to the “real world”? We present a continuum with 4 different options, each with their own pros and cons; and end with a recommendation.

Intersection of Programs and Data

What makes Performance Engineering so interesting and also frought with danger?  It’s the sheer complexity and the continued need to take all factors into account.

Professor Spielman of Yale was recently awarded the 2008 Gödel Award.  The award was for his 2004 paper (I read a 2008 update) on the performance of the Simplex algorithm.  The algorithm has been around since 1947 and most people who went through an Engineering education in the last 50 years learned it.  The theoretical analysis suggests that in the worst case, it’s pretty bad but in practice it seems to work pretty well.  The reason it works so well, it turns out, is that the typical data it gets subjected to makes it perform reasonably well.  Alternative algorithms which have better worst-case performance don’t perform as well in practice.

The lesson from this post, and the paper, is this: You can’t study the performance of systems in a vacuum — you have to make assumptions about the data. Many a performance analysis has been invalidated by the fact that it was done with the wrong data.  We have 60 years of history to prove it!

System Performance Estimation during Requirement Analysis

In a previous blog entry, I have advocated for the urgency of getting an early view into system performance even if the conclusions are approximate.

In this entry, I want to describe a couple of successful techniques: Dimensional Analysis and Scaling. The two techniques are related but complementary. Scaling is the simpler of the techniques but Dimensional Analysis can be extremely powerful. They have been applied for analysis of physical systems in many branches of science. A particularly nice tutorial is available in this Applied Mathematics textbook. You don’t have to read the whole book — Chapter 1 is available here and that is all you need. The treatment of Dimensional Analysis still a bit dense but scaling is sufficient in many instances for the types of estimations we typically require.