Choosing the right Performance Test Tools – An In-depth Analysis

Introduction

So, you want to get your software tested for its performance and you are confused or spoilt for choice in deciding which tool you should use. Unlike earlier times, now there is an abundance of performance testing tools in the market. So much so that the basic web load testing is even commoditized to a great extent.

This blog will help you get a perspective of the performance testing tool landscape. It might be helpful to clear the confusion surrounding the evaluation criteria. We are not going to discuss about the feature of each and every load testing tool out there in the market. Here, we try to pick the five most useful and feature rich tools address business needs.

Let us divide the IT landscape into five different verticals. Your applications can be a mix of any number of towers here or can be a single tower or can be couple. What we are trying to do here is see where most of your IT falls into the technology ecosystem. Web here refers to everything that goes in a browser and covers J2EE, .net, PHP, HTML5 and all those technologies that communicate using http protocol on desktop computers and tablets. ERP consists of package solutions that are a combination of browser based as well as standalone products. Mobile consists of apps for a variety of operating systems – both native and hybrid. Streaming media is where you fall in if you are into video business. Citrix, though doesn’t seem to fit in the way we are dividing things here, made it to the list just because of the complexity and variety of reasons it brings into the scheme of performance testing.

If you are an owner of web applications, you likely to get confused when picking up the tool. Testing performance for web applications was never easier than what it is today. In fact, Performance testing of web applications is a commodity now a days. You can visit blazemeter.com and straight away download a chrome add-in that they call as “FollowMe” and start navigating your website or pages of interest. What follows is a swarm of virtual threads emulating your action from different corners of the world. What more? You can have all your servers monitored for deep diagnostics by opting for a NewRelic add-in feature that blazemeter provides by tying with NewRelic.

But that’s just plain web page testing. If you have much more complex requirement like unique logins, testing multiple paths at the same time, dynamic page loading, flex and flash presentations, streaming media, then, read on.

Apache JMeter

Apache JMeter is an open source load/unit/ functional testing tool written in Java. What does that mean? It’s free to use, can run on Linux and windows and yes, Mac and Solaris as well. It supports Record/Replay feature. It supports only http/JDBC protocols, it does support a lot of others as well. Let’s look at the pros and cons.

Pros:

• It is free.
• It runs on any platform.
• It supports record/replay feature.
• It supports http/s, JDBC, FTP, LDAP, JMS protocols.
• It has a friendly GUI.
• It is easy to install.
• It has pretty good analysis plugins
• Source code is open for modification according to your individual needs.
• Some level of support for Flex/Flash with custom plugins.

Cons:

• If you are stuck with any issue, your only point of help would be those good samaritans answering your questions on forums.
• The tool lacks built-in rules for detecting dynamic values and hence it consumes more time for scripting. Most of the scripting time is spent on making the script “runnable” with multiple users. That involves handling dynamic data. Having built-in rules that can support extraction of these values will be a huge plus.
• It doesn’t support Flash/Flex in all cases.
• Unlike commercial testing tools like LoadRunner from HP, you cannot have control on the quality of test in Jmeter.
• The error logs are not “tester friendly”. There’s a learning curve associated with reading the error logs from Jmeter.
• If you are an enterprise customer with ERP and virtualization technologies, and want to adopt a single tool that fits most of your business needs, probably JMeter will lag a bit here depending on your team size.
• It lacks the drilled down features which most of the commercial tools give you, albeit at an extra cost.

HP LoadRunner

Reports claim that as much as 54% of market share in performance testing tool market is owned by HP Load Runner. And there’s a reason for it. This tool supports a large set of protocols which no vendor has been able to replicate so far.

Being the pioneers in the load testing tool industry, LoadRunner boasts of abundant talent available in the market, right from veteran Gurus to newbies. If your enterprise landscape is a combination of ERP, Mobile applications, web applications and exotic protocols, and if you require diagnosis of performance data and competent workforce , LoadRunner stands at the top of the list. The downside though, is the cost associated with it. Having said that, nothing that is great will ever come free.

Pros:

• A record/replay tool.
• Supports virtually any protocol. For anything that is not supported, a fall back option is available in the name of WinSock. If there’s a protocol that is not officially supported and still communicates using sockets, you can still test your applications using LoadRunner.
• Great GUI. The learning curve is slightly less when it comes to learning this tool. The GUI is so friendly and it offers great flexibility in changing run time settings, data files etc.
• Separate analysis package. The test results can be analysed in a professional and yet easily usable, user friendly analysis component. The analysis package offers great drill down features and correlate graphs that makes it easy to analyze the performance bottlenecks.
• Built in feature to monitor hardware resources of the machines under test. LoadRunner comes with 2000 point SiteScope license which can monitor hardware resources of all the machines involved during the test and provide a correlation of these metrics with the test metrics. The best thing is, the sitescope is free.
• Built in rules that support auto correlation. You can extend these rules with custom rulesas well.
• Data format extensions make it easier to record, enhance and maintain test scripts that are designed using Web 3.0 technologies like GWT, ExtJS, JQuery etc.
• Offers multiple options in designing load test scenarios. For example, you can run a load test with a target of achieving certain throughput during the test.
• Provides support for multiple programming languages – C and Java.
• Exhaustive documentation, excellent learning paths and very good support from HP.
• Integration with diagnostic software.
• If you are trying to make your enterprise a “vendor” shop that caters to all your automation, performance and application monitoring needs, HP offers you really great flexibility when it comes to licensing.

Cons:

• Doesn’t run on UNIX. You can have load generators installed on Unix and have them run but the controller cannot be installed on Unix.
• Learning curve. Loadrunner requires testers to have at least fair amount of programming skill to write scripts.
• Lack of proper support when it comes to testing streaming media applications. Though protocol level recording is available, metrics like buffer fill, lag ratio are not available with LoadRunner.

Silk Performer

SilkPerfomer has almost all features that LoadRunner has. The load tests in SilkPerfomer force a project approach. It keeps all your load test artefacts in a single location and thus provides greater control.

Pros:

• Tighter integration between test artifacts. This makes it easier to maintain all load test related artefacts in a single place.
• TrueLog™ explorer offered by SilkPerformer is unrivalled in its peer space. Visual root cause analysis provided by TrueLog™ makes it easier to pinpoint bottlenecks.
• Superior in terms of emulating real time browser behaviour.
• Correlations in scripting are much simpler when you compare with other tools.

Cons:

• Reporting and Analysis modules are not as matured as that of LoadRunner.
• Uses proprietary scripting language for scripting tests. As a result, it’s learning curve is a bit on higher side.
• Parameterization is not as easy when compared with LoadRunner.
• Scripts, though less memory hungry, are not intuitive when compared to LoadRunner.
• Designed only for Windows.
• Poor integration with deep diagnostics tools for end to end performance monitoring.

NeoLoad

If you have people who have worked on Jmeter, they can become NeoLoad experts in little time. The interface is quite similar and is arguably the easiest of tools to use. Testing mobile applications and streaming media content using NeoLoad is lot intuitive and easier than most other tools. Built in counters like buffer lag and lag ratio for streaming media content makes it easier to test media applications. The downside is the limited protocol support that NeoLoad offers.

Pros:

• Ease of use.
• Excellent licensing options.
• Highly scalable for simple web applications.
• Out of the box support from media content.
• Runs on windows, Linux and Solaris.
• No scripting is required.
• Single GUI for all actions.
• Project based execution similar to silk performer. All your load test artefacts are couple together.

Cons:

• • Limited protocol support.
  • The lack of scripting makes it difficult to maintain complex scripts.
  • Very basic analysis and reporting engines. There is no “raw data” concept and hence the tool has little.

flexibility in terms of obtaining deep statistics from any load test.

• Limited control on virtual users once the test starts.
• Little room for reusability because of a lack of programming support.
• Handling of redirections is error prone “sometimes”.
• Not a best fit for large enterprises that operate with ERP, Citrix.

IBM Rational Performance Tester

If you are an IBM shop, IBM RPT will have an edge with it’s out of the box support for WAS and IBM JVM monitoring, albeit, at an extra cost. It fares equally on Web 2.0 and ERP when compared with SilkPerformer or LoadRunner. The workforce availability is one concern that you need to look into. Having said that, it has the same learning curve as that of any tool and you will need the same time to train a new set of people on this. The visual breakdown that RPT offers for every web transactions helps in finding the high level root cause quickly. But, if you use LR 12.0, this feature is available.

Pros:

• Ease of use. There is no concept of correlation here.
• Gels well when it comes to testing applications built on IBM products.
• J2EE diagnostics add-in that comes with RPT, albeit at an extra cost is really helpful in analysing code level problems quickly.
• Impressive analysis and reporting engines.
• Excellent support.

Cons:

• Limited protocol support.
• Very limited support for mobile application performance testing.
• Very limited support for Streaming media performance testing.
• No support for .net diagnostics.
• Connection handling is not really transparent to the user.
• Cluttered scenario execution. There is no clarity on what is happening during a load test.

Conclusion

There is no ‘one size fits all’ approach when it comes to selecting a performance testing tool. Our attempt is to identify the pros and cons of each tool and where it does best and where it doesn’t. There are situations where we worked with large enterprises in setting up multiple tools in their performance labs. At the same time there are enterprise clients whose necessities were completely met with one single tool.

When selecting a tool, look at the protocol support first. Then, look at the workforce availability. Third criteria will be the ease of use. Do not look at the cost if one of these three is not aligning with your needs.

These inputs are intended to serve as a reference to understand the right approach to  strengthen performance testing initiatives.