Basic JMeter Load Testing of Web Sites and Rest APIs

This is an intro to load testing with Apache JMeter, an Open Source load testing tool.

As a developer, QA or Operations engineer, it’s important to be familiar with what load testing tools can do, and to know how to configure a few actual tools.

I usually reach for a load testing tool in the following scenarios, which appear similar, but really are very different. You can divide stress and QA testing into 4 categories:

1. I want to know what will happen when 100 requests are sent to a single or handful of endpoints in a brief time interval, usually one second (connection and configuration testing)
2. I want to know what will happen under sustained load of 50 requests/second to a single or handful of endpoints, for typically 5 minutes (performance testing)
3. I want to know that all of an application’s pages respond successfully, typically using 1 thread (application testing)
4. I want to know how many synthetic users that application’s pages respond successfully under load, typically 20 – 100 threads. (application load testing)

Note that I don’t consider a simulated load to be meaningful for predicting human loads.

For example, on one intranet project, 70,000 users were happy with a phone book web app that only load tested to 20 simultaneous users. The test was useful in indicating that nothing was misconfigured, but not useful in predicting how many people could actually use it.

Load tests just tell me:

1. if something is misconfigured or broken. If I get less than once response per second, or the server stops listening for a period of seconds, then we know there is a problem to investigate.
2. numbers that I can compare to other runs over time.

Why JMeter?

JMeter is:

• convenient (after the first time you learn it)
• popular in the Java community, so worth being familiar with
• Open Source (free)
• extensible

The disadvantages of JMeter are that:

• it has a complex UI
• Java GC pauses can affect results on longer test runs. You can mitigate that by setting up your tests using the UI, then run the tests from the command line as recommended.

JMeter Installation

1. check your Java version for 1.7 or 1.8 with java -version
2. download and install JMeter
3. read the JMeter Getting Started guide
4. read the first 7 pages of the Basic Scripting with JMeter tutorial by Simon Knight
5. setup an initial test using the JMeter UI. You must include a Response Assertion for a credible test. Then save to a jmx file as bin/Mysite.com.jmx (it’s an XML file with your settings.)

Automating

Make 4 copies of your jmx file with:

cp -p Mysite.com.jmx Mysite1.jmx
cp -p Mysite.com.jmx Mysite2.jmx
cp -p Mysite.com.jmx Mysite2.jmx
cp -p Mysite.com.jmx Mysite3.jmx

Edit each jmx file to customize the properties according to the 4 strategies I listed above:

LoopController.loops
ThreadGroup.num_threads

(If you want to invest some time, you can parameterize those as documented in the JMeter FAQ.)

Create the following bash script make_mysite1.sh so that you can run your test from the command line:

#!/bin/bash

# Program: test_mysite1.sh

rm -f mysite1.log

./jmeter -n -t Mysite1.jmx -l mysite1.samples.log -j mysite1.log

grep "Thread Group" mysite1.samples.log | grep -v [O]K

Running Tests

1. Ensure you’re authorized before running any load test against a server you don’t own.
2. bash test_mysite1.sh
3. Analyze the response codes and timings. Test samples will be in mysite1.samples.log, and reports in mysite1.log.

When to Run

Every time a change is made to your environment, you should re-run the load tests. So include it as part of your release process checklist.

Distributed Testing

After you’re familiar with load testing using a single client with JMeter, you can learn about using multiple load test clients.

Bonus – “Soak Testing”

In the telco industry, historically new systems have undergone “soak testing.” This is operating test systems under a realistic load for one month or more to “provide a measure of a system’s stability over an extended period of time.”

JMeter Too Difficult?

Microsoft’s discontinued Web Application Stress Tool (WAST) aka “Homer” is an incredibly easy-to-use, distributed and powerful Windows graphical tool – “its ease of use means it actually gets used.” If you want to do load testing from Windows client machines, you can download it from here.

WAST is so good that it has fans, which can’t be said for any other load test tool. It was replaced by Visual Studio Team System’s (VSTS) Test Manager.

JMeter: FAQ, Best Practices
SO: Load test with varying number of threads in JMeter