Load / Performance tests (HTTP and Web)

Question

I'm looking for a load test tool, the main features that I need is:

1. Distribution (very important) - I need to be able to run something like 20,000 - 30,000 request per seconds (and more), so one machine is not enough. It should be able to communicate on Amazon EC2 (jMeter for example, can not do it)
2. Collecting data - It doesn't matter if it can produce graphs or complex data or not, but I should be able to know what was the Throughput of the client and the server, how many errors occured etc. Since it will run on multiple computers, the data should be collected by the tool itself (again jMeter does it, but it has a lot of problems), I'm trying to avoid from fetching data from different machines and merging it "manually".
3. Graphs or more complex data are nice to have (for example: http://code.google.com/p/multi-mechanize/), but if the tool doesn't provide it, I should be able to get this data from the logs of the tests.

I didn't find good reviews on load testing tools (and that's why I'm asking here), and currently the main tool that I'm want to check is Grinder, if you've worked with good tools please share :)

I've worked with jMeter, and I've decided to look for a better tool. jMeter is old, it works with old protocols (So I can't work with it distributed on EC2), it slow and hard to work with, and its graphs make it very slow.

BTW, It doesn't have to be free / open source, if it costs up to tens or hundreds of dollars its OK.

Thanks.

Answer 1

Locust is a great load testing tool and it meets all of the OP's requirements.

As far as meeting the OP's requirements, some Locust features:

• can run in master/slave mode for distributed load generation
• gives you access to raw and summarized result data in simple/parseable formats
• integrates nicely with graphite/grafana for your visualization needs

Locust is open source and written in Python... virtual users are also written in Python code. The power and ease of using Python to develop complex workloads is really nice (compare that to JMeter's clunky declarative XML style).

Locust development is hosted on Github and has several active committers.

---

on the Locust website, the tagline is:

"Define user behaviour with Python code, and swarm your system with millions of simultaneous users."

---

screenshot of optional web UI:

Screenshot of Locust Web UI

---

Note: I've worked with all the other tools recommended in other answers, and Locust is far superior

Answer 2

Visual Studio's Web Performance Tests and Load Tests sound like a good fit here. If you get a license for Visual Studio Ultimate and then a license for Visual Studio Controller/Agents the controllers and agents handle the distribution of load. It is all very well documented and pretty easy to get set up. It has some default reporting that may meet your needs, but also has the ability to export to excel where you can create any custom reports or graphs using Pivot Tables (or external tools like PowerPivot).

Here is the quick reference guide: http://vsptqrg.codeplex.com/ and more info is available all over on the web.

Answer 3

The question states that it is not possible to use JMeter on Amazon ec2 [point 1.]. This is not true, it is possible, but it is made difficult because JMeter uses RMI to communicate and you have to play with tunnels and changing ports to make that work.

The other JMeter issue mentioned [point 2.] is that it has a 'lot of problems' collating data. I suspect this refers to a potential bottleneck where all the data from multiple clients is being written to one location. If so, then his issue is typically addressed by using 'batched' or 'statistical' mode in the properties, plus the latest JMeter version, 2.6, has advances in this area.

However, I wrote a script that gets around both these problems and makes running JMeter on ec2 considerably easier.

Answer 4

You can use WebLOAD.

I has built in support for automatically launching load machines on EC2, collects all the data and has good reporting capabilities.

Answer 5

For these numbers of requests I would look a Tsung although (similar to Jmeter) this is XML declarative in terms of load specification. If you are more comfortable with coding a web driver then the Grinder would be worth alook (both Open Source).

Answer 6

Actually we use Amazon EC2 to run cloud load test with JMeter. It's not so hard as you think. Advantage of JMeter than two tools (Grinder, Tsung) is documentation and community.

jMeter is old, it works with old protocols (So I can't work with it distributed on EC2)

JMeter last nightly build was yesterday and it means it's not so old as you think. You can check it here. EC2 uses WSDL for API calls, it's common protocol not new, here is EC2 API documentation.