JMeter distributed test will not finish

Question

I am running a distributed test of an application with JMeter. The slaves are two computers from azure with 320GB RAM each. The heap is increased to 284GB in the .bat-files.

When running the test for 100 users, there are no issues. When running the test with 1000 users, all threads are started correctly, several processes are completed (I checked my database), but no threads ever finishes (see image from command prompt). It was in this state for at least 2 hours before I ended the test forcefully. Screenshot of the commandprompt when running the test for 1000 users

Here is a snippet from the log from jmeter.server.txt file from the slace machines:

java.rmi.ConnectException: Connection refused to host: 10.0.0.4; nested exception is: 
 java.net.ConnectException: Connection refused: connect
 at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source) ~[?:1.8.0_401]
 at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source) ~[?:1.8.0_401]
 at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source) ~[?:1.8.0_401]
 at sun.rmi.server.UnicastRef.invoke(Unknown Source) ~[?:1.8.0_401]
 at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Source) ~[?:1.8.0_401]
 at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source) ~[?:1.8.0_401]
 at com.sun.proxy.$Proxy20.processBatch(Unknown Source) ~[?:?]
 at org.apache.jmeter.samplers.BatchSampleSender.sampleOccurred(BatchSampleSender.java:182) [ApacheJMeter_core.jar:5.6.3]
 at org.apache.jmeter.samplers.DataStrippingSampleSender.sampleOccurred(DataStrippingSampleSender.java:106) [ApacheJMeter_core.jar:5.6.3]
 at org.apache.jmeter.samplers.RemoteListenerWrapper.sampleOccurred(RemoteListenerWrapper.java:94) [ApacheJMeter_core.jar:5.6.3]
 at org.apache.jmeter.threads.ListenerNotifier.notifyListeners(ListenerNotifier.java:58) [ApacheJMeter_core.jar:5.6.3]
 at org.apache.jmeter.threads.JMeterThread.notifyListeners(JMeterThread.java:1042) [ApacheJMeter_core.jar:5.6.3]
 at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:591) [ApacheJMeter_core.jar:5.6.3]
 at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) [ApacheJMeter_core.jar:5.6.3]
 at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) [ApacheJMeter_core.jar:5.6.3]
 at java.lang.Thread.run(Unknown Source) [?:1.8.0_401]
Caused by: java.net.ConnectException: Connection refused: connect
 at java.net.DualStackPlainSocketImpl.connect0(Native Method) ~[?:1.8.0_401]
 at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) ~[?:1.8.0_401]
 at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:1.8.0_401]
 at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:1.8.0_401]
 at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:1.8.0_401]
 at java.net.PlainSocketImpl.connect(Unknown Source) ~[?:1.8.0_401]
 at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:1.8.0_401]
 at java.net.Socket.connect(Unknown Source) ~[?:1.8.0_401]
 at java.net.Socket.connect(Unknown Source) ~[?:1.8.0_401]
 at java.net.Socket.<init>(Unknown Source) ~[?:1.8.0_401]
 at java.net.Socket.<init>(Unknown Source) ~[?:1.8.0_401]
 at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(Unknown Source) ~[?:1.8.0_401]
 at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(Unknown Source) ~[?:1.8.0_401]

I have increases HEAP size. The script contains no listeners, but when running the script from the master machine, we do generate an HTML report and a csv file. Server.rmi.ssl.disable is set to true for an easier connection setup between the master and slaves.

Answer 1

The error means that the slave cannot connect to the master in order to notify it about test completion and send the results back.

In the majority of cases it's due to incorrect RMI configuration. I would recommend explicitly setting client.rmi.localport and ensure that the port is open in your operating system firewall

Also setting heap to 250 GB might be not the best idea because larger heap means longer pauses during the garbage collection, newer versions of Java are smart enough to choose the optimal heap size.

More information: How to Perform Distributed Testing in JMeter

• Anders Stene: Thank you for your response. Do you have an explanation for why it can notify the master when running a test with 100 users, but not when running 1000 users. In my mind, the connection should remain the same?
• Ivan G: I have only one assumption: maybe your operating system runs out of available network sockets, inspect .jtl results file for eventual BindExceptions. Also inspect operating system and jmeter.log files on master and slaves for any suspicious entries. More information on JMeter performance tuning: 9 Easy Solutions for a JMeter Load Test "Out of Memory" Failure