RSS memory growing on idle Java processes

Question

i am running a couple of small servers that are written in java. There is one that listens on a standard socket, the rest communicate with each other using ActiveMQ. i noticed something strange where if i leave the system idle for a couple of hours, the RSS memory either grows by several to tens of megs or shrinks by several to tens of megs. I used jconsole to see what was going on in the servers, but the memory usage and object creation stayed relatively flat. I tested this with both Oracle Java and OpenJDK. I tried using the recommended solution of setting the MALLOC_ARENA_MAX=4, but that did not have an effect. Is there something else going on in the JVM that i am not aware of, and is there a way to stop it?

Setup:

• CentOS 7.4.1708 (2G ram, 4 processors)
• Oracle Java 8u162
• OpenJDK 1.8.0.212
• glibc 2.17

Answer 1

Is there something else going on in the JVM

Yes. Garbage collection, JIT compilation, class loading / unloading, logging, I/O etc. More details here.

RSS of a Java process can easily go up and down by hundreds of megabytes - "several megs" is not typically an issue at all. To find where the native memory is allocated from, turn on Native Memory Tracking feature and/or use async-profiler as described in this answer.

• user9058115: Even when the processes are completely idle, and object creation is flat, and all the processes are doing are waiting on a socket, is it still going to run JIT, run garbage collection, etc? There is no explicit logging going on when the processes are idle.
• apangin: @user9058115 How do you know they are completely idle? You say you connect with jconsole, but this implies transfering lots of data over Java RMI. Only the fact of running JMX server already makes "completely idle" state impossible. Run CPU and allocation profiling, turn on GC and compilation logs, use Native Memory Tracking and other stuff suggested above, and you'll probably see what Java process is really doing when it seems to be idle.