JEP draft: Adaptive Heap Sizing for ZGC

Summary

Enhance the Z Garbage Collector (ZGC) to dynamically adapt the size of the heap based upon the resources available, the application's needs, and the needs of neighboring applications contending for the same resources.

Goals

• Dynamically adapt the size of the heap in response to changes in the workload and the host system.

• Provide the same throughput and latency as a well-tuned, manually configured heap.

• Ensure that existing command-line options for sizing and tuning the heap continue to work.

Motivation

The Java Virtual Machine presents the illusion of an infinite heap. The role of a garbage collector such as ZGC is to implement that illusion with the finite physical memory of the computer on which the JVM is running, using the facilities of the hardware and the operating system.

ZGC is a tracing garbage collector. To reclaim heap memory that is no longer needed by the application, it finds all of the in-use (i.e., live) objects by starting at roots, such as the static fields of loaded classes and the local variables of each method on the stack, and then following references from one object to another. It compacts the heap by copying live objects from old heap regions with few live objects into new regions, after which it frees the old regions.

The cost of a garbage collection operation thus varies directly with the number and sizes of live objects — not the number and sizes of dead objects, the number of all objects, or the sizes of all objects. The best way to reduce an application's overall GC CPU usage is to let ZGC use more memory for the heap so that more objects can be allocated before garbage collection is needed at all. In other words, overall GC CPU usage varies inversely with heap size.

Configuring the maximum heap size is difficult

By default, the HotSpot JVM uses 25% of the host system's physical memory as the maximum heap size. For example, on a system with 128GB of physical memory, the JVM will use 32GB for the heap.

While 25% of physical memory is a reasonable default, no specific maximum heap size is optimal for all applications. You can therefore specify the maximum heap size via the -Xmx option. Choosing a good maximum heap size requires a deep understanding of your application's behavior. This is best done by measuring your application's memory usage, throughput, and latency with different maximum heap sizes in an experimental environment with a representative workload. In practice, however, such experiments are challenging to get right and thus rarely done. To make matters worse, even when they are done, they must be redone each time the application or the JVM is updated, and their results are not portable to other hardware configurations.

Choosing a bad maximum heap size can have adverse consequences. If the maximum heap size is too small, ZGC will use more CPU time to perform more frequent garbage collections. Performance will degrade, and the application might even run out of memory.

Configuring the JVM to be a good neighbor is difficult

In order to minimize its CPU usage, ZGC will use as much memory as it can, all the way up to the maximum heap size. If the maximum heap size is too large, other adverse consequences can arise: The operating system can run out of physical memory and resort to swapping some virtual memory pages out to disk, thereby degrading the performance of the entire system, or, in the worst case, killing some processes.

The maximum heap size is a hard limit, which cannot be exceeded. You can configure ZGC to be a good neighbor to other processes on the host system by configuring a soft maximum heap size , which is smaller than the hard maximum heap size. ZGC will strive to keep the heap size below the soft maximum by performing more frequent garbage collections, but it may grow the heap beyond the soft maximum, all the way to the hard maximum, if not doing so would stall an allocation or cause the application to run out of memory. (Allocation stalls are disastrous for the latency-sensitive applications that ZGC is intended to support.) The soft maximum allows ZGC to leave memory for other processes while still having a reserve of memory it can use if allocation rates spike unpredictably.

For example, consider an application that runs well with a 2GB heap but occasionally sees spikes in demand that require a 5GB heap. You could set the maximum heap size to 5GB (-Xmx5G), but that would risk impacting other processes. Alternatively, if you set the maximum heap size to 5GB and the soft maximum heap size to 2GB (-Xmx5G -XX:SoftMaxHeapSize=2G), ZGC will keep the heap under 2GB unless a workload spike occurs, in which case it will expand the heap to as much as 5GB and contract it back to 2GB after the spike ends.

Unfortunately, configuring the soft maximum heap size is no easier than configuring the hard maximum heap size, and choosing a bad soft maximum heap size can have the same kinds of adverse consequences.

Adaptive heap sizing

ZGC's soft and hard maximum heap sizes indirectly determine the amount of work that the GC does, since GC CPU usage varies inversely with heap size. Configuring these sizes to achieve a desired tradeoff between memory usage and GC CPU usage is challenging.

It would be far better if the JVM would continuously monitor and adapt the resource usage of the GC to automatically find a good balance between using more memory or CPU, based on measured resource needs and availability. This would reduce the need for manual GC tuning for a majority of applications. It would also be more robust across application and JVM updates, and would be more portable across different hardware configurations. Moreover, it would better handle hard predicted changes in the behavior of the application, which might not have been reflected in benchmarks used for tuning. At the same time, adaptive heap sizing would make the JVM a better neighbor to other applications in the host system.

Description

We propose an adaptive heap sizing policy for ZGC, turned on with the -XX:+ZAdaptiveHeapSizing JVM option. The policy is off by default in JDK 28, with the intention to turn it on by default in a later JDK release. This allows a more graceful transition period. When the feature is enabled, ZGC:

• Uses a default maximum heap size of 100% of physical memory, minus a small reserve;

• Allows advanced users to tune the resource usage policy of GC between CPU and memory; and

• Act as a good neighbor by responding to changes in memory and CPU usage across the host system by expanding or contracting the heap, as though a soft maximum heap size had been set.

The case study below, of multiple applications running in a single container, shows ZGC adjusting the heap size and trading off CPU usage to ensure that all applications run equally well within the constraints of the container.

Tuning resource usage

For most applications, there is no need to tune ZGC's resource usage; the default will work well. You need only choose ZGC when running your application:

$ java -XX:+UseZGC -XX:+ZAdaptiveHeapSizing ...

Advanced users may, however, tune the resource usage policy of the GC between CPU and memory. The adaptive heap sizing policy has a target GC CPU usage. You can force ZGC to use less CPU time, leaving more CPU time for the application, at the cost of increasing the heap size and thus using more physical memory. Or, you can allow ZGC to use more CPU time, leaving less CPU time for the application, in order to reduce the heap size and thus use less physical memory.

To alter the target GC CPU usage, you configure the target intensity of garbage collection via a command-line option. This intensity is an abstract measure that defaults to 5 but can be selected from 1 and above. Higher values make ZGC work more intensively, resulting in more frequent collections, higher CPU usage, and smaller heap sizes; lower values make ZGC work less intensively, triggering less frequent collections but requiring a larger heap. For example, to use a bit less CPU time but a bit more memory than the default setting:

$ java -XX:+UseZGC -XX:+ZAdaptiveHeapSizing -XX:ZGCIntensity=4 ...

Specifying an intensity of 1 causes the JVM to significantly increase the heap memory usage and decrease the GC CPU usage. Conversely, an intensity of 10 causes the JVM to significantly increase GC CPU usage and decrease the heap memory usage. You can, however, specify values above 10, which will likely cause the GC to work as fast as it can to reduce the heap size. The default value of 5 leads to a balanced resource usage profile.

The choice to not expose the target GC CPU overhead directly is deliberate. The abstract intensity measure better signals user intent, while the low level GC CPU overhead target is an implementation detail that may depend on multiple factors, changing from one release to another.

A good steward of CPU and memory

The amount of heap memory in use by live objects changes continuously as the application creates objects and ZGC collects them. As the application warms up, ZGC aggressively expands the heap in order to stay ahead of the application's needs. Then, as the application runs, ZGC iteratively adjusts the size of the heap in order to achieve the target intensity:

• If ZGC's actual CPU usage rises above the target CPU usage, ZGC expands the heap at the end of the next GC cycle. ZGC's actual CPU usage will fall toward the target CPU usage, as specified by the target intensity, and the application's throughput will improve.

• Conversely, if ZGC's actual CPU usage drops below the target CPU usage, ZGC contracts the heap at the end of the next GC cycle. ZGC's actual CPU usage will rise toward the target CPU usage, as specified by the target intensity, and the application's throughput will decline.

Adjustments to the heap's size are smoothed out over time in order to avoid disruptive swings, as described by Tavakolisomeh et al. in their paper Heap Size Adjustment with CPU Control.

ZGC is a generational garbage collector. Newly created objects are placed in the young generation of the heap, while long-lived objects are promoted to the old generation. Frequent minor collections are performed on the young generation, while less frequent major collections are performed on the entire heap. In addition to expanding and contracting the heap, ZGC rebalances the sizes of the young and old generations in order to achieve the target CPU usage. If a series of minor collections causes ZGC's actual CPU usage to rise above the target, ZGC expands the heap immediately.

A good neighbor

If we let ZGC use an arbitrary amount of memory, or an arbitrary amount of CPU time, the host system may not have enough memory or CPU time available to run other processes. Therefore, in addition to monitoring the behavior of the application, ZGC monitors the fractions of the host system's memory and CPU time used by the JVM process and reacts accordingly:

• If the JVM process is using a significant fraction of the host system's memory but a small fraction of the CPU time, ZGC contracts the heap, thereby increasing its CPU usage.

• Conversely, if the JVM process is using a small fraction of the host system's memory but a large fraction of the CPU time, ZGC expands the heap, thereby decreasing its CPU usage.

ZGC prioritizes being a good neighbor over achieving its target intensity. It does so by biasing the default or specified target intensity up or down, as appropriate. In this way, multiple JVMs using ZGC on the same host system will use memory and CPU time in a manner consistent with their respective target intensities.

Case study in adaptive heap sizing

In this case study, we create a container with 16 GB of memory. In that container, we start an instance of the SPECjbb2015 benchmark application, and then an H2 in-memory database benchmark application. When the database benchmark finishes, we start two more instances of SPECjbb2015. We monitor the physical-memory usage of the container ("RAM" in the graph), and the heap usage of these four JVMs, for 42 minutes.

Here are the results (click to enlarge):

The horizontal axis is time; the vertical axis is memory usage.

During the first 12 minutes, the initial instance of SPECjbb2015 processes 5,000 requests per second. ZGC finds a reasonable heap size that yields good performance while fitting within the container's limits. The heap grows slowly, since ZGC's CPU usage is not excessive.

In the second 12 minutes, the H2 database application runs in parallel with the initial SPECjbb2015 instance. The database JVM expands its heap in a burst early on; in response, the SPECjbb2015 JVM immediately contracts its heap, allowing the memory-intensive database JVM to claim a larger share of the available memory. Throughout this period, the SPECjbb2015 instance continues to process 5,000 requests per second without saturating the CPU. Its GC CPU usage is higher, however, thereby degrading both latency and throughput.

After the database application exits, at the 24-minute mark, the initial SPECjbb2015 JVM expands its heap to reclaim the memory that it shared with the database. ZGC expands the heap rapidly in order to reduce its CPU usage, thereby improving both latency and throughput.

In the last phase, starting at the 30-minute mark, two additional SPECjbb2015 instances run in parallel with the initial SPECjbb2015 instance, each processing 5,000 requests per second. Within two minutes, the three JVMs have evenly divided the available memory, ensuring good performance for each.

Alternatives

Garbage collectors in some other language implementations allow you to control the memory/CPU tradeoff by setting a target memory residency, i.e., a target ratio of the amount of live data to the total heap size. For the same residency, however, the GC's CPU usage can vary dramatically. Measuring the amount of live data is, moreover, challenging in generational GCs. Measuring the GC's CPU usage is not only more straightforward, but it more directly represents what users monitor and care about.

Risks and Assumptions

Rather than use the original default maximum heap size of 25% of physical memory, ZGC will now use a default maximum heap size of 100% of physical memory, minus a small reserve. There is thus a risk that, for a given application, ZGC will use more memory than it did in past releases, leaving other processes with insufficient memory. Even with the original default maximum heap size, however, leaving other processes with insufficient memory was already a risk when several JVMs using that default ran in parallel.