Puppet Server provides many configuration options that can be used to tune the
server for maximum performance and hardware resource utilization. In this guide,
we’ll highlight some of the most important settings that you can use to get
the best performance in your environment.

Puppet Server and JRuby

Before you begin tuning your configuration, it’s helpful to have a little bit
of context on how Puppet Server uses JRuby to handle incoming HTTP requests from
your Puppet agents.

When Puppet Server starts up, it creates a pool of JRuby interpreters to use
as workers when it needs need to execute some of the Puppet Ruby code. You can think
of these almost as individual Ruby “virtual machines” that are controlled by
Puppet Server; it’s not entirely dissimilar to the way that Passenger spawns
several Ruby processes to hand off work to.

Puppet Server isolates these JRuby instances so that they will only be allowed
to handle one request at a time. This ensures that we don’t encounter any
concurrency issues, since the Ruby code is not thread-safe. When an HTTP request
comes in to Puppet Server, and it determines that some Ruby code will need to be
executed in order to handle the request, Puppet Server “borrows” a JRuby instance
from the pool, uses it to do the work, and then “returns” it to the pool. If
there are no JRuby instances available in the pool at the time a request
comes in (presumably because all of the JRuby instances are already in use handling
other requests), Puppet Server will block the request until one becomes available.

(In the future, this approach will allow us to do some really powerful things
such as creating multiple pools of JRubies and isolating each of your Puppet
environments to a single pool, to ensure that there is no pollution from one
Puppet environment to the next.)

This brings us to the two most important settings that you can use to tune your
Puppet Server.

Number of JRubies

The most important setting that you can use to improve the throughput of your
Puppet Server installation is the max-active-instances
setting. The value of this setting is used by Puppet Server to determine how
many JRuby instances to create when the server starts up.

From a practical perspective, this setting basically controls how many Puppet
agent runs Puppet Server can handle concurrently. The minimum value you can
get away with here is 1, and if your installation is small enough that
you’re unlikely to ever have more than one Puppet agent checking in with the
server at exactly the same time, this is totally sufficient.

However, if you specify a value of 1 for this setting, and then you have two
Puppet agent runs hitting the server at the same time, the requests being made by the second agent will be effectively blocked until the server has finished handling all of the requests from the first agent. In other words, one of Puppet Server’s threads will have “borrowed” the single JRuby instance from the pool to handle the requests from the first agent, and only when those requests are completed will it return the JRuby instance
to the pool. At that point, the next thread can “borrow” the JRuby instance to
use to handle the requests from the second agent.

Assuming you have more than one CPU core in your machine, this situation means
that you won’t be getting the maximum possible throughput from your Puppet Server
installation. Increasing the value from 1 to 2 would mean that Puppet Server
could now use a second CPU core to handle the requests from a second Puppet agent
simultaneously.

It follows, then, that the maximum sensible value to use for this setting will
be roughly the number of CPU cores you have in your server. Setting the value
to something much higher than that won’t improve performance, because even if there
are extra JRuby instances available in the pool to do work, they won’t be able
to actually do any work if all of the CPU cores are already busy using JRuby
instances to handle incoming agent requests.

(There are some exceptions to this rule. For example, if you have report processors that make a network connection as part of the processing of a report, and if there is a chance
that the network operation is slow and will block on I/O for some period of time,
then it might make sense to have more JRuby instances than the number of cores. The JVM is smart enough to suspend the thread that is handling those kinds of requests and use the CPUs for other work, assuming there are still JRuby instances available in the pool. In a case like this you might want to set max-active-instances to a value higher than the number of CPUs.)

At this point you may be wondering, “What’s the downside to just setting
max-active-instances to a really high value?” The answer to this question, in
a nutshell, is “memory usage”. This brings us to the other extremely important setting to consider for Puppet Server.

JVM Heap Size

The JVM’s “max heap size” controls the maximum amount of (heap*[1]
memory that the JVM process is allowed to request from the operating system. You
can set this value via the -Xmx command-line argument at JVM startup. (In the
case of Puppet Server, you’ll find this setting in the “defaults” file for Puppet
Server for your operating system; this will generally be something like
/etc/sysconfig/puppetserver or /etc/defaults/puppetserver.)

Upgrade note: If you modified the defaults file in Puppet Server 2.4.x or earlier, then lost those modifications or see Service ':PoolManagerService' not found warnings after upgrading to Puppet Server 2.5, be aware that the package might have attempted to overwrite the file during the upgrade. See the Puppet Server 2.5 release notes for details.

If your application’s memory usage approaches this value, the JVM will try to
get more aggressive with garbage collection to free up memory. In certain
situations, you may see increased CPU activity related to this garbage collection. If the JVM is unable to recover enough memory to keep the application running
smoothly, you will eventually encounter an OutOfMemoryError, and the process
will shut down.

For Puppet Server, we also use a JVM argument,
-XX:HeapDumpOnOutOfMemoryError, to cause the JVM to dump an .hprof file to
disk. This is basically a memory snapshot at the point in time where the
error occurred; it can be loaded into various profiling tools to get a better
understanding of where the memory was being used.

(Note that there is another setting, “min heap size”, that is controlled via
the -Xms setting; Oracle recommends setting this value to the same value that you use for -Xmx.)

The most important factor when determining the max heap size for Puppet Server
is the value of max-active-instances. Each JRuby instance needs to load up
a copy of the Puppet Ruby code, and then needs some amount of memory overhead
for all of the garbage that gets generated during a Puppet catalog compilation.
Also, the memory requirements will vary based on how many Puppet modules you
have in your module path, how much Hiera data you have, etc. At this time we
estimate that a reasonable ballpark figure is about 512MB of RAM per JRuby
instance, but that can vary depending on some characteristics of your Puppet
codebase. For example, if you have a really high number of modules or a great
deal of Hiera data, you might find that you need more than 512MB per JRuby
instance.

You’ll also want to allocate a little extra heap to be used by the rest of the
things going on in Puppet Server: the web server, etc. So, a good rule of thumb
might be 512MB + (max-active-instances * 512MB).

We’re working on some optimizations for really small installations (for testing,
demos, etc.). Puppet Server should run fine with a value of 1 for
max-active-instances and a heap size of 512MB, and we might be able to improve
that further in the future.

Tying Together max-active-instances and Heap Size

We’re still gathering data on what the best default settings are, to try to provide
an out-of-the-box configuration that works well in most environments. In versions
prior to 1.0.8 in the 1.x series (compatible with Puppet 3.x), and prior to 2.1.0
in the 2.x series (compatible with Puppet 4.x), the default
value is num-cpus + 2. This value will be far too high if you’re running on
a system with a large number of CPU cores.

As of Puppet Server 1.0.8 and 2.1.0, if you don’t provide an explicit value for this setting,
we’ll default to num-cpus - 1, with a minimum value of 1 and a maximum value of
4. The maximum value of 4 is probably too low for production environments
with beefy hardware and a high number of Puppet agents checking in, but our
current thinking is that it’s better to ship with a default setting that is too
low and allow you to tune up, than to ship with a default setting that is too
high and causes you to run into OutOfMemory errors. In general,
it’s recommended that you explicitly set this value to something that you think
is reasonable in your environment. To encourage this, we log a warning
message at startup if you haven’t provided an explicit value.

Footnotes

[1] The vast majority of the memory footprint of a JVM process can usually be
accounted for by the heap size. However, there is some amount of non-heap
memory that will always be used, and for programs that call out to native
code at all, there may be a bit more. Generally speaking, the resident
memory usage of a JVM process shouldn’t exceed the max heap size by more
than 256MB or so, but exceeding the max heap size by some amount is normal.