Looks like concurrent GC that we use by default is no longer as affective as it used to be. Even on our own production, public XMPP service the memory collections are not very effective. We see steady memory usage grow and then large collections from time to time.

This causes some problems of service delays during long collections and even OOM on some systems.

We need to find better GC settings which would really give us concurrent and steady collections while the server is running.

History

Testing and experimenting different GC settings is extremely time consuming and expensive. I think we should start with reviewing recent changes in GC for JVM8 and propose best settings this have most logical sense for a typical high load Tigase installation. Then, if we have time and resources we can experiment with different settings.

I think it could also be prudent to include short description of each GC setting that we would include in tigase.conf so it could be possible to enable them selectively (by uncommenting lines) only with intended settings.

Could you provide me with details about Tsung installation used for our daily load tests? I would like to utilize it for this task. I remember Tigase is on cXX machines - correct? And cron job runs sometime in (my) morning - between 1-6am CET ?

it's not only that default ratio of YoungGen/TenuredSpace is bad (2:1 by default) - it can get automatically adjusted and as a result usually ends up in way worse ratio (for example 150M when whole heap was configured as 6G!)

CMS seems to be slightly better on our installations while G1GC results in more and longer pauses;

our statistics doesn't play well with GC, especially when history is enabled (will create separate ticket for this later on):

statistics labels are not interned hence resulting in thousands of duplicated strings:

One last bit is missing. We need a concrete JMV settings that we recommend for our installations and for our customers.

As concluded (and described in JVM settings and recommendations) - there is no one perfect setting that would be ideal for all installations. For majority of medium-to-big installations CMS with enforced NewSize=2 (or adjusted ratio depending on particular usage pattern) should be best choice. In addition, while describing it in the linked guide I've also updated the settings in etc/tigase.conf with those recommendation -- should it be made even more explicit?

A small, single installation with up to 10k users with let's say no more than 4GB RAM 4 core CPU

Something else if you have any idea

The thing is that most of customers/users do not have enough knowledge and understanding of JVM and GC to tweak settings on their own, so we have to provide them with some ready to use defaults which are a good starting points.

A small, single installation with up to 10k users with let's say no more than 4GB RAM 4 core CPU¶

The thing is that most of customers/users do not have enough knowledge and understanding of JVM and GC to tweak settings on their own, so we have to provide them with some ready to use defaults which are a good starting points.

This may be related in a way to #3254.

At any rate - I've updated the documentation with settings for particular setups, which should be a good starting point.