Garbage Collection tuning - G1 is now a good option

With the latest Java 7 or Java 8 version, and a couple of tuning
options, G1GC has grown up enough to be a viable choice. Two of the
settings on that list were critical for making the performance
acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.

I've included some notes on the wiki about how you can size the G1 heap
regions appropriately for your own index.

Re: Garbage Collection tuning - G1 is now a good option

> I've been working with Oracle employees to find better GC tuning
> options. The results are good enough to share with the community:
>
> https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning>
> With the latest Java 7 or Java 8 version, and a couple of tuning
> options, G1GC has grown up enough to be a viable choice. Two of the
> settings on that list were critical for making the performance
> acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
>
> I've included some notes on the wiki about how you can size the G1 heap
> regions appropriately for your own index.
>
> Thanks,
> Shawn
>
>

Re: Garbage Collection tuning - G1 is now a good option

On 1/1/2015 6:35 PM, William Bell wrote:
> But tons of people on this mailing list do not recommend AggressiveOpts
>
> Why do you recommend it?

I haven't done any comparisons with and without it. To call it a
"recommendation" is a little bit strong. I use it, and I am seeing good
results.

My reading indicates that AggressiveOpts basically enables settings that
are being considered for defaults in a later Java version. If they are
being seriously considered for new defaults, they are probably something
that I want to be using. I have also heard that there are sometimes
bugs with that option, but I have not personally had any problems.

If you don't want to use that option, feel free to leave it out. I will
update my wiki page with a note about AggressiveOpts.

Re: Garbage Collection tuning - G1 is now a good option

bq. But tons of people on this mailing list do not recommend AggressiveOpts

It's up to you to decide - that is why it's an option. It will enable more
aggressive options that will tend to perform better. On the other hand,
these more aggressive options and optimizations have a history of being
more buggy. Depending on your needs and risk tolerance, you might make a
different choice.

You probably want to research the specific issues it has been found to
cause and whether those issues are important to you or affect the JVM you
are using. You also have to keep in mind that probably all of the issues
are not known or documented and that updates may introduce new issues. You
face some risk like that no matter what though.

> On 1/1/2015 6:35 PM, William Bell wrote:
> > But tons of people on this mailing list do not recommend AggressiveOpts
> >
> > Why do you recommend it?
>
> I haven't done any comparisons with and without it. To call it a
> "recommendation" is a little bit strong. I use it, and I am seeing good
> results.
>
> My reading indicates that AggressiveOpts basically enables settings that
> are being considered for defaults in a later Java version. If they are
> being seriously considered for new defaults, they are probably something
> that I want to be using. I have also heard that there are sometimes
> bugs with that option, but I have not personally had any problems.
>
> If you don't want to use that option, feel free to leave it out. I will
> update my wiki page with a note about AggressiveOpts.
>
> Thanks,
> Shawn
>
>

Re: Garbage Collection tuning - G1 is now a good option

Not sure about AggressiveOpts, but G1 has been working for us nicely.
We've successfully used it with HBase, Hadoop, Elasticsearch, and other
custom Java apps (all still Java 7, but Java 8 should be even better). Not
sure if we are using in on our Solr instances.

Re: Garbage Collection tuning - G1 is now a good option

> I've been working with Oracle employees to find better GC tuning
> options. The results are good enough to share with the community:
>
> https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning>
> With the latest Java 7 or Java 8 version, and a couple of tuning
> options, G1GC has grown up enough to be a viable choice. Two of the
> settings on that list were critical for making the performance
> acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
>
> I've included some notes on the wiki about how you can size the G1 heap
> regions appropriately for your own index.

A note was just recently added to the Lucene wiki, saying that the G1
collector should never be used with Lucene, because there are bugs that
might cause index corruption. Solr uses Lucene, so this would apply.

Re: Garbage Collection tuning - G1 is now a good option

It looks like 32 bit is affected.

> On 2013-08-14 08:27, Dawid Weiss wrote:
>>
>> Hi everyone,
>>
>> I am a committer to the Lucene/Solr project. We've recently hit what
>> we believe is a JIT/GC bug -- it manifests itself only when G1GC is
>> used, on a 32-bit VM:

> On 1/1/2015 12:10 PM, Shawn Heisey wrote:
> > I've been working with Oracle employees to find better GC tuning
> > options. The results are good enough to share with the community:
> >
> > https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning> >
> > With the latest Java 7 or Java 8 version, and a couple of tuning
> > options, G1GC has grown up enough to be a viable choice. Two of the
> > settings on that list were critical for making the performance
> > acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
> >
> > I've included some notes on the wiki about how you can size the G1 heap
> > regions appropriately for your own index.
>
> A note was just recently added to the Lucene wiki, saying that the G1
> collector should never be used with Lucene, because there are bugs that
> might cause index corruption. Solr uses Lucene, so this would apply.
>
>
> https://wiki.apache.org/lucene-java/JavaBugs#Oracle_Java_.2F_Sun_Java_.2F_OpenJDK_Bugs>
> I have never had any problems with it, but perhaps I spoke too soon when
> recommending G1. Does anyone else have anything to add?
>
> Thanks,
> Shawn
>
>