We encountered data replication performance issue in both Infinispan 5.1.3.FINAL & 5.1.0.FINAL. I attached a simple example, which could reproduce the issue. In general, the issue occurs on Windows platforms (I tried on XP & Server 2003) more frequently. On Linux, it may only occur once in a few hours. Also, it happens more frequently to TCP than UDP. So my example is using TCP.

To use the attached example, unzip it to a Windows directory and follow these steps:

Go to cache-write-test directory, run "mvn install".

Go to cache-write-test\target\release diretory, run "run.bat" and leave it there.

Open another cmd window, go to the same directory, run "run.bat >out.log", and leave it running for 20-30 minites.

Open another cmd window, go to the same directory, run "grep --regex="[0123456789]\{3,\} milliseconds" out.log". It will show all replication times more than 100 milliseconds, while the normal case takes no time (0 milliseconds). Sometimes the replication could go more than 800 milliseconds.

Could you please suggest what introduces the big jitter in the replication time? My JGroups configuration is under cache-writer-test\src\main\resources\my-jgroups-tcp.xml, please let me know if you see any issue in the configuration.

Per your suggestion on GC, I updated my script, which disabled explicit GC, used CMS GC, and turned on GC logging. I also upgraded to Infinispan 5.1.4.FINAL, which is using JGroups 3.0.9.FINAL. I re-run the test, and the jitters are NOT related to GC.

What you mentioned about thread pools & flow control are the general rules. In my test, I have one cache instance stand by without any activities (just to support the data replication), and another instance updating 50 data objects every 2 seconds. So I don't see those 2 rules would be applicable to my test either.

I attached my simple example again with the new script. Now run the first instance using "run 1", and the second instance using "run 2 >out.log". So could you please try running the test on your Windows platform, and see if you can find out what's the problem? Thanks in advance.