Help! Sitecore History Engine does not index some of my items!

We, together with our client, have run into an issue with the Sitecore History Engine recently. The story is fairly simple yet coming to a conclusion was not that obvious. Some of our articles did not index in a multi-management server scenario properly. When articles were edited on one server, the second one didn't always pick up and index the changes.

The scenario is as follows:

We have a search based application delivered on top of Sitecore.

The application supports business with entering articles, collecting feeds and otherwise getting content into Sitecore.

The ingestion happens on the Content Management (CM) servers which are load balanced.

From time to time articles edited on one of the servers don’t make their way into our Lucene indexes on the other machine.

Editors assigned by Load Balancer to the other server sometimes can’t see the changes made on the other server.

We have the History Engine enabled and set up to support the multi-server scenario.

My first two thoughts are:

Race condition and timing issues between those two servers or…

Content issue (I’ve ruled that out pretty quickly as this would result in not indexing the article on the other server either).

So what happens? How does the Sitecore History Engine synchronizes between servers?

How history entries are committed

When you dig into the HistoryEngine class you will see that the AddEntry method instantiates the entry as follows:

You can probably start seeing the problem already... The code uses local server dates to determine entries to pickup. The good thing is that it takes the time from before the index rebuild, but this will still work only if your server times are synchronized to a time-period below the time it takes to rebuild your index.

Get the chronology straight…

How does the timeline looks like then for a following scenario:

Two servers in farm synchronizing index,

Server_1 has proper time – rebuilds the index based on the history table,

Server_2 is 5 seconds late – the article is being saved right after the index was triggered for rebuild,

Index rebuild trigger is set to update the index every 30 seconds.

So what is this all about?

Basically if you use Sitecore search/indexing in a distributed scenario and you're going to rely on the index in your daily activities and you're going to rely on the History Engine - you need to make sure that your time is synchronized and your time zones are set properly or you might start seeing items not being picked up by your index.