Posts Tagged ‘Elasticsearch’

The system I was working might need more than one Elasticsearch node configured as Master or Load balancer. I wanted to come up with a logic to connect to a different node in case of node failure. That means I have to write logic to check the status of a node before sending any index or search request. If the status failed, take the next node from the configured list and check the status. This is a viable approach but more concerned about the performance impact of my logic.

I started searching for a system that can open up a proxy IP and group the elasticsearch instance’s under that ip. This way the client will be aware of this proxy ip and not the individual ip and port details of Elasticsearch nodes. Also any node failures will be handled by the proxy system. After some exploration I decided to try nginx, it has built in load balancing with very simple configuration. I downloaded the mainline version of nginx for Windows and installed it.

To modify the nginx configuration go to conf\nginx.conf and open the configuration file in any text editor and add your configuration. See my configuration below.

In the upstream I group all my Elasticsearch nodes under elasticcluster. In the server’s proxy_pass I specify the upstream name, eg. http://elasticcluster. That’s it, now I can access my elastic search node via 127.0.0.1:8080 and nginx will route the traffic to any one of my Elasticsearch node in round robin manner. As you can see nginx ip and port is configured via listen and server_name settings.

Now all my search or index request goes to nginx ip and nginx will route the traffic to any of the Elasticsearch node running in my dev machine. I test failover by stopping one of my Elasticsearch node and nginx routed the traffic to the other live nodes.

Issue with localhost

Initially I configured nginx and Elasticsearch to use different ports of localhost. This causes some huge delay in nginx to route traffic to Elasticsearch nodes. I found this post from nginx forum and realized that the issue is with localhost. So I configured all my Elasticsearch instance to use the ip address instead of localhost. Also configured nginx.conf and specify 127.0.0.1 as server_name, you can also give the IP address instead of 127.0.0.1.

Leave your comments below if you find it helpful or have any questions. Thanks for reading.

Last couple of days I was experimenting with ElasticSearch and different client libraries for .NET. In this post I will detailed the implementation of Indexing and searching using ElasticSearch in .NET. For detailed documentation of Elasticsearch visit official site or Joel Abrahamsson post.

I use PlainElastic.Net as my Elastic search client. PlainElastic is a very simple lightweight library for Elasticsearch. It uses plain json for indexing and query, this gives me more freedom and tooling to create json from the user inputs for indexing and query.

To make it more flexible, our system gets data from the database using views. Datareader class converts this data to Dynamic objects. Then converts it to json and pass it to PlainElastic for indexing. Dynamic object makes life more easier as we can reuse this class with different views without worrying about strong types. Below is the Dynamic class I created.

Our views will fetch the data and return as IDataReader. While indexing data, the index helper will iterate through the reader and from the reader the data get loaded to Dynamic ElasticEntity as shown below.

Before indexing the ElasticEntity, it will be serialized to json using the solution provided in Stackoverflow, it uses JavaScriptSerializer and is very fast. Same approach used while deserializing the json result from Elasticsearch while searching. For deserializing Elasticsearch result, I used json.net initially but deserializing is very slow compare to Javascript serializer.

Below is my Elasticsearch Indexer. It’s a very simple class that uses PlainElastic.Net.

I added another class to convert the ElasticEntity to typed object. This helps the caller to convert the ElasticEntity to the Domain objects or to DTO.

/// <summary>
/// This class maps ElasticEntity to any DomainSpecific strongly typed class. Say for e.g.
/// the requesting class wants the search result as a Customer class. This mapper will

/// create an instance of Customer and get the value from ElasticEntity and set /// it to the Properties of Customer. One rule you should follow is
/// the Columns in the index and Properties in the Domain class should match.

This post gives a very simplistic and basic view of our ElasticSearch layer that I created. We have more functionality tailored for our needs. Hope this post helps some one to build a system using ElasticSearch.