Archive

I use tribe nodes quite a lot at $work. It’s how we federate disparate ELK clusters and able to search across them. There are many reasons to have distinct ELK clusters in each data center and/or region.

Some of these are:

1. Elasticsearch does not work well when there is network latencies, which is guaranteed when your nodes are located geographically distant places. You could spend a lot of money to get fast network connection, or you can just have only local clusters. (Me? I pick saving money and avoiding head aches :-)).

2. It can get insanely expensive to create an ES cluster that span data centers/regions. The network bandwidth requirement, the data charges, the care and feeding of such a latency sensitive cluster…. OMG!

3. I don’t really think a 3rd reason is needed.

Although tribe nodes are great for federating ES clusters, there are some quirks in setting them up and caring for them (not as bad as ES clusters that span datacenter though).

One big gotcha for many people who are setting up tribe nodes for the first time is that tribe node can not create index. Tribe can only update, modify an existing index. What this mean is that if you point Kibana at a tribe node, you must first make sure you Kibana index is already created in one of the downstream ES cluster. Otherwise, you will have to create it yourself.

Otherwise, the first time you create an index pattern and tried to save it, you will get an error similar to the subject of this post.

MasterNotDiscoveredException

The error message is wrong and misleading. It has nothing to do with Master node. It has everything to do with tribe node not able to create (PUT) a Kibana index.

Personally, I prefer to make the Kibana index that I use with tribe to have its own unique name. So I run a dedicated Kibana instance pointing to the dedicated tribe (client) node.

Here are the steps I do to get a tribe node and its associated Kibana ready for use.

1. Configure the tribe node to know all the ES clusters I want to federate data from.

FAQs

How to fix corrupted elasticsearch translog.

In 5.0 there is a tool which can be used to truncate corrupt translog files. This doesn't exist in 2.x but there is a workaround:
POST my_index/_close
PUT my_index/_settings
{ "index.engine.force_new_translog": true }
POST my_index/_open
PUT my_index/_settings
{ "index.engine.force_new_translog": false }
NOTE: Any data in the corrupted translog will be lost.

How to size a cluster?

I want to create a new Elasticsearch cluster. What are the recommended sizing guidelines?
Answer:
This is very much a use case dependent answer. The factors that should be taken into considerations are:

How much data do you expect to index?

Frequency of new data. How often is new data to be indexed? Daily? Hourly?