logstash-forwarder TLS handshake errors

I started using logstash-forwarder to send logs from my cloud hosted servers to my ELK server for analysis. Since it’s just a simple setup, I used the self-gen cert as described on logstash-forwarder’s github page.

Unfortunately, using the example generated a cert that is only good for 30 days. So suddenly my kibana graph show no data for my cloud servers…. ??? After some digging, I found errors like this in the log.

logstash-forwarder[4367]: 2014/07/01 23:24:08.559691 Failed to tls handshake with 172.25.28.52 x509: certificate has expired or is not yet valid

openssl x509 -in logstash-forwarder.crt -noout -text show that the Validity period was only 30 days. D’oh! 🙂

So I generated a new set, this time for 10 years. Why not, it’s for my use and if I am still using it 10 years from now…

Update 2014-07-28

Tried to bring up another server with logstash-forwarder. Except I used latest logstash-forwarder (git pull today 2014/07/25) and started getting this error when starting up LS.

Failed to tls handshake with 172.25.28.52 x509: certificate is valid for , not foo.bar.le.org

After a bit of debugging, comparing certs (exact same MD5 as the ones on working servers), I went googling and bingo!

https://github.com/elasticsearch/logstash-forwarder/issues/221

I see people blaming Go v1.3 TLS changes, but I am still using the same Go v1.2.1 that I built the currently working logstash-forwarder. And as a matter of fact, copying logstash-forwarder from existing working servers over to the new one and it works just fine! So I do not think that it’s Go, but something in the latest commits to logstash-forwarder that broke TLS.

Update 2014-08-17

Turned out to be my self-gen cert ;-P I created a new one, using properly filled out openssl.cnf and a wildcard domain. That works fine with latest trunk and built using go v1.2.1. I’ll update to go v1.3 soon.

FAQs

How to fix corrupted elasticsearch translog.

In 5.0 there is a tool which can be used to truncate corrupt translog files. This doesn't exist in 2.x but there is a workaround:
POST my_index/_close
PUT my_index/_settings
{ "index.engine.force_new_translog": true }
POST my_index/_open
PUT my_index/_settings
{ "index.engine.force_new_translog": false }
NOTE: Any data in the corrupted translog will be lost.

How to size a cluster?

I want to create a new Elasticsearch cluster. What are the recommended sizing guidelines?
Answer:
This is very much a use case dependent answer. The factors that should be taken into considerations are:

How much data do you expect to index?

Frequency of new data. How often is new data to be indexed? Daily? Hourly?