Hi Guys, I'm using elasticsearch Version "1.4.5" and "lucene_version" : "4.10.4" on elasticsearch cluster consist of 4 nodes. I want to take anapshot from this cluster, I have configured NFS server and made it available to all nodes, but when I take snapshot I'm recieving the following error and snpashot not completed successfully.
Note: the cluster status is green, sometimes I'm getting shard not found from kibana dashboard. any ideas to get rid of this error and make complete successfull snapshot
[2018-05-24 12:47:02,680][WARN ][snapshots ] [MAS-GSDB02-DR] [[logstash-2017.01.17][3]] [smartpass_es_repo:prod_masdar_logstash-2017.01.17_2018-05-24] failed to create snapshot
org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: [logstash-2017.01.17][3] Failed to perform snapshot (index files)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:503)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:139)
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:86)
at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:817)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: /elasticsearch_snapshot/indices/logstash-2017.01.17/3/__0 (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(Unknown Source)
at java.io.FileOutputStream.(Unknown Source)
at java.io.FileOutputStream.(Unknown Source)
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.createOutput(FsBlobContainer.java:85)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshotFile(BlobStoreIndexShardRepository.java:555)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:501)
... 6 more