Elasticsearch DSL

Elasticsearch info

Compatibility

This client is developed following the latest stable releases, currently v6.3.0. It is generally compatible with older versions of Elasticsearch. Unlike the Python client, we try to keep as much compatibility as possible within a single version of this client, as that’s an easier setup in R world.

Security

You’re fine running ES locally on your machine, but be careful just throwing up ES on a server with a public IP address - make sure to think about security.

Shield - This is a paid product provided by Elastic - so probably only applicable to enterprise users

DIY security - there are a variety of techniques for securing your Elasticsearch. A number of resources are collected in a blog post - tools include putting your ES behind something like Nginx, putting basic auth on top of it, using https, etc.

Note: for the 1.6 and greater upgrades of Elasticsearch, they want you to have java 8 or greater. I downloaded Java 8 from here http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and it seemed to work great.

Upgrading Elasticsearch

I am not totally clear on best practice here, but from what I understand, when you upgrade to a new version of Elasticsearch, place old elasticsearch/data and elasticsearch/config directories into the new installation (elasticsearch/ dir). The new elasticsearch instance with replaced data and config directories should automatically update data to the new version and start working. Maybe if you use homebrew on a Mac to upgrade it takes care of this for you - not sure.

Start Elasticsearch

I create a little bash shortcut called es that does both of the above commands in one step (cd /usr/local/elasticsearch && bin/elasticsearch).

Get some data

Elasticsearch has a bulk load API to load data in fast. The format is pretty weird though. It’s sort of JSON, but would pass no JSON linter. I include a few data sets in elastic so it’s easy to get up and running, and so when you run examples in this package they’ll actually run the same way (hopefully).

I have prepare a non-exported function useful for preparing the weird format that Elasticsearch wants for bulk data loads, that is somewhat specific to PLOS data (See below), but you could modify for your purposes. See make_bulk_plos() and make_bulk_gbif()here.

Shakespeare data

Elasticsearch provides some data on Shakespeare plays. I’ve provided a subset of this data in this package. Get the path for the file specific to your machine:

More data sets

Initialization

The function connect() is used before doing anything else to set the connection details to your remote or local elasticsearch store. The details created by connect() are written to your options for the current session, and are used by elastic functions.

Known pain points

On secure Elasticsearch servers:

HEAD requests don’t seem to work, not sure why

If you allow only GET requests, a number of functions that require POST requests obviously then won’t work. A big one is Search(), but you can use Search_uri() to get around this, which uses GET instead of POST, but you can’t pass a more complicated query via the body