How to protect your Apache Solr Cluster against the Meltdown and Spectre Vulnerabilities

Meltdown (CVE-2017-5754) and Spectre (CVE-2017-5753 and CVE-2017-5715) are two of the most widespread security issues in the design of modern CPUs and can affect your Apache Solr deployments. These vulnerabilities were disclosed on January 4, 2018. They affect almost every computer made in the last 10 years. These vulnerabilities take advantage of certain processor optimizations. They make it possible for hackers to reveal data stored in memory. In this article, we’ll discuss: how to know if your Solr servers are affected, how to protect your Solr instances, and how to apply the patches to your Solr cluster.

How to know if your servers running Apache Solr are affected?

Spectre and Meltdown affect most major processors. Unless your servers that are running Apache Solr or Zookeeper are specifically patched, these servers are most likely affected. Most Operating System providers have released patches that protect against Meltdown. However, updates to protect against Spectre are still being released as it requires more extensive remediation.

If you are running in the cloud and / or in a virtualized environments, the cloud providers need to update the underlying infrastructure. For major cloud providers, below is the current status:

The infrastructure that runs Azure and isolates customer workloads from each other is protected. This means that other customers running on Azure cannot attack your application using these vulnerabilities. Install updates from your operating system provider when available.

Infrastructure patched against known attacks. Customers must patch/update guest environment.

You, as the user still need to protect your servers against these vulnerabilities. And patch the guest operating system that is running on your servers.

How to protect your Apache Solr instances running in the cloud?

A complete protection against these vulnerabilities would likely require CPU design changes. However, software patches can provide mitigation against these exploits. Unfortunately, these patches are also known to decrease the performance of your servers.

As of January 9th, the following Ubuntu Distributions have released updates to provide some patches to these vulnerabilities. Ensure that your OS kernel is updated to at-least the below patch versions to be certain of some mitigation.

Ubuntu 17.10

kernel 4.13.0-25-generic

Ubuntu 16.04

kernel 4.4.0-109-generic

Ubuntu 14.04

kernel 3.13.0-139-generic

To check which version of the kernel you can execute a “hostnamectl status” command that would provide you the details on the Architecture, Operating System and Kernel. Below is what a sample output on a Microsoft Azure VM looks like:

As you can see above, this server has Ubuntu 16.04 with Kernel 4.11.0-1015. This is higher than the above mentioned, this show that it has been patched.

How can the patches be applied to your Apache Solr Cluster?

Below are the steps you can follow to update your Apache Solr Cluster. These steps can be performed on cloud providers that support VMs (e.g. AWS, Microsoft Azure, Google Cloud Platform). For mission critical sites, our recommendation would be to migrate your search traffic to a different cluster or different data center. However, you could perform these steps in a rolling fashion. If you are setup on SolrCloud with two or more replicas, a single replica can handle your application workload.

We would also like to recommend that you should ensure you have about 200-500mb available storage on your servers to ensure appropriate packages can be downloaded.

Login to one of your Solr servers

Stop Solr Service

sudo service solr stop

Grab the latest of all packages on the server

sudo apt update

Upgrade the Kernel and Packages

sudo apt dist-upgrade

Cleanup all old kernel and OS related packages

sudo apt autoremove

Reboot your machine

sudo reboot

Once the server is back online, ssh in and check the active kernel against the list above to ensure that your kernel has been upgraded.

hostnamecrt status

If Solr service is not started yet, start the solr service

sudo service solr start

Go to your Apache Solr admin and ensure the replica is synced up with the leader and is in active state

Logout

exit

Our recommendation is that you should check for new updates frequently to ensure that you receive future patches as they become available and apply them.