nmslibA similarity search library and a toolkit for evaluation of k-NN methods for generic non-metric spaces

Non-Metric Space Library (NMSLIB)

The latest pre-release is 1.7.3.6. Note that the manual is not updated to reflect some of the changes. In particular, we changed the build procedure for Windows. Also note that the manual targets primiarily developers who will extend the library. For most other folks, Python binding docs should be sufficient. The basic parameter tuning/selection guidelines are also available online.

Non-Metric Space Library (NMSLIB) is an efficient cross-platform similarity search library and a toolkit for evaluation of similarity search methods. The core-library does not have any third-party dependencies.

The goal of the project is to create an effective and comprehensive toolkit for searching in generic non-metric spaces. Being comprehensive is important, because no single method is likely to be sufficient in all cases. Also note that exact solutions are hardly efficient in high dimensions and/or non-metric spaces. Hence, the main focus is on approximate methods.

NMSLIB is an extendible library, which means that is possible to add new search methods and distance functions. NMSLIB can be used directly in C++ and Python (via Python bindings). In addition, it is also possible to build a query server, which can be used from Java (or other languages supported by Apache Thrift). Java has a native client, i.e., it works on many platforms without requiring a C++ library to be installed.

Should you decide to modify the library (and, perhaps, create a pull request), please, use the develoment branch. For generic questions/inquiries, please, use Gitter (see the badge above). Bug reports should be submitted as GitHub issues.

NMSLIB is generic yet fast!

Even though our methods are generic (see e.g., evaluation results in Naidan and Boytsov 2015), they often outperform specialized methods for the Euclidean and/or angular distance (i.e., for the cosine similarity).
Below are the results (as of May 2016) of NMSLIB compared to the best implementations participated in a public evaluation code-named ann-benchmarks. Our main competitors are:

A popular library Annoy, which uses a forest of trees (older version used random-projection trees, the new one seems to use a hierarchical 2-means).

A new library FALCONN, which is a highly-optimized implementation of the multiprobe LSH. It uses a novel type of random projections based on the fast Hadamard transform.

The benchmarks were run on a c4.2xlarge instance on EC2 using a previously unseen subset of 5K queries. The benchmarks employ the following data sets:

We changed the semantics of boolean command line options: they now have to accept a numerical value (0 or 1).

General information

A detailed description is given in the manual. The manual also contains instructions for building under Linux and Windows, extending the library, as well as for debugging the code using Eclipse. Note that the manual is not fully updated to reflect 1.6 changes. Also note that the manual targets primiarily developers who will extend the library. For most other folks, Python binding docs should be sufficient.

The k-NN graph construction algorithm NN-Descent due to Dong et al. 2011 (see the links below), which is also embedded in our library, seems to be covered by a free-to-use license, similar to Apache 2.

FALCONN library's licence is MIT.

Prerequisites

A modern compiler that supports C++11: G++ 4.7, Intel compiler 14, Clang 3.4, or Visual Studio 14 (version 12 can probably be used as well, but the project files need to be downgraded).

64-bit Linux is recommended, but most of our code builds on 64-bit Windows and MACOS as well.

Only for Linux/MACOS: CMake (GNU make is also required)

An Intel or AMD processor that supports SSE 4.2 is recommended

Extended version of the library requires a development version of the following libraries: Boost, GNU scientific library, and Eigen3.

To install additional prerequisite packages on Ubuntu, type the following

sudo apt-get install libboost-all-dev libgsl0-dev libeigen3-dev

Limitations

Currently only static data sets are supported

HNSW currently duplicates memory to create optimized indices

Range/threshold search is not supported by many methods including SW-graph/HNSW

We plan to resolve these issues in the future.

Quick start on Linux

To compile, go to the directory similarity_search and type:

cmake .
make

To build an extended version (need extra library):

cmake . -DWITH_EXTRAS=1
make

You can also download almost every data set used in our previous evaluations (see the section Data sets below). The downloaded data needs to be decompressed (you may need 7z, gzip, and bzip2). Old experimental scripts can be found in the directory previous_releases_scripts. However, they will work only with previous releases.

Note that the benchmarking utility supports caching of ground truth data, so that ground truth data is not recomputed every time this utility is re-run on the same data set.

Query server (Linux-only)

The query server requires Apache Thrift. We used Apache Thrift 0.9.2, but, perhaps, newer versions will work as well.
To install Apache Thrift, you need to build it from source.
This may require additional libraries. On Ubuntu they can be installed as follows:

After Apache Thrift is installed, you need to build the library itself. Then, change the directory
to query_server/cpp_client_server and type make (the makefile may need to be modified,
if Apache Thrift is installed to a non-standard location).
The query server has a similar set of parameters to the benchmarking utility experiment. For example,
you can start the server as follows:

There are also three sample clients implemented in C++, Python,
and Java.
A client reads a string representation of a query object from the standard stream.
The format is the same as the format of objects in a data file.
Here is an example of searching for ten vectors closest to the first data set vector (stored in row one) of a provided sample data file:

Quick start on Windows

Building on Windows requires Visual Studio 2015 Express for Desktop and CMake for Windows. First, generate Visual Studio solution file for 64 bit architecture using CMake GUI. You have to specify both the platform and the version of Visual Studio. Then, the generated solution can be built using Visual Studio. Attention: this way of building on Windows is not well tested yet. We suspect that there might be some issues related to building truly 64-bit binaries.

Data sets

We use several data sets, which were created either by other folks,
or using 3d party software. If you use these data sets, please, consider
giving proper credit. The download scripts prints respective BibTex entries.
More information can be found in the manual.