Setting up a basic Restify API powered by Cassandra and Solr

Dependencies:

What is Solr?

Solr is a powerful search platform built on Lucene. Similar in nature to the more popular ElasticSearch, Solr focuses on text search whereas ElasticSearch provides more analytics capabilities.

Features include:

Full-text search

Highlighting

Faceted search

Real-time indexing

Dynamic clustering

Database integration

XML, JSON, and HTTP support

NoSQL features and rich document handling (Word and PDF files, for example)

There is an actively maintained analysis on ElasticSearch vs Solr better than what I can offerhere.

My Overall Architecture

In this app, the API reads and queries Solr while writing to Cassandra. A separate Data Import service was created to continuously update the indexes in Solr.

The Data Import Service should regularly ping Cassandra using a delta import query after the initial full import, preferably after each write to Cassandra.

Setting up your API with RESTIFY

First, let’s bootstrap a TypeScript project. Get TypeScript if you don’t already have it.

npm i typescript@3.1.6 — save-dev

Init a simple project withtsc — init. I also recommend using Google’s style in tsconfig. You can use their hassle-free setup with thisproject.

Next, get Restify and the CORS middleware

npm i restify@7.2.2 restify-cors-middleware

Then, make a file calledapi.tsand put the following content in there. I will explain each section.

The CORS set up is standard from what you expect from a services with CORs setup. There’s the preflight request, the sanitization of parameters, body parsing, etc. If your service requires more nuanced features, feel free to look at therepofor guidance.

The next bit of code with theinitializeControllersreads fromsrc/controllerdirectory and loads each subdirectory. You can easily customize the logic here to your liking, but I found this configuration to be pretty handy.

Note, that we also need to declare a simple settings for the app.

Below is an example of a controller. It’s just a health endpoint that returns a 200 when you call it.

Every controller in Restify requires a function that intakes the 3 arguments shown above, typed to provide clarity. (You might need to install the respective typings as well if you want Intellisense). Eg,npm i @types/restify --save-dev

This exports the controller method we declared earlier to the corresponding verb and route. Line 46 inapi.tsloads the routes from each route.

You’re now ready to test if the Restify service works. Here’s my npm script that I use to boot up to clean the build dir, compile, and then start up the server.gtscomes from the Google TypeScript. It simply removes everything in the build dir.

Let’s now connect to Cassandra in our API so we can make it execute queries.

GetDatastax’s NodeJS driverwithnpm install cassandra-driver --saveGetting the types can also be helpfulnpm i @types/cassandra-drive --save-dev .

Create a new file calledcassandra-client.tsto house our newCassandraClientclass that we’re going to instantiate on app start and fire off requests from.

In this file, I hardcode the host and port (‘127.0.0.1:9042’) and keyspace (‘cassandrasolr’). It might be prudent for you to move them to a editable config file (same as config.ts).

The interesting stuff is theseedDatafunction that basically seeds the data for you on app start. I create the table if it doesn’t exist, and insert each Object into Cassandra usingthis.client.execute . You do not need to use Batch query as in Cassandra,Batch does not improve speedbut ensures atomicity (either everything is OK or nobody get’s through)

Hereis the JSON data that I use. Drop it in your app accordingly. I also exportCASSANDRA_CLIENTso it can be used elsewhere.

Now, when you restart your server, data should be seeded to Cassandra accordingly. You may confirm it with a query oncqlsh , egSELECT * FROM PEOPLE;

Setting up Solr

Get Solr with homebrewbrew install solr(if it doesn’t get 7.5, you might need to dobrew install solr@7.5).

You can just boot up Solr Cloud with their example configuration for nowsolr -e cloud .

You can navigate tolocalhost:8983and access the Solr Admin console. There isn’t any data in your collection, but we’re going to update your indexes now with the pieces we’ve set up.

Importing Service

For the purpose of this quick demo, I’ll reuse the API app. In the future, I would segregate this functionality and use Cassandra’s built-in triggers to decide when to fire a partial import query. For now I’ll just fire once with a full import of the records in Cassandra.

Here is my ImportController:

I brought in Axios to fire some HTTP requests. Feel free to use whatever HTTP library you’d like. In this endpoint, I fetch the Cassandra data and simply send them over to the/updateendpoint of Solr. Make sure the commit parameter is set to true.

A 200 is returned if the update was successful. Else, a 400 would be returned with the error message. One thing that I would like to have done is to have thesolrResfrom a successful Solr update to have more info. For example, how many records were successful, and if failed, how many and which one failed the import.

You should now be able to directly query your Solr instance using HTTP.

Completing the endpoint

With all the parts set up, we can finally build the endpoint that searches for People.

I took a quick and dirty approach here by allowing the client to pass in query params, but filter it with thebuildQuerymethod and convert the query param into a Solr query.

This allows some sort of control as to what can be exposed. I would recommend a more scalable approach going forward than littering the app with buildQuery functions. Perhaps a single file that maps all the allowed fields per resource.

To investigate

There is so much to go from here on. Here’s a running list of stuff I came upon while working on this tutorial.

Automatically reload and restart the server upon saving

Separating out the DataImport feature into its own service and have a continuously poll the Cassandra instance OR use a Cassandra triggers

Deployment, sharding, and horizontal scaling

Exploring other search options and features of Solr

Feel free to leave a comment or questions!

Join the #1 fastest growing bootstrapped software company in New York!