Search Application with Blacklight

Have you ever faced a problem in your application where your search engine is a bit slow, even though your database is huge? Or maybe you wanted to add a facet search field in just a few minutes? If you encounter these problems and need a solution, then you are reading the right article! While it may be impossible to create an entire eBay-like application in its entirety, within an hour you can certainly create search engine and facets. This is where Blacklight comes in.

So what is Blacklight?

An open source Solr user interface discovery platform (what a mouthful!), Blacklight has highly-configurable Ruby on Rails front-end and can be used to enable the searching and browsing of your collections. Using the Apache Solr search engine to search full text and/or metadata, it’s really a powerful tool which helps us to connect Solr into Rails applications. It has a lot of features and extensions, but in this article, we will be focusing purely on faceted search. You can find more information about other features here.

What is Solr?

Solr is an open source search platform built in Java. It’s a popular search platform for the web because it can index and search multiple sites. It is designed for scalability and fast searching, whose major features are:

Full-text search

Faceted search

Real-time indexing

Database integration

Rich documents handling (DOC, PDF)

What’s next?

Now we know something about Blacklight and Solr, let’s create our eBay clone – on which will sell computers! We will be able to perform a full text search then filter results with multiple facet fields, like:

Condition (new, used)

Screen Size (13.3 inches, 14 inches etc.)

Format (buy now, auction)

Price

Category

Brand

Memory

Step 1 – Check installed Java version

For the first step, you need to have a version of Java already installed. You need to check that you have at least version 8.0:

BrandID and CategoryID will be just an integers for now, which will point to hard-coded categories and brands in the Item class.But we won’t create ActiveRecord associated models. Right now, we are just focusing on Blacklight and SOLR.

Step 6 – Update the Item class

We’ve just created scaffolded Item model. From here, we can create a new item going through the http://localhost:3000/items/new, but first, let’s add some changes to our Item model. We need to validate all needed fields to be present, enumerate some of them and create two constants:

one with available screen sizes

second with memory sizes.

# app/models/item.rb

Step 7 – Update item’s form

Let’s also modify items’ form view:# app/views/items/_form.html.erb

Step 8 – Install Blacklight

Ok, now you are able to create your items, but you can’t search for them yet! Let’s add the Blacklight gem!

Step 9 – Create an empty Solr core

You have installed Blacklight in your application. Now you need an empty core, which you can create using:

$ solr create -c ebay_like

In Solr, the term core is used to refer to a single index and associated transaction log and configuration files (including the solrconfig.xml and Schema files, among others). Basically, it’s a place where we keep our data and search through it. You can check to see if the command was successful by visiting the newly created core: http://localhost:8983/solr/#/~cores/ebay_like

Step 10 – Edit solrconfig.xml

Now you need to modify the solrconfig.xml (/usr/local/Cellar/solr55/5.5.0/server/solr/ebay_like) file. You don’t need to use managed schema, as you want to use classic schema.xml. To achieve it, first let’s add these lines:

Step 11 – Edit schema.xml

Finally, remove the managed-schema file and create the schema.xml in the same folder as solrconfig.xml is localized. Just paste the ready-made file from here. We recommend just extending a default schema.xml file then adding a few custom fields:

Step 13 – Edit config/blacklight.yml

Step 14 – Create a wrapper between Rails and Solr

Now we need to connect Solr with our Rails application. We wrote our own wrapper which connects with Solr – for now Blacklight doesn’t include it.

Add a new class in under the app/models/solr_service.rb:

Step 15 – Modify the Item class

You are able to establish a connection between Rails and SOLR, but you need to store data into SOLR – how? You can do this during a model creation/update. Send a request to SOLR while you create, update or destroy a record. For creating and updating, there is the self.add method. To remove a document, we will use the self.delete_by_id. For our needs, callbacks would be a great idea! We will add two filters, after_commit and before_destroy. After the commit callback will be used when a record will be saved via Rails and SQL query is fully done. The before_destroy callback is used before Rails runs the destroy method and SQL query is done. So let’s write some code…

Add these two callbacks to the Item class:

To_solr method formats a record’s data into a hash, which looks like this:

Step 18 – Let’s test our search engine!

It’s time to put it to the test. Let’s search for a notebook! Search for the phrase ‘Air’.

An entry has been found! Wuhoo! Our search is working…

Step 19 – Add facets

Filtering items is not so easy. You need to add a few facets to your UI. This also means you miss item info in a search results. In the app/controllers/catalog_controller.rb, you should define facets and index fields:

Take a look at the left side of our page. The facets have appeared! See how easy it is? By just adding a few lines of code with specified facets then refreshing, they appeared! Trust us, writing a facet from scratch is not as fast as adding a single line of code!

Now it’s time to play with them to filter search results.

Step 20 – Modify search results

A single search result header will point to the show action from the CatalogController. You want to change it so you have the link to the show action for the ItemsController. To achieve this, you must override the standard Blacklight views – find a correct partial view and change with your file.

The document header is inside the div with ‘DocumentHeader’ class. To find it, we use Github’s search inside the Blacklight repository:

You will find it inside the _index_header_default.html.erb file. You need to add this file to your application. It’s under the app/views/catalog.

Add _index_header_default.html.erb inside in the catalog folder. All data from SOLR is under the document variable. Data from an item is stored in the document._source. If you know how to get item’s data, you can modify link, which points to an item.

Refresh the browser and check if partial has been updated. If it has, success! Now if you click on a link, you will visit an item’s show page: http://localhost:3000/items/2. Now you are inside ‘normal’ view, you can fully customize it – add images, change content, grid etc.

Step 21 – Add more data – fill a database with sample data

In a few steps you added a high-performing search engine to your application. Now searching 2-5 million records is not a problem! You need to create test data. We like to use Faker gem. Add it to the Gemfile under the development and test group:

gem ‘faker’

Then run:

$ bundle install

We modified seeds.rb insite the data catalog, so we are able to run rake db:seed to populate our database with some data:

An important point to note is that you don’t want to run the SolrService.commit method after every item creation. If you have 1 million documents in your core it will be slow, because it makes a commit to Solr. Instead of this, after every item creation, you can add item’s data to Solr by using the SolrService.add method and at the end, run the SolrService.commit once. This will commit all documents to Solr in one run, so it will be much faster! Let’s modify the Item class:

Now you can run rake db:seed. It will take a while – it creates 200_000 records.

$ rake db:seed

When you are done, you can test and see results.

Let’s search for the phrase ‘Black’…

We found 52,812 results. Let’s look how long it took to render and search for everything:

So without any big effort, we created the full application in which we can search for computers then filter them in a very effective way. That’s SOLR and that’s Blacklight!

If you want to check the application’s full source code, you can get it from the repository in our Github account. We have also attached the folder with files from our Solr core to the repository. You can copy it to your Solr cores’ folder (eg: /usr/local/Cellar/solr55/5.5.0/server/solr/).

Blacklight in a real application

In our last project – Manifest App (the full case study article we wrote can be read here), we integrated Rails and Solr using Blacklight. We wanted to customize faceted search and other Blacklight features and we did it without any problems. Blacklight is really flexible for any customizations, and thanks to its open source, it is easy to do so. So if you consider Blacklight a good solution for your application, what are you waiting for?

I hope that you liked this article and it would be useful for you in the future! If you have any questions, feel free to post a comment or contact directly with us at hello@nopio.com.

Related Posts

As you've probably guessed by the title of my article, I still consider Ruby on Rails as a relevant technology that offers a lot of value, especially when combined with ReactJS as it's frontend counterpart. Here's how I approach the topic.

Gitlab Pipeline for Rails is the main part of a powerful GitLab CI/CD tool and can be a useful alternative for other applications like Jenkins and TeamCity. If you’re looking for some more detailed information on exactly how it works, we’ve compiled an example configuration that can help you.