Not A Big Deal

7 April 20164 May 2016

Create your own datasource with Python, Heroku and mLab

Introduction

In a previous post I have implemented an AngularJS client capable to browse the NYT Best Sellers through the NYT API. Back in time I struggled to find an open REST datasource that can be used to play with different technologies, so I have decided to implement my own CRUD services on top of a Mongo database sitting in the cloud. To do so I took advantage of mLab, which provides DB-as-a-service, and Heroku, which provides Platform-as-a-service.

Resources

All the source code for this project is available on GitHub, so you may also consider to watch, fork or star the project! The application is also under continuous integration on TravisCI and all the tests are available at this link.

Structure of the Project

CRUD services have been implemented in Python through the use of the Flask micro-framework. The project has been structured as follows:

The code is divided into a source (test_data_crud_services) and a test package (test_data_crud_services_test). The source code is then organized into the core, rest and config packages, with a self-explanatory content.

The app.py script is the core of the application, and it is in charge to start-up the Flask framework, register the blueprints, initialize CORS filters and finally run the service. The full script is available below:

The root service return the JSON Hyper Schema that describes the available services, their inputs and related outputs.

Setup the DB with mLab

The first thing to do is to create a free account on the mLab signup page. After that, it is possible to create a new database. To have a free hosting you want to select Single Node and then Sandbox(shared, 0.5 GB) in the creation page. Finally, choose a name for the new instance and hit the Create new MongoDB deployment button.

For this project I have created two instances, one for the production environment and one for the tests.

Once all the instances have been created, it is possible to go back to the MongoDB Deployments page. The website will redirect you to a management page by clicking on any of the instances. This page provides tools to create the users who can access the instance, manage the collections, import/export the DB and so forth.

The most important part of this page is the box on the top that states the MongoDB URI required to connect to the DB. This URI looks like something like:

mongodb://<dbuser>:<dbpassword>@ds059145.mlab.com:59145/kalimadata

We will need this URL in the next step: time to implement the business logic!

Business Logic

The datasource I am implementing is very simple: each collection contains only one kind of object (e.g. a book) and there are no relationships between the different collections.

The business logic is implemented by a single DAO which provides the CRUD functionalities we need. The next table summarizes these functions:

Method

Inputs

Output

create

collection_name

item

The id of the new item.

get

collection_name

All the items in the collection.

get_by_id

collection_name

item_id

The item corresponding to the given id.

update

collection_name

item_id

item

Status of the update operation.

delete

collection_name

item_id

Status of the delete operation.

The dao.py script also exposes another method: get_dao(environment). This method is used to configure the datasource for the DAO, either production or test. Through this differentiation it is possible to run the tests on the DB and keep the production environment consistent. The connection parameters (username, password, MongoDB URI and so forth) are stored in two dictionaries in the settings.py script.

Blueprints

The application has a single DAO, but, through the use of blueprints, it is capable to operate on separate collections. It is possible to define a blueprint for each collection, and keep the same business logic. The following example shows the (partial) blueprint for the books collection:

Each blueprint is then registered on a different endpoint in the app.py script as follows:

Each service requires the environment parameter, in order to select the appropriate datasource, as seen in the previous section.

Deploy on Heroku

At this point, the application is almost ready to be deployed. In order to do so through Heroku, we need to provide two more files. The first one, requirements.txt, is a simple list of the project’s dependencies:

The second file, Procfile, tells Heroku how to run the application, and it contains a single line:

Conclusions

This post describes how it is possible to create a simple datasource to provide test data with a cloud DB and a simple Python application. The business logic is implemented by a single DAO. Such DAO is then used by multiple blueprints in order to access several collections in the same DB. The MongoDB is hosted by mLab, while the application is deployed on Heroku.