The blog that aims at the largest possible balance of pleasure over pain

Main menu

Tag Archives: nodejs

Post navigation

I’ve been using Restify to create all my REST endpoints for the various services that I write. Invariably, I need to protect these services end-to-end. For client-facing UI’s I do that with OAuth, but my OAuth proxy needs to access the actual underlying service. To do that safely, I do this over HTTPS and I created a service account and used the passport-http module to do BASIC authentication with the backend service.

To use the module, all you have to do is “require” it and make a call to it passing in a callback function to the REST service that you’ll end up calling once authenticated. This is a simple example of that:

If you plan to proxy sites or services that are SSL enabled and are signed with self-signed certs, then you need to be aware that you have to configure a few extra parameters to make sure the SSL handshake happens properly. Otherwise, the request goes through without validating the self-signed certs (which is a strange default behavior IMO).

Found this awesome little npm package called nodemon that allows you to continually develop your node application without having to restart your application every time you make a change to your code. It basically watches the files in your dev directory and restarts the node.js process for you. All you need to do is install it globally and then use nodemon to start your app:

npm install nodemon -g
nodemon server.js

I realize this might be a little lazy, but so be it. Thanks @rem for this!

Overview

If you’ve been following my blog, you’re probably well aware of my penchant for node.js and the few blog posts I’ve already posted on the subject. More recently, I wanted to explore RabbitMQ, a messaging platform similar to JMS, that can be easily leveraged with node.js as an alternative to saving messages with HTTP POST using a REST interface. In this blog post, I’m going to extend the DogTag application I blogged about in DogTag App in 30 minutes – Part 1: Node.js and MongoDB and show you how you can use RabbitMQ to push messages into MongoDB using Node.js on the server and a simple Groovy client as the publisher of those messages.

What is RabbitMQ?

RabbitMQ provides robust messaging that is easy to use and supported by a large number of developer platforms as well as operating systems. Its low barrier to entry makes it quite suitable to use with node.js as you will be amazed at the remarkable little code that is required to establish a RabbitMQ exchange hosted within a node.js application. RabbitMQ is similar to JMS in that it supports many of the same paradigms you may have seen in this or other messaging platforms (e.g., Queues, Topics, Routing, Pub/Sub, RPC, etc.). Primary language support includes Python and Java, but there are so many others supported as well.

In this blog post, I’ll be using a NPM package aptly named node-amqp. AMQP stands for “Advanced Message Queuing Protocol”. AMQP is different from API-level approaches, like JMS, because it is a binary wire-level protocol that defines how data will be formatted for transmission on the network. This results in better interoperability between systems based on a standard supported through the Oasis standards body. AMQP started off as a beta but released a 1.0 version in 2011 and has since grown in popularity (see here for comparison).

RabbitMQ and Node.js

In example I’m about to show you, I’m basically providing a simple means to batch several messages to save new “dog tags” in the DogTags application I referenced earlier. I am going to post the messages to a queue and have the server save them into the database. Let’s take a look at the JavaScript code necessary to do this in node.js:

On lines 1-2, you can see how I’m including the amqp package from NPM and establishing a connection with a provided URL (e.g., amqp://guest:guest@localhost:5672). Once the connection is established we create what’s called an exchange on line 4. Here you define the type of exchange, the name and whether it is durable or not. For further details on this and other settings, I will refer you to the docs here. You may be asking why not just send directly to a queue? While this is possible, it’s not something that is practiced given the messaging model of RabbitMQ; namely, that you should never send any messages directly to a queue. Instead, the producer should send messages to an exchange which knows exactly what to do with them. In my case, I chose “fanout” as my exchange type, which simply broadcasts all the messages to all the queues it is aware of, which in my case is just the one queue defined on line 5. Other exchanges you may want to look into include Direct, Headers and Topic exchanges which are described in detail here.

Once I have defined the exchange and the queue, it is time to bind the queue to the exchange name and subscribe to any messages published to the queue. I do this on line 8-11. It is then a simple matter of processing each message and using the api object I’ve established to save the message to MongoDB and to record an appropriate log message. As an added measure, I also think it is worthwhile to see if the queue is bound to the exchange prior to any processing and report the results the log. I do this on lines 14-16.

The Groovy Publisher

So now that we have the server-side component in place ready to subscribe to any messages, we need to create a publisher to publish messages to the exchange. To do this, I chose to write a CLI tool in Groovy and leverage Spring MQ to publish the message. Groovy is great language to create CLI tools and since it is maintained by the folks at SpringSource, it was just a natural fit to use the SpringMQ framework which can generate AMQP messages without much fuss. Of course, as a prerequisite, you’ll have to grab the latest Groovy binaries so you can run the script. Let’s take a look at the code:

So after our discussion in the previous section, you can follow the code here to see how simple it is to publish a message to the RabbitMQ exchange we set up. One thing I simply love about Groovy is that I don’t need to grab any dependencies or build any jars or define any classpaths. For dependencies, I simply use the @Grab annotation to pull those required for SpringMQ, RabbitMQ and the JSON parser down to the client so that they can run the groovy script without any set-up. In lines 15-20, I set-up the configuration to point to my host (which I recommend using a free aPaaS like OpenShift or Nodejitsu to host your Node.js app). Lines 22-24, I define the RabbitMQ template based on the config, set the exchange name and establish a JSON converter since MongoDB likes JSON format. Finally, in lines 34-55, I parse a CSV file provided as a command line parameter and send each line as an AMQP message to our server.

To run this program, I simply run: groovy RabbitMqUploader.groovy ./dogtags.csv.

Conclusion

With a few simple changes to my original application, it was easy to augment it to support RabbitMQ and define an exchange to consume messages. With Groovy, it was easy to write an autonomous, concise client to establish a connection to my node.js server and broadcast messages over the AMQP wire protocol for the server to process. I expect to use this approach in the future where perhaps a REST interface may not be best suited and using well established messaging paradigms make more sense. I encourage you to explore this interesting messaging protocol for your future needs as well.

Overview

In the previous two parts, you saw how to create a Node.js application and store JSON data in MongoDB (Part 1). You also saw how to present this data in a mobile Web application using Sencha Touch complete with Google Maps and geospatial searching (Part 2). Now, we are going to take both of these two pieces and deploy them on a Platform as a Service (PaaS). I looked at three different provides in this space: OpenShift, Heroku and Nodejitsu. The latter is Node.js specific while the other two you can deploy other types of applications (Java EE, Ruby, PHP, Python, etc.). I really liked RedHat’s OpenShift model. It has several nice dev features that I thought worked well which I will go over in this post. Nodejitsu worked well too but didn’t use Git (a distributed SCM tool) to manage changes. Heroku required a credit card at the point I wanted to introduce MongoDB, so I opted not to continue with Heroku. It doesn’t hurt to experiment with multiple vendors since you can sign-up for free accounts on the respective sites.

One of the main motivations to use a PaaS is that it wouldn’t be very maintainable or production worthy SSH’ing to a server and running your node process every time you wanted to launch your app. Moreover, even if you run your process, it eventually will shutdown when you close your SSH session. There is an npm package forever that can keep this from happening, but I think running Node on a PaaS is ideal.

Using OpenShift to create a Node App

The first thing you’ll want to do is create a free OpenShift account here. Once you have an account created, you can create the application directly from the website (very convenient) or you can install Ruby and the RHC gem (sudo gem install rhc) that allows you to do the same thing from command line. The website is pretty intuitive so I won’t cover this and should you want to use the CLI, the set-up is sufficiently covered here. Note that there are a few prerequisites to use the CLI, namely installed Ruby and Msysgit. Be sure to have those installed as per the provided link before continuing.

For the purposes of deploying the app we created in Part 1, you would do the following after you’ve established your OpenShift domain and added your SSH public keys to commit code from Git:

First, we create “dogtag” as a nodejs app:

$ rhc app create -a dogtag -t nodejs-0.6

Second, we add the MongoDB cartridge:

$ rhc app cartridge add -a myMongo -c mongodb-2.0

Last, we add the RockMongo cartridge to administer MongoDB from the browser (context URI: /rockmongo from your domain)

$ rhc app cartridge add -a myMongo -c rockmongo-1.1

Working with your application in OpenShift

After you have created the app, the next part is copy the DogTag Node.js code to the “dogtag” directory that was created by OpenShift. Since Git is used to manage changes, you can run the following commands to push the code to the server:

$ git add .

$ git commit -am ‘Initial check-in’

$ git push

What will happen then is that the code changes will push to the server, stop the node.js instance, deploy your code and restart the server…very cool! It will pick-up any NPM packages you need as long as you define them in the package.json file (in our case, we need express and mongoose). After that you can go to the provided URL from OpenShift and test the various URL routes we defined in Part 1 (e.g., http://nodetest-loutilities.rhcloud.com/dogtag). You’ll notice that there is no data, so I created a JSON file that you can import into MongoDB to have some initial data. To do the import, you need to run the following command when you SSH to your server’s node:

Note the $OPENSHIFT* provided constants so that you don’t have to figure out the actual host, port, username and password you are on which is especially helpful in a cloud environment running on multiple nodes. Once you import, you can check the files and add a geospatial index, which is required for the geospatial queries as follows:

Now, if you go to the URL again you will have JSON data returned in a JSONP callback. You can change the callback simply by adding &callback=[your callback name]. This is necessary to prevent cross-site scripting issues and required when using the JSONP proxy in Sencha Touch.

Deploying the Mobile App

Since the Mobile App was created with Sencha Touch, I just used a “DIY” (Do it yourself) app from OpenShift to deploy my app there. The process is very similar:

$ rhc app create -a dogtags -t diy-0.1

This creates a very basic Web app. You can cd to the “diy” directory that is created and copy the code from the Mobile app there. It will prompt you to replace index.html, which is fine. There is one thing you need to do in the existing code; you will need to update the store’s proxy to point to your “dogtag” node.js app you created in the previous section as well as the Pettracker controller which also has these references. To do that, simply update the URL with the one OpenShift provides and append “dogtag” on the end to get the list as follows:

Lastly, use git to push the updates to the server. It will deploy the new code and restart the server. Launch your app from a Webkit browser or from your mobile device and you should be in business similar to what you see here.

Once the app is deployed to OpenShift, you can tail the logs by running the following:

$ rhc app tail -a <app-name>

You can also start and stop the server as follows:

$ rhc app [start | stop ] -a <app-name>

And to snapshot a particular revision and go back to it you run the following:

$ rhc app snapshot save -a <app-name>

$ rhc app restore -a <app-name> -f <path to snapshot file>

Conclusion

I hope you found this three part series useful. In addition to creating a Node.js app and front-ending it with a Mobile application, you saw how easy it is to deploy such an application to the cloud.

Overview

This is the second part of a three part series to show how we can use Sencha Touch (a mobile web JavaScript framework) to create the GUI to integrate with our Node.js REST API that we built in Part 1. As you may recall from the previous post, I’m tracking dogs and storing their coordinates in a MongoDB. Now we will create a mobile app that will list the dogs and when selected will show them on a Google Map relative to your current location. Let’s get started.

Connecting to Node.js using MVC

Connecting to Node or really any REST/JSONP service is fairly trivial. Since Sencha Touch supports MVC on the client side, I will illustrate how to create the application using that approach. This is very useful because it keeps with one of the main programming tenets: Separation of Concerns. In this way we can create concise JavaScript applications that break-up the application along the lines of model, view and controller components. First step is to create the model as follows:

Notice above that I’m defining a “Pet” model by extending the Ext.data.Model class and assigning the fields to the model. Additionally, I set types as needed so they aren’t considered strings. Next, we create the store that will populate the model:

The PetTracker Store uses a JSONP proxy and binds the output of the service to the model we previously created. Notice, that I simply put the URL to the Node.js URL binding that will list out all dogtags and use a JSON reader to put them in the model.

Binding the Model with the View and Controller

Now that I have the data, where do I put it in the GUI? Sencha Touch provides several mobile GUI widgets that look very much like native mobile apps. In my case, I decided to use the panel with a docked toolbar at the top. Let’s take a look:

In the example above, I create a panel with fit layout (aka, responsive web design) and put a toolbar and list component in it. To get the data in the list, all I have to do is set the store to the store name I created earlier (line 16). On line 22, you can see that I’m using expression language to templatize the output of the store data in each list row (look Ma, no FOR loops!). Finally, we need to create the Controller to respond to events in the code. In the View code, you see on line 36 that I fire the ‘petSelectCommand’ when a list item is tapped. Let’s look at the controller code for this.

Covering all that’s inside this controller is beyond the scope of this article and besides there’s already really good articles on this such as the one here. If we pick-up from the event we fired in the view, you can see on line 16 where I’ve bound the onPetSelected() method to the event. On line 47, you can see the implementation of that method where I do the slide transition to the map view panel and then render the map in that panel. Finally, there’s this small piece of code to bootstrap the application and launch it.

Conclusion

Well there you have it: how to create a mobile web application integrated with Node.js and Google Maps. You can see this application in action by pointing your mobile device to this site, http://dogtags-loutilities.rhcloud.com/, or internally here. I hope you see the value in implementing client side MVC so that your code is concise and maintainable, especially if multiple developers were to work on it. You can view the complete code base here. Look for the final part of this series where I will show you how easy it is to deploy this application to the cloud using a PaaS like OpenShift, Heroku or Nodejitsu.

Overview

Are you interested in learning how to use Node.js with a NoSQL database like MongoDB to create REST services? Would you like to know how to create a mobile app with Google Maps that can perform geospatial queries with ease? Do you want to see how you can deploy all this to a PaaS with the click of a few buttons and a rudimentary understanding of Git? The answers to this and more are part of my three part series where I will cover the details on how you can create a “Pettracker” like application using Node.js, MongoDB, and Sencha Touch hosted on OpenShift. In this first part I will discuss how to build the middle tier and data store with Node.js and MongoDB.

What is Node.js and MongoDB?

Node.js is a JavaScript framework (API to a C library) for creating highly scaling, non-blocking IO applications that does not create multiple process threads for each invocation incurring a lot less O/S overhead under load. This is accomplished using an event-driven programming model that executes an IO bound call but does not block; it continues to the next line of code. It uses a callback mechanism to conclude the operation with the response once the event is complete. JavaScript is perfect to accommplish this because its native support for closures. Node by itself isn’t that interesting, but akin to Maven in the Java world, Node has npm so that you can add modules to it and expand its functionality further to all sorts of different use cases. In this article you’ll see how I leverage npm for working with MongoDB and creating a Web server router. Finally, Node together with Socket.IO can be used to create real-time Web applications (more on this in the future).

MongoDB is a highly scalable, high performance, open source NoSQL database that stores documents (rows in SQL parlance) into collections (tables in SQL) and is maximized for performance using in-memory cache, replica sets and sharding. It has a very nice query API and supports map/reduce functionality found in Hadoop/HBase. One of the pecular things with NoSQL DB’s is that there is no schema defined ahead of time. So other than installing MongoDB, there isn’t much else that you need to do. Given the fact that there’s no schema it doesn’t mean that we don’t have structure to our data. NoSQL allows for a very flexible schemas that can change easily as your application needs changing. Finally, MongoDB has built-in geospatial querying to support “near” and polygon searches. We’ll explore this in detail.

The DogTag Application

As you may be aware, Pettracker (aka Tagg) is a Qualcomm device/application that can be used to track lost Pets. I work for Qualcomm but I didn’t work on this project;however, I thought it would be interesting to see how I could replicate this application using Node.js and MongoDB, since it would be the perfect use case for something that would require high throughput and geospatial searching. Node.js will provide the middleware both to consume the JSON data from the emitters (i.e., the dog tag transponder) and it will provide the REST interface so that the mobile device can locate the dogs or perform geospatial queries of what’s near the user.

The first step to do all this is to create a Node.js application. I used the Express and Mongoose Node modules to allow me to create a website and to interface with MongoDB. You can add these modules using npm to your node install or add them to your package.json. I’ll talk more about this later in Part 3. Also, you can download the code from my GitHub account here.

Mongoose is a nice Node module that allows me to work with MongoDB data in JavaScript, so in the code above you can see that I am defining “DogTag” schema (a document in Mongo) and setting fields for what should be stored. Mongoose will take care of the unmarshalling/marshalling the JSON data to MongoDB. Next, I need to define the api for the operations I wish to perform shown as follows:

This code provides the CRUD operations to work with MongoDB. I used the schema I created previously to create dogtag objects and I use their save method to persist them into MongoDB. It’s that simple! If I need to find all the dogs, I use find(), or to find one dog by name, I use show(petname). Now, the last API, “near”, is doing a geospatial query using $near and $maxDistance against my “coords” field in the database. One thing to note is that you have to tell MongoDB to index fields that contain lat/lon. For details on this, see here, but all you really do is run this command from a mongo prompt: db.[collection_name].ensureIndex({ [field_name]: "2d" }).

One more thing to note is that I need to return JSONP back to the client by providing a callback name. To do that, I need the client to pass my a queryParameter (called “callback”) that will be the name of the callback method in the JSONP that is returned. I also need to call the built-in JavaScript function JSON.stringify to convert the schema object to JSON. You can see this put to use on lines 17 and 38.

The last piece is do build the node server. Yes, that’s right…you’re actually going to create the HTTP server that will respond to the request. Let’s take a look to see how:

As described earlier, I need to tell node that I’m going to use express and mongoose in my application. One line 3, I simply create the HTTP server. On lines 6-9, I use mongoose to connect to the database. On lines 18-23, I configure URL routing to the controller api’s from the previous example. Note how I use colons to express parameters on the URL and then bind them to a given api. Finally, I tell the server to listen on a certain port and host spit out a start message if everything went successful.

So that’s really all there is to create the necessary middleware to support our DogTag application. It takes in JSON data from the emitters to store in MongoDB and then uses the Express framework to create a router to respond to requests for finding dogs whether by name or through geospatial searching. In the next part I’ll explore how we can use Sencha Touch to create a mobile application that will help a user locate their dog. For a little preview of that application, you can go here to experiment with it here.

Post navigation

Hi, I'm a Solutions Architect for a major Telecom company. I design and develop software solutions leveraging several Java and JavaScript based platforms including ExtJS, Node.js, Spring, etc. I am also experienced using relational and NoSQL database solutions like MongoDB. I help evangelize Rich Internet Applications and open source technologies in Web application development, Web services, search, and developer collaboration. I graduated from the University of Michigan with a M.S. in Computer Science.
Note these views are my personal views and do not necessarily reflect the views of my employer. All code on this site is licensed under BSD.