http://justjs.comNodeJS RSS ModuleFri, 18 Aug 2017 03:03:49 GMT
Recently I gave a talk on Object Oriented Functional Programming at LibertyJS in Philadelphia. Object Oriented Functional Programming is a distillation of many of the ideas in presented in this blog, focused around removing pain points and delivering on the promise of code reuse and high-level programming in an asynchronous environment. Recommended viewing.

]]>http://justjs.com/posts/object-oriented-functional-programming-at-libertyjs-2016object-oriented-functional-programming-at-libertyjs-2016
Object oriented programming in JavaScript can be complicated, with pitfalls and sneaky details to remember. And while certain aspects usually stay the same, there are as many styles of JavaScript OOP as there are JavaScript frameworks, because there is no official notion of a "class" in JavaScript.

Most JavaScript implementations of "classes" do have certain features in common. They rely on the "this" keyword to refer to the current object; after all, it's built into the language. They provide a convenience function to implement subclassing, because it's tricky to get right, especially in older browsers. And they have a really tough time handling callbacks in methods, because "this" ceases to refer to the object you expect.

But there is an alternative pattern that is surprisingly popular, in a quiet way. Developers tend to discover it independently. I think it has many benefits, and while it may not be perfect for every project, there's a strong case to be made for it in many. I like to call it the "self pattern."

All the methods are tucked right inside the constructor, which begins with "var self = this". And methods always refer to self... never this. Now we have a safe, lexically scoped way to refer to the object we really want.

Subclassing is quite easy with this pattern, and there's no need for a convenience function. If I want to subclass Dog to make Doge, I just do this:

In addition to "var self = this", a subclass must invoke the parent class constructor with "call" and any additional arguments. Not much overhead, considering you only have to write one constructor per class.

The self pattern also makes it easy to call the original version of a method that I'm redefining in a subclass. If I want to take advantage of the existing "bark" method while still extending its capabilities, I do this:

And If I want to invoke a method as a callback, I can just pass it like any other function:

var doge = new Doge('doge');
setTimeout(doge.meme, 1000);

Advantages of the "self" pattern

This simple approach has the following advantages over prototypal inheritance:

1. Subclassing is simple and straightforward. JavaScript has no "extend" keyword out of the box, just a pile of tinker toys from which it might be built. Every JavaScript framework has its own implementation of subclassing, working around JavaScript's design flaws. If you don't mind locking out 12-20% of all web users (IE8), you can use a solution based on Object.create like "util.inherits" in node, but it's still a lot of code just to get what class-based languages give you for free, and it's different for each framework.

2. Calling the original version of a method you're overriding is simple and straightforward. In plain vanilla prototypal inheritance, writing your own constructors, it looks like:

Dog.bark.call(this);

I have to play with "call" and "this" each time, passing the original arguments to bark (if any) after passing "this". It's a little goofy.

I also have to hardcode the superclass name (Dog) or do gymnastics to avoid it without polluting the global namespace.

With Backbone it looks like this:

MyModel.__super__.clone.call(this);

With the "self" pattern, we can just grab a reference to the "old" method before we install the new one, following what I refer to as the "var super pattern":

var superBark = self.bark; // Save the superclass version of bark
self.bark = function() {
superBark(); // Just call the darn thing. It can already see "self"
// Now do my own stuff
};

3. "Private" properties and methods are a fringe benefit. Want to have properties and methods that other code can't see? Just declare ordinary variables and functions in the constructor, without making them properties of "self." Now you can call them from your constructor and the other methods, but outsiders can't see them.

4. Handling callbacks inside a method is simple and straightforward. This is a big one. As javascript developers, we are constantly creating callbacks.

But when those callbacks are invoked, "this" no longer points to the object we had in mind.

Here's an example of how this goes wrong when we use the "this" keyword in our methods. Sensible-looking code like this just doesn't work:

Either way it's extra work each time and creates a lot of potential for buggy code.

When all of your methods are created inside the constructor in the first place, so that they can always see "self", this bug never happens. You can just write:

setTimeout(self.bark, 1000);

... And because "bark" was defined inside a closure that defines "self" and never refers to "this," the "callback bug" never happens. In fact, you can pass a method directly as a callback function.

Memory and time: a valid concern

There is one area where the prototype keyword does beat the self pattern. It is true that creating objects in this way, with a closure to contain "self" and methods being added to each individual object, takes more memory and time than the prototypal version.

The flyweight pattern

First: do we care? How much this matters depends on your application. But if you're going to create tens of thousands of objects, I would recommend that you follow the well-known flyweight pattern, in which all of the instance objects are simple JSON-friendly objects without methods and a single "manager" object manipulates them.

Using simple JSON-friendly objects in this way has another benefit: you can pass them back and forth to browsers and NoSQL databases as plain JSON objects, because they are.

Subclassing and the Flyweight Pattern

Note that you can still take advantage of subclassing and code reuse. The heavyweight "manager" objects can still subclass each other. And if you have mixed breeds of dogs in your data, a "type" property of each dog can be used to decide which manager to call:

managers[dog.type].bark(dog);

Yes, the manager pattern loses some elegance over calling methods directly on the dog objects. But because it is explicit — each method explicitly takes an object as its first parameter — it is still safer and less bug-prone than the counterintuitive behavior of "this" in callback-driven code.

Still, your mileage may vary. Perhaps you feel it's worth typing "var self = this" at the top of each method to combine the performance advantages of the prototype chain with sane callbacks.

In any case, you really don't need the flyweight pattern unless you're creating enough distinct objects for performance issues to come to the fore. And modern implementations of JavaScript are very fast and efficient to begin with, especially compared to other languages of comparable convenience like Python and Ruby. You may find it takes quite a lot of objects for this issue to become relevant.

Save Your Skills For Paying The Bills

In the end what matters to me is writing simple, intelligible, bug-free code. The "self pattern" helps me get there. In cases with many thousands of objects or more, I add the flyweight pattern to avoid performance issues and still code in a simple, gotcha-free style.

The less time I spend wrestling with my tools, the more I can spent wrestling with the problem at hand. Reducing the cognitive load imposed by the language itself is a great way to reduce the number of WTFs per minute.

Today I made the uploadfs npm module's main .js file jslint compliant (or as close as a node module can be, anyway). jslint is an extremely powerful tool for catching "lint" in JavaScript code, not just excellent at finding likely bugs but also at encouraging code that will be easy for others to maintain and teaching you things you never knew about JavaScript along the way.

jslint is also a little annoying. But as author Douglas Crockford, one of JavaScript's greatest contributors, puts it right up front: "jslint will hurt your feelings." So I guess I shouldn't be too surprised.

Here is a revealing log of what I had to change, what felt valuable or even super-important, what felt unnecessary but ultimately kinda helpful, and what just seemed over the top.

But first I should clarify something that was confusing for me: jslint is frequently used to check (partially) for compliance with JavaScript's new-ish "strict mode," and indeed jslint insists you turn strict mode on. But strict mode and jslint are not the same thing. jslint enforces many things, like indentation conventions, that strict mode doesn't care about. And strict mode, because it is in effect at runtime, can flag things a static analyzer like jslint usually can't see, like attempts to delete properties that can't be deleted.

In general, you shouldn't judge strict mode's value by what jslint complains about.

Thanks to @slicknet for de-confusing me on this point. Let's jump in and see what jslint has to say.

jslint considers leading and trailing underscores a bad practice. Why? Frankly, I'm not sure. Sometimes they are used to signify "private" properties, which could be harmful because JavaScript doesn't actually enforce that in any way. But that's just speculation.

At any rate, this is problematic for me in two ways:

1. The underscore module, which itself helps avoid many worst practices (especially in browser-side code) by providing safe, correctly written tools for functional programming, is traditionally assigned to the _ variable, which makes it convenient to use and thus more likely to be used.

2. Node itself introduces variables like __dirname which also violate this rule.

We can override it by introducing this comment which is interpreted as a command (a "pragma") by jslint:

This feels like sheer pickiness to me, but a case can be made that a language with one code formatting convention is a language that's easier for everyone to pick up and maintain existing code with. So let's suck it up and change every instance of function( to function ( and move on.

jslint wants us to indent everything four spaces rather than two. I like two, but again: conventions make code easier to maintain. Also, it reminds us not to get too crazy with nested callbacks because the code starts dancing off the edge of the screen. (For those wondering how to avoid getting crazy with nested callbacks, see my article on the proper use of the async module.)

Fine, four spaces it is. Batter up!

#1 'uploadfs' was used before it was defined.
return new uploadfs(); // Line 13, Pos 16

OK, this is irritating. The complaint is that I haven't defined uploadfs yet, but I do define it with the function keyword later in the same file. Computers can sort. If it's more readable to do things in this order, that may produce more maintainable code, not less.

But... perhaps this just means I should move my uploadfs constructor to a separate file, and require that file. There's something to be said for that, even if the main module file is about three lines long at that point. For now we'll just move our declaration of module.exports after the constructor and go on with the show. Declarations first, immediate execution second.

Now we get a sea of complaints about separate var statements for each variable:

Now this I simply don't get. You can definitely make the case that separate statements to define each variable are more readable, not less, than one long statement. It's hard to find anybody finds extra var statements befuddling. I'm going with this one under protest.

Fortunately, a few runs prior to this point I did see a better error message, warning that constructor functions (those used with the new keyword) should have uppercase names. I am not sure why this error message no longer appears by the time I reach this point, but it does turn out to be the issue. And I have no problem with the idea that a convention for constructor names is useful. So changing the name of the uploadfs function to Uploadfs, we can move on:

OK, now this is awesome. I have written too much code in other languages in which the || operator returns 1 or 0, or "true" or "false." jslint is reminding me that in JavaScript, the || operator always returns the value of the first "truthy" argument it encounters, whether that is the left or right side. And that means you can use it as a more succinct alternative to the ternary operator in some cases:

imageSizes = options.imageSizes || [];

To complain about this, you'd have to resort to an argument like "I have to write code in other languages too and now I'm going to screw up over there," and I'm not crazy about that line of argument, because it leads to lowest common denominator programming. So I like this rule.

OK, now I have a bunch more functions that complain that I use them before I define them... and now things are getting annoying. I happen to think this is a very readable pattern:

This is good top-down programming (hey, remember your first programming class ever?) and makes the overall intent easy to understand. Our best alternative is to... I don't know what our best alternative is. We can nest the functions directly in the array in the async.series call, which decreases readability. We can declare the functions first, which decreases readability (I want to understand the overall intent before I see these details). Or we can pretend we're writing Java by moving them all into a "class file" and requiring that first. I am not crazy about any of these approaches to what should be a simple problem.

I am going to declare the functions inline. I can collapse them in my editor when I'd rather see an overview of what's going on, at least. That looks like:

Wait, what? Why would it not be cool to use an else statement after a return statement inside a conditional?

Well, it's not wrong. But it is entirely unnecessary, because the return statement means there's no possibility that execution will continue past this point anyway unless the condition was false. So we can write:

This feels like jslint just telling me to stop typing so much, rather than actually protecting me from anything. But there's a larger point, which is that jslint is calling me out to understand my own code fully. If I don't "get" that the else clause can never execute, maybe I don't understand that I've returned...

OK, yeah, that seems like a stretch to me too. But let's move on.

Now we have:

#1 Expected ';' and instead saw ','.
}, // Line 52, Pos 6

This is at the end of a block that began like this:

self.init = function (options, callback) {

Hey, good catch, jslint. This is an assignment statement, so it really ought to end with a semicolon. Ending with a comma will work, but it implies I don't really get what's going on here and that's worth flagging. (Or that this code started out following the object literal notation before I switched to assignment statements. But anyway.)

This is interesting because the "if" statement seems to imply we shouldn't:

var somethingElse;
if (...) {
var t = context.info.width;
}

Can we really marge them? Doesn't that "if" statement create a new scope? Repeat after me: the only safe way to create a new scope in JavaScript is to nest another function! Otherwise the result is very much browser-dependent. Chrome will create a new scope for a "while" statement, Firefox won't (unless you use a Firefox-specific keyword). Yes, we're in node and could make assumptions, but let's not get in the habit of writing code that will bite us if we use it in the browser. Besides, this is an if statement, and I'm not sure if even Chrome creates a new scope there.

Better to write:

var somethingElse, t;
if (...) {
t = context.info.width;
}

Which should mean the same thing in all environments. So that's another good catch from jslint.

The rest of the results for this file were similar, with jslint pointing out when I followed a "function name(args) { ... }" block with a semicolon (hey programmer, you know this wasn't an imperative statement, right?) and, similarly, when I didn't follow an imperative statement with a semicolon.

Overall jslint's input was helpful. It did not catch any bugs, though I suspect it would have told me if I'd had any variables without var statements hanging about. But to be fair to jslint, this module already has strong unit tests, so major bugs have likely been flushed out already. Had I used jslint throughout its development I might have passed those tests sooner.

My one major beef with jslint is the objection to using functions before they are declared, something all browsers in even remotely common use today do support (as does Node, of course). Avoiding that can lead to an awkward style in which it is difficult to understand the overall purpose before diving into the details. But again, to be fair, you can shut this off with the "undef" pragma.

jslint is good for you

jslint bugs me, but it's good for me. I intend to make it part of my regular practice and encourage its use company-wide at P'unk Avenue, though I may decide to shut off the "undef" rule to allow top-down development, and I will certainly keep the rule against leading underscores shut off. One of JavaScript's few serious lingering significant negatives is the availability of significantly dangerous things, like accidentally declaring a global variable in the middle of a function. jslint and strict mode work together to effectively address such risks.

]]>http://justjs.com/posts/jslint-a-tool-for-better-code-quality-and-a-certain-amount-of-picky-nonsensejslint-a-tool-for-better-code-quality-and-a-certain-amount-of-picky-nonsense
Every developer eventually creates something nice they'd like to reuse in other projects. And Node is generally great at that, thanks to the npm repository.

But there is one area where things get sticky. While you can release any code you'd like as an npm module, it is more difficult to package up a collection of Express routes, template views and static asset files in such a way that other developers can just drop your module into their project, initialize it and go.

Adding Routes: Finding A Strategy That Works

Express has a somewhat under-documented mechanism that sounds really useful. I was excited when I first found it: you can make a second app and "app.use" it from the first. This seems a natural way to add routes specific to a module.

Unfortunately, this doesn't work as well as you'd hope, because routes of the first app that are attached after this point still get considered first. Which means you can't have catch-all routes and sub-apps at the same time:

An alternate strategy is to pass the main app object to the cats module and allow it to add routes directly, always with a prefix. And this does work. The catch-all route is matched last as you might expect:

Adding Views: View Folders And Template Languages Are Not What You Think They Are

OK, so we've managed to add routes in our module. Great. Now we need views for those routes to render...

Oh, geez! What template language is the main project even using? Who knows? It might not be the one we've chosen for our module.

Worse, Express has a single setting for the folder where views reside. There is no support for loading views from an alternate location.

What's our way out of this mess? Simple: pick the template language we want and use it directly. Most template languages for Node have their own mechanism to load a file, compile a template, cache that template if it has been asked for before, and execute it on demand. And we're writing this module, so we can pick the template language we want and point it at the folder where our views are kept.

Here's how my Jot module renders the views that come with it. I use the Nunjucks template language, because it's familiar to folks who know Twig, Jinja, Jinja2 or any of several other fully featured template languages. But you can write similar code for other template engines:

Here I require nunjucks, then create a nunjucks "environment" pointing to a subdirectory of my module, then write a simple "partial" function that I can use to render any template. What's more, I also inject "partial" itself into the template so that I can have templates that invoke other templates. This is nice if, like me, you like clean namespaces for every template and dislike variables being shared by included files. I think that cuts down on bugs and results in cleaner code, but you can skip it if it doesn't excite you. Why not? This is your own module. You're not taking away my ability to do it elsewhere in the project.

I also have a simple "render" function, which I call when I'm ready to render a template as a response to the browser:

All I have to do now is put it to work, passing data to the template just as I would if I were calling res.render normally:

return render(res, 'page.html', { name: 'bob' });

Static Assets: get the timing right

We're almost there! There's one more thing our reusable module needs before it can do everything your project-level code can do: a way to serve up static asset files. (What? Your module doesn't need any? Great. But mine often have images, CSS and browser-side JavaScript code they need to deliver.)

Let's say we want static files to be delivered if they appear in the "public" subdirectory of our "cats" module and a matching URL starting with "/cats" is seen. This is similar to what the "express.static" middleware typically does at the top level of a project.

But if we just call:

app.use('/cats', express.static(__dirname + '/public'))

In the "addRoutes" method of our module, after routes have been added to the project, and there are any wildcard routes in the project, we're out of luck. Why? Because by default, as soon as a route is added to an Express app, the Express "router" is added to the app, which means in a nutshell that all routes will be considered first before our static files are considered. And I do mean all routes, not just those in our module.

There are two reasonable workarounds for this:

1. Provide two different methods in your module, "addStatic" and "addRoutes." Require developers to call "addStatic" before they add any routes to their Express app objects. This will work, but it's confusing and people get it wrong a lot. It can be difficult to guarantee that no code has added a route yet in a big project.

2. Consider working around the problem with your own route to deliver static files. We're not really avoiding Express, because we'll take advantage of a function provided by Express to send static files efficiently. Here's a working example:

What does this route do exactly? It matches anything beginning with /cats/, captures the rest of the URL as req.params[0], and uses the handy res.sendfile method to deliver the file. Boom! Almost as simple as an express.static call.

There's one catch though: the user could submit a devious URL containing sequences like /../ that, if taken literally, would allow them to view files anywhere on the system. We address this by rejecting all URLs that contain two periods together (".."). If your static files actually do that on purpose, you can use more subtle code to validate the URL before treating it as a path.

My personal preference is for solution #2, because it makes the process of adding modules to a project more foolproof and predictable. Routes match in the order they were added, including the route that delivers static files. And it becomes intuitive that a generic wildcard route should be added last after all modules have been added to the project. There are no nasty surprises.

No, I'm Not Crazy, or Try These Test Programs

Here are some test programs that demonstrate the issues I ran into above. Be sure to consider these (and try them) if you have an alternative solution in mind. Then, if you think there's a more elegant way, definitely drop me a line. Thanks!

]]>http://justjs.com/posts/creating-reusable-express-modules-with-their-own-routes-views-and-static-assetscreating-reusable-express-modules-with-their-own-routes-views-and-static-assetsToday I realized that singletons in node.js modules cannot be trusted in a project in which several npm modules depend on each other.

Most Node developers have figured out that if two files in your project require the same module, like this:

The cache mechanism of "require" does not consider these to be the same file. In my tests, the two modules get different objects back from require('shared').

Yes, there's a way around this. If your main project also requires "shared," and that copy is installed with npm first before either one or two is installed with npm, and they depend on the same version, then npm will be clever and install the module only once at the main project's level, allowing the require statements in one/index.js and two/index.js to search upwards in the tree until they find the same file and return the same object.

But you'll note there are an awful lot of conditions and provisos in the above paragraph. It's very magical, and things that are magical tend to go wrong in mysterious ways.

This may be one reason why Express, to take a popular npm module as an example, returns a constructor for creating apps rather than just returning an app directly when you require('express'). This forces you to construct an app object and then explicitly pass it to other modules that need it. There is no temptation to rely on the above dodgy, sometimes-not-really-there mechanism to share the same singletons.

There are other reasons to avoid singletons of course. Instances aren't that hard to keep track of, and as long as your code sticks to constructing instances when asked for them and avoids returning singletons, you will never have the "gee, I wish I'd written this module to support more than one instance in a project" conversation. Which can be very unpleasant.

Here's a simple example, taken from my uploadfs module, of an npm module that does this the right way:

]]>http://justjs.com/posts/singletons-in-node-js-modules-cannot-be-trusted-or-why-you-can-t-just-do-var-foo-require-baz-initsingletons-in-node-js-modules-cannot-be-trusted-or-why-you-can-t-just-do-var-foo-require-baz-init
You've started creating your own npm modules, giving code back to the open source community and getting it debugged and improved for free in the bargain. It's fun and easy. But it's also a hassle, publishing your module over and over just to use the latest version in your own project. In this article I'll explain how to use "npm link" to develop npm modules without pain in tandem with the projects that inspired them - and how to avoid doing it all with root privileges.

Creating npm modules: being famous and awesome isn't hard

Just about everybody in the Node community is developing and releasing npm modules for everyone else to use. Node was born well after the simplest business case for open source was well-understood: other people using your stuff "for free" are also debugging and improving it "for free." Whether to open source those tricky bits of readily reusable code isn't even a question in the Node community.

Creating an npm module is ridiculously easy. Create a free, public github repository, clone it to a folder on your computer, type "npm init," answer some questions and you're up and running. Type "npm install --save underscore" in your module's folder to add other npm modules that your module depends on (like, say, underscore) and they are automatically added to your package.json file. Every time you have a new release for the masses to appreciate, type "npm publish". Now everyone (including you) can bring that module into any project with "npm install" and use "require" to summon it into their code, just as they would with any other npm module. It ain't hard.

What is hard is figuring out a good workflow for developing both an npm module and a project that depends on it at the same time.

The obvious way is to maintain your npm module in one folder (let's say, src/appy) and your project in another (src/mysite). Every time you make a change to the appy module, you run "npm publish" and wait. Then, back in src/mysite, you run "npm update" and wait...

npm link: symbolic links to the rescue

Fortunately npm provides a tool to avoid this tedium. And it's easy to use. But there's a catch.

Here's how it's supposed to work:

1. cd to src/appy

2. Run "npm link". This creates a symbolic link from a global folder to the src/appy folder.

3. cd to src/mysite

4. Run "npm link appy". This links "node_modules/appy" in this particular project to the global folder, so that "require" calls looking for appy wind up loading it from your development folder, src/appy.

Mission accomplished... almost. If you installed Node in a typical way, using MacPorts or Ubuntu's apt-get, then npm's "global" folders are probably in a location shared system-wide, like /opt/local/npm or /usr/lib/npm. And this is not good, because it means those "npm link" commands are going to fail unless you run them as root.

Get away from sudo: npm without root

Running everything as root is both tedious and not a good idea. So let's fix our npm installation so that we don't run things as root anymore:

1. Fire up your editor and open the file ".npmrc" in your home directory (note the leading dot).

2. Add this line, substituting the location of your home directory for mine, of course:

prefix = /Users/boutell/npm

3. Create that folder:

mkdir ~/npm

Now npm will install and look for things in ~/npm/bin (the ~ stands for your home directory) rather than in a global folder that only root can write to. You have your own personal npm, and you can run "npm link" without root privileges.

Just remember that any command line tools you install with "npm install -g", such as "forever," are going to wind up in ~/npm/bin. So let's fix that by adding a line to ~/.profile (or ~/.bashrc if that's your preference):

export PATH=~/npm/bin:$PATH

Start a new terminal window and you'll find that commands installed in ~/npm/bin can now be found.

4. One more catch: you probably have a folder called ~/.npm. And if you've been running npm commands with sudo (stop doing that! Never do it again!), you'll get errors when you try to use "npm link" because it can't overwrite what's already there.

It's easy to clean this up, because there is nothing in ~/.npm that can't be recreated by running npm commands again. But just to be on the safe side, let's move it aside:

mv ~/.npm ~/.npm-old-and-busted

Now "npm link" should work for you exactly as advertised. You can make changes "live" in src/appy and they will be immediately visible when "require"d by src/mysite. You can still run "npm publish" when you have a new stable release that's worth publishing to the rest of the world.

Victory!

That's all it takes! Now you have a recipe for painless npm development. Go publish that amazing module we're all waiting for.

]]>http://justjs.com/posts/npm-link-developing-your-own-npm-modules-without-tearsnpm-link-developing-your-own-npm-modules-without-tears
(Or "What I Did On My Summer Vacation")

Hey, did you miss me? Sorry, I've been busy and haven't written much about node lately. But this weekend I attended Startup Weekend Philly. I was tasked to build an interactive website with a lot of data entry needs in a hella hurry. I wanted to make sure I learned something and had fun, so I built it in node and deployed it to Heroku. And here it is.

Some things I learned from the experience:

Mongoose is a big win with table-like data

Sometimes I recommend going straight to MongoDB. But if your data has a clear schema and you don't need inheritance, Mongoose is absolutely worth using. It adds exactly the right amount of schemasauce to MongoDB and no more. You can write this:

Notice that req.params.slug is available to the middleware because the slug parameter is part of the route. Sweet.

nunjucks is a great template engine

nunjucks is definitely ready for prime time. For those of us who think Jade is too weird, too imcomplete or too fussy about not letting you put an overrideable block inside an element's class attribute if that's how you roll (and it totally is - I like overrideable classes on the body element)... and those of us who discover we're the only node developer at Startup Weekend and really don't want to antagonize the frontend developer we just met with a template language that doesn't let them write normal-looking HTML... nunjucks is the way to go. nunjucks is a port of the jinja template engine from the Python world, which is part of the same extended family as django and twig.

I ran into almost no nasty surprises. I added custom filters easily (the famous javascript dateFormat function for instance). And the frontend developer I'd just met, who has previously used PHP templates, had zero trouble getting up to speed with it.

You can refer to the jinja templates developer documentation, it's a very faithful port.

I did hit ONE nasty surprise: if you already have a variable 'job' that was passed into a template (including, say, the template that 'include'd the template you're looking at), and you introduce another variable by that name in a 'for' loop, the original variable shadows the loop variable and you keep getting the same value.

I worked around it by changing the variable name in the loop. It looks like a better way is to just deliver macros in your include files. Macros can take parameters, etc. I didn't get a chance to try that yet.

Installing node on Windows isn't hard, don't be afraid to tell people to do it

I'm a Mac guy. But Alex, my teammate and frontend developer for the weekend, is a Windows guy and he had never seen node before. He got it up and running in 20 minutes, pulled the app from our private github repo, and was pushing code within the hour. You can totally just do:

node server.js

You don't have to carry out some weird marriage of IIS and node as I've seen at some demos, for development at least.

Heroku + node = super easy... but you need Amazon S3 for file uploads

Calls to fs.writeFile, etc. will work and may be useful for temporary files during the execution of the current request. But that file may be gone on the next page refresh because your dyno may be running on another machine in their cloud by then.

Make sure you specify the Content-Type explicitly or it'll be served as application/octet-stream

Make sure you specify 'x-amz-acl': 'public-read' if you want your file to be visible to web browsers

Before you hit "create" on that bucket, pick a region with "read-after-write consistency" if you need to be sure that file can be read as soon as you're told S3 has saved it! As far as I know the only region you can NOT use is "US Standard," which is the default. I used "US Oregon."

Specify the right endpoint explicitly or you'll get "307" redirects, which knox does not follow automatically for you even when using convenience APIs like putFile. knox will invoke your callback without an error, but your file will never make it to S3. Ouch.

Here's some code. Note the 'endpoint' argument. This is really just the hostname knox will do PUT requests to. It has to be made up of your bucket name, plus the subdomain name for the Amazon S3 region your bucket is in, or the 307 redirect issue will clobber you.

Go to startup weekends

They are full of creative people with ideas worth pursuing. You'll meet cool folks, have fun and stretch your brain. You might even start a company.

Bring a skeleton app

I arrived at Startup Weekend with a skeleton application that rendered pages with tabs and breadcrumbs, stored things in MongoDB and was ready to deploy to Heroku. That helped a lot - Node is a very "a la carte" environment in which you can spend a lot of time making decisions about the fundamentals, time you don't have at a startup weekend. My skeleton app should have been even more complete - I should have made sure I already had user logins baked in via Passport local authentication, for instance, with a reasonable security policy. Live and learn!

Evangelize Node

It's not hard! I was the only person developing with Node at this particular startup weekend. I let folks know I'd be using Node on the facebook wall for the event, and many people came by to check out what I was doing. It's a great opportunity to spread the word.

]]>http://justjs.com/posts/node-lessons-from-startup-weekend-phillynode-lessons-from-startup-weekend-philly
You can do just about anything with JavaScript these days. That includes altering the behavior of your web browser to suit your needs - not just on your own sites, but on any site you care to improve. And anyone who shares your definition of "improvement" can get the same benefits if you package your work as a Google Chrome extension and make it available via the Chrome store.

My friends are in a band. The band is called Hot Breakfast. They are "Delaware's premier acoustic dork-rock power duo." I heart them and their silly covers and clever original songs. But the point of this love fest is that, last week, they tweeted:

Getting Started: The Manifest

Here's how to create a Google Chrome extension that enhances Google Calendar - or any website - even if that site insists on constantly re-creating its buttons with different IDs (I'm looking at you, Google).

Start by creating a folder called "chrome-extensions", then create a folder called "today" (or whatever your extension will be called) inside that.

Now pop that folder open in your favorite text editor and create a file called "manifest.json." Here's mine:

"version" is also under your control. Versions like "1.0.0" work. Versions like "1.00" do not. Use the x.y.z syntax. You need to change "version" to a higher number with every new update of your extension that you release.

"manifest_version" should always be 2 (as of this writing).

"description" is yours to change.

"background" should contain a list of JavaScript files that will be launched by Chrome, attached to "background pages" - invisible pages the user never sees. A typical extension just needs a single "background page" on which to listen for Chrome events that are made available to all extensions. Your JavaScript code running in your background page can respond to those events by injecting other JavaScript files into pages the user can see - such as the Google Calendar tab. So this should be all you need here:

The "permissions" setting determines what your extension is allowed to do. There are several permissions, including "tabs," that refer to aspects of the browser. You need the "tabs" permission in order to respond to events that tell your extension when the user opens a new tab or navigates to a new page in a tab, which is important for our purposes so that we can spot access to Google Calendar. We also request access to https://www.google.com/ so that we can interact with the Google site from the extension's background page if we want to.

Finally, "web_accessible_resources" allows us to make files we bundle with the extension visible to our JavaScript code, like this:

chrome-extension://ID-OF-YOUR-APP/neildiamond.png

"Great, what's the ID of my app?" Good question. Google generates these for you. You'll have one ID during development and another once you add it to the Chrome store. Which means you'll need to update your JavaScript with the final ID once you spot it in your address bar after registering the extension. I'll explain that part in more detail when we get to that stage.

Background Pages and Chrome Events

OK, our manifest file is complete. Let's look at some code. The background.js file, as I mentioned, runs on a "background page." What that really means is that there is no visible page in the browser for it to interact with, but it can still listen to Chrome events in order to learn about what's happening in any tab and start interacting with pages in any tab. A background page can in fact use pretty much all of the features documented in the Chrome Developer's Guide. But it can't directly modify a page that is visible in a tab.

Since this file can't interact directly with the Google Calendar tab, its main job is to take notice when the user opens Google Calendar and then inject a separate JavaScript file into that tab. Here's how it works:

What's going on here? When we call chrome.tabs.onUpdated.addListener, we're passing a callback function that will be invoked every time a tab is updated - meaning that the user has created a new tab or navigated to a new page in a tab. This allows us to listen to what the user is doing and spot the Google Calendar tab when it is opened.

The callback function receives a tabId, which can be used to identify the tab we want to manipulate, a changeInfo object, which has a "status" property that tells us what is happening to the tab right now, and a "tab" object that has a "url" property. The url property is useful for figuring out whether this is the tab we want to interact with.

We're only interested when a tab has loaded a page and is ready to interact with, so we'll ignore all events that don't specify that the tab is "complete":

if (changeInfo.status !== 'complete')
{
return;
}

And we're only interested in URLs that are part of Google Calendar, so let's spot those by using tab.url.substring() to check whether the URL begins appropriately:

Now we can inject a second JavaScript file into the tab. We use tabId to identify the tab we're interested in and specify the file we want to inject. We'll be sure to bundle that file with our extension:

chrome.tabs.executeScript(tabId, { file: 'today.js' });

"What does injecting a JavaScript file into a tab do?" It loads the JavaScript code in that file and executes it in the context of the page, so it can do anything that a JavaScript file loaded directly in that page via a script tag would be able to do.

today.js: adding our own JavaScript to Google Calendar

So far, so simple. Now let's check out today.js, the file that does the tricky work of overriding Google Calendar's "today" button... even though Google loves to replace that button. Really, really loves it a lot.

We'll start by wrapping the entire file in an anonymous function. We do that to avoid polluting the global namespace. In other words, we do that to avoid jamming a named function into the page that might conflict with Google's functions that we know nothing about. After all, we have no need to do so! Our own functions can call each other perfectly well from inside the closure created by the anonymous function. And since we're trespassers on somebody else's lawn, it behooves us to avoid leaving behind a mess.

This is a great way to wrap code in an anonymous function and then immediately invoke it:

(function() { ... })();

I'm honestly not sure why the first set of parentheses are necessary. But they are.

Inside our closure, let's get started by establishing a variable to hold the id of the extension, so that we only have to change it in one place:

var id = 'myextensionid';

Now let's preload the audio file that we're bundling into our extension, so that we can play it whenever we're ready (specifically, when the user clicks the "today" button). In my experience there is still a brief delay on the first playback, which is annoying. If you know a workaround, I'm all ears.

Notice that we're taking advantage of the ability to access files bundles with the extension via a chrome-extension:// URL, as I mentioned earlier. Just be sure you include them in the web_accessible_resources section of your manifest.json file.

Next we'll preload an image to be animated when the "today" button is clicked:

We'll be adding this image object to the page, exactly as if it had been placed there with an <img src="..." /> tag. But first, we want to make sure it's not visible until we want it to be. We also want it to be absolutely positioned so that we can easily animate it by changing its CSS properties. It's easy to access the CSS properties of any HTML element with [removed]

myImg.style.position = 'absolute';
myImg.style.display = 'none';

And now we'll add it to the end of the body of the page. Since today.js runs in the context of the page this is just as easy as it would be for a script tag inside the page:

document.body.appendChild(myImg);

"Hey, you're doing a lot of DOM and CSS manipulation here and you're not using jQuery. What gives?" Well, you're right: I could use jQuery, and it would make my code shorter. However, that would require that I bundle jQuery with the extension, or rely on it already being present on the page. I'm not crazy about either of those ideas. Google is free to change how their page works, and I want as few dependencies on that as possible. And the more we bundle with the extension the more we affect the performance of the browser. An extension like this is fun right up until the moment it's perceived as a performance killer; then it's game over. Chrome extensions that modify existing web apps should be as light as possible.

Also, since we know for a fact that we're running in Chrome, we can rely on modern JavaScript functions that might be missing in other browsers - functions that jQuery uses internally to implement its snazzy selectors with better performance in modern browsers. In a nutshell, working without jQuery is a lot more pleasant in Chrome.

Enhancing Buttons: Every 100 Milliseconds, They Come To America

Now the job is simple: find the "Today" button, add an event handler, do some animation and play some audio. That's all there is to it, right?

Well... not quite. Since Chrome replaces the "Today" button pretty much every time you navigate, any event handlers we add to it are likely to be gone by the time we want them.

Fortunately, we can work around it. Here's the basic strategy:

1. Every 100 milliseconds (10 times a second), check the page for buttons with the "date-nav-today" class.

2. Dig into those and locate the "goog-imageless-button-content" class. This is the actual text of the button, and it doesn't already have a click event handler attached to it, which means we can attach our own. We'll call this element "inner."

3. Check whether "inner" has been enhanced already. If it has, leave it alone so we don't install two click handlers. If doesn't, add our enhancement and set a data attribute so we can detect that we already did so.

Setting up a function to be called every 100 milliseconds is easy:

setInterval(function() { ... }, 100);

Note that when you set up functions to be called on an interval like this and you are enhancing someone else's page, you should make very sure they use only a small fraction of the time between calls to do their work. If you can't do it fast, do it less often.

Now let's delve into the function we're calling at regular intervals. Here's the logic to grab the outer wrapper of the "today" buttons:

var buttons = document.getElementsByClassName('date-nav-today');

document.getElementsByClassName doesn't exist in some older browsers, but it's safe as houses in Chrome.

Now loop through those wrappers. For each one, locate the inner button:

The click handler callback plays the audio, after first resetting it to its beginning so that we can play it more than once. And it also kicks off the animation, which we'll look at in a moment.

Once we've added the event handler, we need to set our data attribute to avoid enhancing the same button over and over:

inner.setAttribute('data-' + id + '-set', 1);

A Little Light Trigonometry

That's it! Hot Breakfast's original request has been satisfied. Honor has been served. Today is a good day to click "Today." Except... I'm vaguely dissatisfied. Sure, the band will be amused, but they won't be awestruck. We should fix that.

So let's animate a picture of the beloved artist to swoop up from the lower left corner and fade out at right after sweeping out a semicircle.

We'll start by taking note of the usable dimensions of the browser window and note that our "x" coordinate starts out at far left:

Now let's use another interval timer to call a function every 20 milliseconds (50 times a second) to move the artist across the screen, one step at a time:

var interval = setInterval(function() { ... }, 50);

This time around, we capture the interval timer in a variable called "interval." We do that so that we can cancel the interval timer when the artist reaches the far right side of the screen.

Let's look inside the callback of our interval timer. The first step is to move the artist's image element on the X axis. This is easy; just set the "left" CSS property. Don't forget to add "px" to the string, otherwise it's not valid CSS:

myImg.style.left = x + 'px';

The Y axis is a little trickier. We'll start by scaling our X coordinate to a value between 0 and 1 rather than a number of pixels. That allows us to think about a unit circle (a circle of radius 1), which is easier:

var cx = (x - (width / 2)) / (width / 2);

Now we can compute the corresponding Y coordinate on the unit circle. Remember, x squared plus y squared equals the radius. This is a simple unit circle (radius 1), so solving for y:

var cy = Math.sqrt(1 - cx * cx);

Now we can pick a reasonable Y coordinate, scaling up our Y coordinate on the unit circle by half the height in pixels and offsetting that from a reasonable starting point in the window:

myImg.style.top = (height - 225 - cy * (height / 2)) + 'px';

Finally, don't forget to make the image element visible (recall we started out with display = 'none'):

myImg.style.display = 'block';

We'll also set the z index to ensure the artist floats majestically above the rest of the page:

myImg.style.zIndex = 999;

Having the artist sweep across the screen and just disappear is a bit jarring. We could move him all the way off the screen, but when we do that a distracting horizontal scrollbar appears for a moment. It'd be nicer if the artist faded in and out at the beginning and end. So let's use the "opacity" CSS property to achieve that. An opacity of 1.0 is 100% opaque, while an opacity of 0.0 is 100% transparent (completely faded out). While the artist is at the far left and right, we'll fade him in and out; while the artist is in the middle we'll leave him 100% visible:

Once you're there, check the "Developer Mode" box at top right. You'll gain the new option "Load unpacked extension."

Click the "Load unpacked extension" button, then browse to your "today" folder (or whatever you called your extension's folder) and select the folder. Google Chrome will load your extension and display it along with the rest of your Chrome extensions. The ID of your extension will be visible in the list.

Now go to your background.js file and paste in the ID of your extension on this line:

var id = "....";

Then click "Update extensions now" to reload your extension with the correct setting for the id. The icon of your extension should appear at upper right, next to the wrench to the right of the address bar. Clicking the icon won't do anything unless you configured a popup file.

Now open a new tab and visit your Google Calendar. Click the arrow to navigate to next week, then click "Today," and...

Well! Wasn't that special!

(If an extension doesn't work at first, you may find that restarting Chrome is helpful.)

Sharing Your Extension With the World via the Chrome Web Store

We've learned how to inject a little culture into the "Today" button of Google Calendar... but naturally we want to share our extension with the rest of the world. The easiest way to do that is by releasing your extension in the Chrome web store via the developer dashboard. This is a refreshingly simple process for anyone who has suffered through releasing an iPhone app! You do have to make a one-time $5 payment, but I trust $5 is not a huge dealbreaker for you. After that, all you have to do is upload a .zip file of your extension's folder. You'll be prompted to describe your extension and given an opportunity to add a separate icon for the store as well as screenshots.

But don't announce your extension to your friends yet! Once you have uploaded your extension, you'll be able to see your ID in the URL in the address bar:

https://chrome.google.com/webstore/detail/your-id-is-here

This is your final ID in the chrome store, which is different from the one you had in development. Why yes, it is rather annoying.

Update the ID in your background.js file, change the "version" property in your manifest.json file to the next minor version (change 1.0.0 to 1.0.1 for instance), update the zip file of your folder to include that updated file, and click "Edit" on the developer dashboard to upload the updated zip file.

Now go to your own Chrome extensions page and remove the development version of your extension. Then return to your extension's page in the Chrome web store. Make sure the displayed version number is 1.0.1 (there may be a delay; you may need to refresh the page a bit). Then click "Add to Chrome."

If you get a complaint about your manifest.json file at this point, try restarting Chrome. This only seems to happen to me as a developer; my end users have no trouble adding the extension.

Once you've successfully added the published version of your extension to your browser and tested it out with Google Calendar, you're ready to tell the world about your extension. You can do that by sharing the URL of your extension's page in the web store.

We've learned how to do a frivolous, pointless, totally absurd thing! However, we've also learned how to do a deeply practical thing: enhance web apps we didn't write ourselves. Chrome Extensions provide a way for users to opt in to enhanced experiences all over the web. It's a very democratic phenomenon. I like to think the artist would approve.

Copyright note: the song "Coming to America" and Neil Diamond's publicity photographs are the property of Neil Diamond and his assigns. I will gladly remove the brief audio sample and the image if the artist or those to whom he has assigned the relevant rights state any objection. I hope this will be viewed as a harmless form of fan fiction dedicated to an artist many of us rediscovered via the Pulp Fiction soundtrack. But it's their call to make.

]]>http://justjs.com/posts/chrome-extensions-how-to-enhance-google-calendar-and-other-web-apps-you-didn-t-writechrome-extensions-how-to-enhance-google-calendar-and-other-web-apps-you-didn-t-write@synomat is a Twitter-powered thesaurus. It's easy to play with: tweet a word to @synomat and she tweets back a list of synonyms. Simple enough, but how does it work? node.js and the ntwitter module to the rescue!

It might surprise you to learn that not every node.js app is a website. Fundamentally, node.js is not a webserver, although it contains modules that are stupendously good at it. At its core, Node is an engine for doing work asynchronously as the need arises, responding to events as they happen. Node is good for pretty much anything that meets that description. And running a Twitter bot - an automated Twitter account that provides helpful services to the public - definitely meets that description.

ntwitter is a Node module that provides access to the Twitter API. That includes lots of things: logging in, tweeting, searching, and monitoring "streams" of tweets by particular users or groups of users. That last feature is really, really great for writing automated Twitter clients or "bots."
Rather than using the http module (either directly or indirectly via Express), @synomat uses ntwitter as its main point of departure.

Of course, it takes one more thing to make a Twitter thesaurus bot: a decent thesaurus! For that we should thank Grady Ward, who released a public domain thesaurus back in 1996.

Ward's thesaurus, aka the "Moby thesaurus," is as simple as they come. Each line contains a comma-separated list of words. The first word is the "base word" the user is looking up; the rest are synonyms. And as it turns out, it's very easy to parse this format with Node.

We'll start out by importing modules. You should be familiar with all of these by now, except for ntwitter. We also require an options.js file which will contain Twitter credentials for both a Twitter app - our bot must be registered as a Twitter application - and a Twitter user:

The file also contains an access_token_key (which Twitter refers to as an "access token") and an access_token_secret. These grant the application access to a specific user's account. In the vidmood application, users grant permission to the vidmood app manually, and then the access token and secret come into being without our intervention. But for a simple app like @synomat that will be tweeting on behalf of our own Twitter account, we can skip that manual process and generate the token and secret directly on dev.twitter.com. To do that, first register your app, then click on the "details" tab and scroll down. You'll see a "create my access token" button. Click that, then you'll be able to copy and paste the access token and secret. If you don't see them right away, try clicking over to the "setting" tab, then back to "details." (Twitter does claim there may be some delay before the information becomes available.)

Here's a full-fledged options.js file. Naturally for your own bot you would change the username as well as the credentials:

"Why can't I just put a username and password in there and be done with it?" Twitter requires that OpenAuth be used by all clients. Yes, it's a pain for simple apps like this that don't act on behalf of third parties, but I don't make the rules. Fortunately the ntwitter module does the tough bits for us. We just supply the credentials.

Now we're ready to take advantage of the ntwitter module. It's possible to interact with Twitter on behalf of more than one user or app, so rather than use the ntwitter module directly, we use it to construct a new object representing this particular user and app:

var tweeter = new ntwitter(options.twitter);

Now let's stash our bot's Twitter username in a global variable where we can use it more conveniently:

var me = options.twitter.username;

Our next steps are to load the thesaurus file, then start up the Twitter bot. I've abstracted these tasks to functions to make the code easier to read:

var thesaurus = loadThesaurus();
bot();

Loading the thesaurus is easy. Since the bot hasn't started up yet, it's acceptable for us to read the whole file with a synchronous call, even though that takes some time. We can split it into lines with JavaScript's familiar split() method, then iterate over those lines with Underscore's each():

Of course, we also need to split up each line of the thesaurus file. The words in that file are separated by commas, so we'll split each line at the commas:

var words = line.split(',');

We'll ignore empty lines by checking for an empty array. If the array isn't empty, we know the first word is the term the rest are synonyms for. Pop the list of words, nicely punctuated, into our thesaurus object with the term as the name of the property, and we're ready to go... right?

Well... almost. The thesaurus contains lots of synonyms for most words. And tweets can only be 140 characters long. We could respond with more than one tweet, but it's not really in the Twitter spirit.

The laziest solution? Join the words up again, separated the way we'd like them to be punctuated when we tweet them out. Check the length. If it's too long, pop the last word off the list and try again. If it's short enough, we're done. This isn't the fastest way, but it's good enough for our purposes.

Here's the code to shorten the list before adding it to the thesaurus. Note the arithmetic to make sure our tweets aren't too long. We need space for the username we're replying to, the punctuation around it and the response itself. The "pop()" method is a handy way to remove the last element from an array.

"Hey, why are you loading the thesaurus into memory each time we start up the app? Can't you load it into a database?" Well, sure we could. The thesaurus does take up a fair bit of memory, and if we pushed it into MongoDB, we wouldn't have to execute this code each time the app starts up. But the app doesn't start up very often, and we can make that transition at any time. Meanwhile, having the thesaurus in memory means that lookups are even faster than they'd be with MongoDB.

The "verifyCredentials" method of our tweeter object confirms that our Twitter credentials are good. Now we're ready to do more exciting things.

All of the methods of the tweeter object are "chainable." That means that each method returns the tweeter object itself, allowing us to call another method without starting a new statement. It looks like this:

However, keep in mind that you cannot assume verifyCredentials() has actually succeeded just because it has returned. As always in Node, it works asynchronously and invokes its callback when it's truly complete (or has truly failed). In our case, we call verifyCredentials() just to log a message to the console if anything is wrong, so there's no harm in getting started with the next thing while that is going on. But you should keep in mind that method chaining is not a way to do asynchronous things in a predictable order. For that, you need to use callbacks and perhaps async.series, as we saw in a previous installment.

We're ready to listen to tweets! Recall that our goal is to respond to incoming tweets by providing a thesaurus service. But how do we listen?

Twitter provides a "stream API" that offers a way to get a continuous flow of updates regarding a particular user. And the ntwitter module wraps this API in pretty much the nicest way possible. Here's how it works:

The first argument to the .stream method is the name of the stream API we want to access. We're interested in a single user's tweets, so we'll specify the "user" API.

The second is an object containing parameters for that API. Here we specify the string we want to search for - in this case, a username - as the "track" property. We'll filter the results to get the tweets we want. There are also ways to specifically receive "replies," which would seem appropriate for catching messages to @synomat, but those properties only return responses from people who follow @synomat. We want to be more inclusive than that so we'll dig through all mentions of synomat and filter them on our own.

Listening to Events

You might think that the callback function for the stream() method would receive tweets. But that's not the case. Instead, it receives a "stream" object on which it can choose to monitor various events. This make sense because incoming tweets are not the only event that can happen to a stream.

In Node, it is common for objects to support the "on" method, which is used to watch a particular named event relating to the object, invoking a callback when the event happens. In our case, we want to monitor the "data" event. Thus this line:

stream.on('data', function (data) {
// Do something with the data
});

The callback of this method receives the actual tweets mentioning synomat - and, curiously, other things as well.

When we use the track property we receive tweets that are sent "to" the username we're tracking. But we also receive other things. The full list includes:

Tweets by the synomat user (rather than to it)

Tweets to them (tweets beginning with "@synomat ")

Tweets mentioning them (tweets with "@synomat" somewhere in the body)

Things that aren't tweets at all, such as arrays of friend ids

This is clearly the API we want. But since only the second case above relates directly to our needs, we'll need to make sure each "data" object we receive is really a tweet, and really a tweet "to" synomat.

We'll begin by ruling out data objects that don't have a "user" property, which does an effective job of filtering out things that aren't tweets at all. Then we'll grab the username of the person who sent the tweet:

if (!data.user)
{
// Not a tweet. For example I've received a list of friend ids
// here for some reason
return;
}
var them = data.user.screen_name;

Now we'll make sure the tweet begins with @synomat. Twitter makes this easy for us with the in_reply_to_screen_name property:

if (data.in_reply_to_screen_name === me) {
// Get excited and do things!
}

Now we can look at the tweet text, which is in the data.text property. We'll check to make sure that it ends with a word, using the regular expression pattern "\w+", which matches one or more word constituent characters (in English, at least), followed by \s* (to tolerate whitespace) and $ (to signify it must come at the end of the tweet). Then we'll convert the word to lowercase.

Notice that the match() method returns an array. Element 0 of that array contains the entire regular expression match. Element 1 contains the portion that matches the first subpattern in parentheses. And this is good, because it lets us capture the word with "(\w+)".

You should always use _.has to test whether an object that you are using to store your own data really has a particular property or not. This avoids confusion with properties that actually belong to all objects, such as the hasOwnProperty method itself.

If the regular expression didn't match, we send back another polite note:

else
{
reply("just tweet me one word and I will tweet back synonyms, analogues, equivalents.");
}

Great! But something is missing. Where is the reply() function?

We'll nest it right here inside the stream.on("data", ...) callback function. That way, the reply() function can see the "me" and "them" variables for extra convenience:

The "updateStatus" method of our tweeter object lets us send any tweet we wish. And its callback function also lets us capture errors if they take place. Since this is just a thesaurus service, we won't do anything dramatic with the errors, apart from logging them to the console. We'll get errors fairly often - because someone asks for the same word twice (Twitter forbids duplicate tweets), because we've hit Twitter's API rate limit, or simply because Twitter is currently displaying the fail whale.

We're nearly through! In fact, the app is complete at this point and works just fine. But there are two more events worth listening for: "end" and "destroy." The "end" event occurs when Twitter closes the stream politely. The "destroy" event occurs when Twitter hangs up not-so-politely. Either way, our response is the same: wait a second, just to let Twitter calm down and regret its actions. Then call Twitter back and demand an apology.

]]>http://justjs.com/posts/synomat-a-twitter-thesaurus-powered-by-node-jssynomat-a-twitter-thesaurus-powered-by-node-js
Folks have asked me how I deploy the justjs blog, as well as the vidmood sample app. Fair question. I've explained how to run node apps on your own computer, using silly URLs like http://localhost:3000/, but http://justjs.com/ and http://vidmood.justjs.com/ require reliable hosting and a way to respond on port 80 (the standard HTTP port) to take that ":3000" business away. Even more important, deployment to a production server requires a solid strategy for pushing code to the production server after you've tested it... rather than hacking on your live site and crashing it as you go.

But why stagecoach?

There are several cloud hosting options out there. Heroku and, yes, Microsoft's Azure are cloud platforms that support Node. And Nodejitsu is the up-and-comer as well as being a major contributor to Node.

But you're not limited to cloud platforms. You can also run Node yourself on any Linux server. The Linux VPS (Virtual Private Server) market is highly competitive, and you can easily run many node apps on a single VPS, so there's an opportunity to save a lot of money here. Hosting your own also allows for the possibility of an Intranet site not accessible by the outside world. And you can also guarantee high performance by running on a dedicated server.

For me, the biggest win is the ability to host many apps on a single VPS I've already paid for. But the second biggest win is performance. You probably chose Node in part for speed. But most Node cloud hosting companies don't provide MongoDB hosting that runs on the same CPU. You can use MongoDB hosting companies like MongoDB, but you can expect some latency due to the network connection between your node app and the MongoDB server. That means a delay in the response of your app. And this is completely avoidable by hosting your own apps on a VPS or dedicated server that also runs MongoDB.

That's why I developed stagecoach, an open source solution for deploying and hosting Node apps, as part of my work at P'unk Avenue. Let's check it out.

Requirements for installing stagecoach

stagecoach requires a Unix server to deploy to. It's been tested a lot with Linux in particular, including Ubuntu (which it loves) and CentOS (which it can get by with).

Ubuntu Linux (10.04 or better) is strongly recommended, because stagecoach comes with Ubuntu-specific scripts to take care of a lot of busywork for you:

Installing node properly, in such a way that it can be updated

Installing npm

Installing MongoDB properly, in such a way that it can be updated

Setting up "upstart" scripts to start and stop all of your apps properly when your server reboots or shuts down.

Also, Ubuntu lets you upgrade from one major release to another. CentOS and Red Hat don't. Yuck.

Yeah, I like Ubuntu. How'd you notice?

However, you can still use stagecoach on other Linux systems, as long as you don't mind taking care of some of the setup tasks yourself. In fact, I often use it on CentOS.

Installing stagecoach

For simplicity I'll assume you're deploying to Ubuntu Linux.

We'll start by setting up stagecoach on our production server (the Ubuntu server). Later we'll install it on our development Mac as well, in order to use the deployment tools.

Let's start by cloning the stagecoach github repository on our production server. If you're not yet familiar with git, you soon will be, as almost all Node projects use git for version control.

First make sure git is actually installed on your Ubuntu server. Open an ssh connection to your production server as root, or log in as a non-root user and use the sudo command to obtain a root shell your server is set up that way:

Once this command succeeds, we can install stagecoach from github. Stagecoach insists on being installed in /opt/stagecoach. Every modern Unix system - be it Ubuntu Linux or MacOS X - has a /opt folder. By convention, this folder is an acceptable place for third-party software.

Here's what we need to do:

cd /opt
git clone git://github.com/punkave/stagecoach.git

This command will copy stagecoach from the official git repository to the /opt/stagecoach folder on your server. If you think you might make interesting changes to stagecoach, you could "fork" the project first on github.com and then clone your fork, which gives you the option of pushing your own changes and sending us pull requests with your thoughtfully designed and tested features. But that's up to you.

Now that stagecoach is on the system, we can take advantage of an included script that installs Node and MongoDB correctly, from repositories maintained by responsible folks associated with each project. They aren't official Ubuntu repositories - they are actually better and more up to date when it comes to these packages. The script also configures MongoDB to listen to connections only from the same server it's running on, which is great since our Node apps will be on the same server. You can adjust that, but you should not run without some security restriction for MongoDB.

The script also installs the extremely useful "forever" utility, which automatically restarts Node apps if they should happen to crash. This script is an important part of how we keep sites running in a stable way with stagecoach.

Here's the command to run:

sc-proxy/install-node-and-mongo-on-ubuntu.bash

If you have trouble with this command, triple-check that you are running Ubuntu Linux (at least 10.04). If you still have trouble, please open an issue on github and we'll talk about it.

Now MongoDB is up and running and waiting for connections, and the "node" command should be available at the command line. You can type "node" and press enter to check. Then press "control-D" to exit the interactive node shell.

With Node and MongoDB ready to go, we could start launching apps right away. But only one app can listen on port 80, which means we can only have one app on the server... or can we? That's where sc-proxy comes in. sc-proxy is a simple node app, included in stagecoach, that listens on port 80 for us and forwards connections to Node and non-Node apps as necessary, allowing us to have many apps on one server, much as one does with Apache virtual hosts. In fact, you can point traffic for Apache-based sites to an Apache server listening on a port other than 80. That's all thanks to a great Node module by the Nodejitsu team called node-http-proxy.

We'll set up sc-proxy soon. First let's create a directory to deploy our apps to, and tell stagecoach what Linux account to run them under (since running everything as root is a silly security risk).

First, take note of thenon-root Linux username you want to run your apps as. I'll use "boutell" in my examples, but you should use your own non-root account that already exists and accepts ssh connections on your server. Indeed, once you've set up stagecoach, you'll never need to do anything as root again, except when stopping and starting the entire stagecoach system.

Now copy /opt/stagecoach/settings.example to /opt/stagecoach/settings and edit the file. Change USER=nodeapps to the user you have in mind, in my case boutell:

USER=boutell

Leave DIR set to /opt/stagecoach (this is not something you can currently change).

Your node apps will be deployed to subdirectories of /opt/stagecoach/apps. Create that folder and use the "chown" command to give it to the appropriate user:

mkdir -p /opt/stagecoach/apps
chown boutell /opt/stagecoach/apps

By default, stagecoach provides hosting for staging sites, and automatically sets them up to respond as subdomains of a shared domain. It's a great strategy for making sites available to your clients before the production domain name is available. But right now our goal is to host an app in production. stagecoach can do this too.

For the sake of example, let's get ready to deploy the "vidmood" app from the previous installment. We want to run it as vidmood.justjs.com. So let's create the /opt/stagecoach/apps/vidmood/data folder, and pop a "hosts" file in that folder containing the hostname we want:

Again we take care to give the folder to the non-root user that will be running the app later.

You can list more than one hostname in that file if you wish (such as example.com and www.example.com). Just list one per line.

The "data" folder for each app is special because it is not overwritten every time we deploy a new version of the app. So both data files (like photos uploaded by users) and server-specific configuration files (like "hosts", "port" and "options.js") should live there. After deployment a symbolic link will make sure this folder is visible as the "data" subdirectory of the folder where the app was launched.

Don't Forget Your App's Local Configuration File

There's one more file that we should pop into the data folder before deploying the app: options.js. This file is specific to the vidmood application we checked out in the previous installment. Recall that the vidmood app reads its settings from data/options.js, and that some of those settings specify Twitter app credentials. To run the app in production, you'll need to register a separate Twitter application. Those credentials don't belong in the data/options.js file in your dev environment. That's why the file is not deployed and is separate on each server.

Use a command-line text editor like "nano" or "vim" to create /opt/stagecoach/apps/vidmood/data/options.js on the production server. Paste in your development options.js settings, then edit them to suit the production environment. The host setting will be different, of course, with no port number. And your Twitter credentials will be different as well.

sc-proxy: our HTTP port 80 traffic cop

Nearly ready to deploy! We still need to set up "sc-proxy," the node app that acts as a traffic cop (aka "proxy server") for all of our node apps, allowing them to share port 80 harmoniously. In the "sc-proxy" subdirectory, you'll need to copy the provided config-example.js file to config.js:

cd sc-proxy
cp config-example.js config.js

Now edit the config.js file. If you are new to using the shell prompt to edit things over an ssh connection, you can use your favorite sftp program to copy it to your machine first, edit it there, and push it back. Or try:

nano config.js

nano is a user-friendly command line editor that should be fairly easy for Mac users to pick up. Just remember to use that "control" key you're used to ignoring.

Here's a reasonable config.js file. PLEASE NOTE: listening on port 80 means that Apache CANNOT listen to port 80 at the same time! You'll need to tweak your Apache configuration if you want both on the same server, as explained below. You could also bind on separate IP addresses if you have them.

module.exports = {
// Subdomains of this are used for apps that don't have a hosts file
domain: 'mystagingdomain.com',
port: 80,
// Listen on all IP addresses
bindIp: '0.0.0.0',
// Do not change this
appsDir: '/opt/stagecoach/apps',
// If the request doesn't match any app or hosts file, forward to
// this port (usually Apache)
defaultPort: 9898
};

Running Apache At The Same Time

Those who are not running Apache on the same server can skip the "defaultPort" setting entirely. Those who are will need to change:

Listen 80

To:

Listen 9898

And:

NameVirtualHost *:80

To:

NameVirtualHost *:9898

In their Apache configuration. You'll also need to restart Apache before starting up sc-proxy for the first time, so that Apache "lets go" of port 80.

Enough about Apache - let's turn our attention back to Node! In the sc-proxy folder, we're ready to install the npm packages for the sc-proxy app. Thanks to the package.json file this is a snap:

npm install

We could launch sc-proxy manually at this point and put it in the background:

sudo node server.js &

And then start all of the apps, if we had deployed any yet (we do NOT run this command as root, we run it as the non-root user we chose for the USER setting):

./sc-start-all

But since we're running Ubuntu, there's a better way. We can install an Upstart script that allows Ubuntu to automatically start sc-proxy and our apps every time the server boots up, and also makes it easy for us to manually start and stop the whole shebang. In fact, Upstart scripts are what Ubuntu uses to start and stop pretty much everything (at least in Ubuntu 10.04 and newer).

Just copy upstart/stagecoach.conf to the /etc/init folder:

cp upstart/stagecoach.conf /etc/init

Now you can just type:

start stagecoach

To launch sc-proxy and any apps that have been deployed. Of course, we haven't deployed any apps yet! So let's take care of that important detail.

Deploying apps with sc-deploy

stagecoach includes a simple deployment script called sc-deploy. sc-deploy uses the rsync utility to deploy the site in such a way that the cutover from one version of the site to the next is almost instantaneous. That's done by copying the latest code to a new, date-stamped subdirectory, stopping the old version, changing a symbolic link from /opt/stagecoach/apps/vidmood/current to point to the new subdirectory, adding a symbolic link from the new subdirectory to the shared "data" folder, and starting up the new version of the app.

You don't have to sweat those details, though. You just have to install stagecoach on your Mac.

Start by installing stagecoach in /opt, just as you did on the production server. Enter your password when prompted by "sudo" (remember, we are NOT using ssh anymore, we are entering local commands in the Terminal app):

sudo git clone git://github.com/punkave/stagecoach.git

Now make a symbolic link from your /usr/local/bin folder to /opt/stagecoach/bin/sc-deploy so that the terminal can find the sc-deploy command in your PATH:

sudo ln -s /opt/stagecoach/bin/sc-deploy /usr/local/bin/sc-deploy

Just one more step and we can deploy the app! The sc-deploy command is designed to be flexible - it runs "stop," "migrate," and "start" scripts that it expects to find in the "deployment" subdirectory of your app. If you peek in the "deployment" subdirectory of "examples/vidmood-1" in the justjs github repository, you'll see these simple bash scripts. Here's the good news: you can copy them, and you don't have to change them at all for new sites, unless they have special needs. So we won't study them in detail. Their main job is to ensure that each app gets a separate port number to listen to and that the "data" symbolic link is established for each new version deployed. If you have additional migration tasks to carry out when a new version is deployed you can add them to the "migrate" script.

You'll also need a "settings" file. Again, you can copy mine; just be sure to change the PROJECT setting to the name of your app. Also, if your host requires you to make your ssh connections at a nonstandard port number, change SSH_PORT appropriately.

USER is the Unix user the ssh connection should be made under (usually the same as the USER setting in /opt/stagecoach/settings on the production server). SERVER is, of course, the hostname of your production server. Create this file with appropriate settings for your server and account.

You can also have a "settings.staging" file if you have a separate staging server. A great idea if you need to demonstrate the latest modifications to a client project without pushing them straight to production.

Finally, you'll want an rsync_exclude.txt file. This file tells sc-deploy what files not to deploy. Its function is very similar to .gitignore. Here are the contents of rsync_exclude.txt for vidmood. The data folder, of course, shouldn't be copied to the server because that would crush production data files as well as the data/options.js and data/hosts files. .git and .gitignore are skipped because you don't need to use git commands on production and your git history database can be quite large.

data
.git
.gitignore

If your "deployment" folder is set up properly, it should contain the following files:

migrate
settings
start
rsync_exclude.txt
settings.production
stop

Now we're ready to deploy the app! Just use the sc-deploy command, like this (NOT as root). Be patient, it takes a little while to copy files over the internet tubes sometimes:

sc-deploy production

(If the sc-deploy command is not found, make sure you have a symbolic link to it from /usr/local/bin and that /usr/local/bin is part of your PATH setting.)

When stagecoach finishes deploying the app, it should become accessible right away. If not, make sure you installed the Upstart script for stagecoach and ran:

start stagecoach

On the production server.

"What if I want to deploy a new version of my app?" Just run:

sc-deploy production

Each time you finish testing a new set of changes. Remember, your app is only down for a split second at the very end of the process when the symbolic link changes over and the app is launched again.

"What happens to the old deployment directories when I deploy new versions?"

They are still there, in /opt/stagecoach/apps/appname/deployments. You can manually purge the older folders here if you wish (not the newest one, of course). There's an open issue in github for me to write a convenient tool for purging old deployments now and then.

"Can I revert to an old deployment?"

Yes, but at the moment this is manual. Note the name of the deployment folder you want to roll back to. Then, on the production server, do this (not as root):

Again, there's a github issue open for me to write a handy rollback script. Perhaps you'll contribute it as a pull request. That would be nice.

Make sure your app respects "data/port"!

The vidmood app was written with support for stagecoach in mind. Your app probably wasn't. What do you need to change?

Just one thing: your app needs to look for a file called data/port, and if it's there, read its port number from that file, so that sc-proxy can figure out where to send traffic for the appropriate host. If data/port doesn't exist, feel free to listen on a default port for convenience on your development Mac.

We made it to the mountaintop!

Yes, that took some doing. But look what we have: a way to deploy as many apps as you can stuff into your server. Once stagecoach is in place, each new app is just a matter of copying the deployment folder, editing two settings files and running "sc-deploy production" for the first time. All of your apps can share port 80, and you can even run Apache at the same time with a little fiddling. And your mongodb connection is pretty much instantaneous because it's running on the same machine. Isn't the future outstanding? Sure, for hugely popular apps you may need more, but this is a great way to get there, and at minimal cost for those of us who run lots of little apps.

Nevertheless, in an upcoming installment I'll talk about how to pull off the same trick with at least one of the cloud hosting options, like Heroku and Nodejitsu. Not everyone wants to be an Ubuntu Linux administrator. Just cool people with lots of friends and toys and pet dinosaurs.

]]>http://justjs.com/posts/deploying-node-js-and-mongodb-in-production-with-stagecoachdeploying-node-js-and-mongodb-in-production-with-stagecoach
In the previous installment, we checked out vidmood, a JavaScript-powered social web app contained in a single HTML file. Of course, that's a little bit of a lie. To be social, the app needs to share data with other users. And it also needs to find appropriate YouTube videos to match your mood, which requires talking to the YouTube API. JavaScript in the browser can't do either of those things on its own. We need a backend server to talk to. And that's where Node comes in.

Unlike previous projects, this time I've kept the server code in a single server.js file, just to demonstrate how manageable that can be for simple apps. Especially when we take advantage of npm modules like Express and Mongoose rather than reinventing wheels.

Keep Your Development Chocolate Out Of My Production Peanut Butter

Enough prologue! Let's check out server.js. We begin by loading our options from a separate .js file (hey, it doesn't count if it only contains settings):

var options = require('./data/options.js');

Notice that I put options.js in a "data/" subdirectory. My preferred style is to keep everything that is server-specific and should not be crushed by every new deployment in a folder called "data." Since options.js for my Mac has different settings than the version on my Linux VPS production server, it makes sence to keep this file in the data folder.

"What's all that Twitter stuff?" vidmood uses Twitter authentication to log the user in before they can share their mood. We've seen the Passport authentication module before; in a previous installment we used it with gmail accounts. This time we'll use it with Twitter.

Of course, you need to get your own Twitter consumer key and consumer secret. You can get these by registering your own app on dev.twitter.com. You should be able to use the same callback URL I've used here. This is the URL that Twitter will bring users back to in order to complete the signin process.

"How about the session secret?" This makes it tough to steal user sessions. Use a suitably random string of words.

"What about the host setting?" We're used to testing our apps by visiting:

http://localhost:3000/

And it's tough to stop typing that. But because of Twitter authentication, we really need a distinct hostname instead of localhost.

"Yeah, but my computer isn't vidmood-dev.justjs.com." No, but you can make your computer think it is. Just edit your /etc/hosts file and add this line at the end:

127.0.0.1 vidmood-dev.justjs.com

"How about the db setting?" You should be able to leave that alone, as long as you're running MongoDB on your own Mac as I've recommended for testing purposes (it's not that uncommon in production either). But you can change the host name and database name if you need to. This fancy "mongodb://" URL is understood by Mongoose, a handy ODM (Object-Document Mapper) that makes working with MongoDB a little more convenient, as we'll see in a moment.

Let's turn our attention back to server.js. Now that we've set our options, we're ready to require the usual suspects... I mean modules:

These are old friends of ours. Express gives us easy routes, among other features. Underscore makes coding in JavaScript generally pleasant. Passport provides authentication. And fs (which is standard and does not have to be installed) allows us to manipulate files.

We've used MongoDB directly before. Mongoose is a wrapper for MongoDB that reduces the amount of work involved in working with MongoDB. Mongoose lets you write more natural object-oriented code based on MongoDB. We'll check out the small differences between working with Mongoose and working directly with MongoDB.

We'll create the Express app object in the usual way, then add middleware to make sure we are always on the "canonical host" (the host setting in our "options" object):

By checking req.headers.host we can see what URL the user was trying to access. If it doesn't match the host we had in mind (note that the "host" property also includes the port number), we redirect the user to the correct host, plus the rest of the URL from the request (req.url is a bit misleadingly named; it ought to be req.path, really.)

If the hosts do match, we just call next() to continue the chain of middleware. Without this call the normal handling of the request would never happen.

We'll fire up support for serving static files from a "/static" subdirectory in the usual way:

app.use('/static', express.static(__dirname + '/static'));

And we'll add the standard middleware for accessing the parameters of POST form submissions without a fuss, so that we can write"req.body.mood" in our routes:

app.use(express.bodyParser());

Next we'll configure passport to give us Twitter login support. That's a nontrivial chunk of code, so we'll package it in a function:

We'll start by requiring the passport-twitter module and registering it with Passport. We pass in our Twitter options (the consumer key and secret for our app, from dev.twitter.com) and a function that converts a Twitter profile to an object representing the user:

We're ready to set up the route that actually sends people to Twitter for authentication. There's a Passport function that takes care of it; we just have to register it as the callback function for /auth/twitter.

app.get('/auth/twitter', passport.authenticate('twitter'));

When the user comes back, they'll arrive at the Twitter callback URL we set when we registered our Twitter app, which is /auth/twitter/callback. So we'll need a route at that URL that finishes the authentication process and sends the user to an appropriate place on success or failure. passport.authenticate() creates a suitable callback function for us. For now we'll send the user to the home page regardless.

Keeping Your Data Clean With Mongoose

Now that we can log users in, we're ready to access and store moods for them. For that, we need MongoDB. And we've chosen to use the Mongoose ODM to make working with MongoDB a little nicer and cleaner.

Let's start by connecting to MongoDB. Mongoose URLs are a handy way to do this:

mongoose.connect(options.db.url);

Next we'll create a schema for our mood objects.

"WAIT JUST A MINUTE HERE! Isn't MongoDB a schema-free database?" That's true, MongoDB doesn't force a schema on us. But when the format of your data is clearly known in advance, a schema is a helpful way to make sure you always include what needs to be included and to check that all of your properties are of the expected type. The schema can also be extended with "setter" methods that change the way data is stored without the need to change your code in every place you use it. Mongoose adds the concept of a schema to get these benefits.

For convenience, let's grab references to the Schema and ObjectId types from the Mongoose module:

We've defined all of the properties that should exist for a mood object: the twitter username, the twitter user ID (which is unique), the "name" of the mood (like "happy" or "sad" or "verklempt"), the YouTube video ID of the video we'll associate with the mood, the URL to the YouTube thumbnail image and the "date" (date and time, in a JavaScript Date object) when the mood was shared. Mongoose will automatically let us know if we screw up the type of any of the properties, which helped me debug the app quickly.

Some of the properties are followed by an object containing both a "type" attribute and other attributes, such as "index: true" to ensure we can quickly sort moods in chronological order. Others just have a type alone. If you don't need any other attributes, Mongoose lets you take a shortcut by just specifying the type.

We're almost ready to use Mongoose. First we need to convert our Schema to a Model. A model is a constructor for new mood objects. It's also an object we can use to query the database.

We need to pass the name of the model as the first argument to mongoose.model, then the schema object:

var Mood = mongoose.model('Mood', MoodSchema);

Fetching Feeds with Mongoose

Now we can rock the house! Let's skip a head a little in server.js to see how Mongoose powers the "/feed" route:

What's happening here? We're setting up a criteria object to pass to Mongoose. If the browser-side app passes a "since" parameter in the query string, we are only interested in moods shared after that time, so we use the $gt (greater than) operator. The updateFeed() function we looked at in the previous installment takes advantage of this. Similarly, if the browser passes a "before" parameter, we're only interested in moods prior to that time, which we can match with the $lt (less than) operator.

If neither parameter is present, we leave the criteria object empty so that it matches all moods.

This syntax may seem a little strange, but it is very fast because Mongoose doesn't have to parse yet another language that is not JavaScript(i.e. SQL). Also keep in mind that it is very flexible: we can combine criteria, using combinations of different operators like $gt and $lt as well as simple equality tests. For instance, if we only wanted moods for a particular twitter id, we could add that easily:

criteria.twitterId = req.user.id;

"But what's with $gt and $lt? Why not just '>' and '<' or something?" I'd like to know that too. It's just one of MongoDB's quirks (Mongoose didn't invent $gt and $lt). Perhaps it runs a little faster than comparing strings. Unless strings are folded in the v8 JavaScript environment. I'll have to read up on that sometime... OK, OK, more than you wanted to know!

Now we're ready to find the moods we want - in the order we want. Mongoose's find() method is chainable: it returns a query object that can be modified by calling additional methods on the returned value, until finally we call exec() to get the results. This is one of those times when the code may be simpler than the explanation:

The sort() method adds a sort order to the query; we can specify any property along with an order ("1" for ascending, "-1" for descending). The limit() method limits us to returning the 12 most recent moods, just to avoid overwhelming the server if justjs.com winds up on slashdot while I'm sleeping; I'll revisit this choice at some point. And the exec() method takes a callback that receives the actual mood objects.

Speaking of callbacks, what do we do with the moods once we have them? We hand them to the browser in handy JSON format, of course:

Congratulations, you just implemented your first JSON API! Our JavaScript application in the browser will be able to phone up this API with jQuery's $.getJSON method in order to fetch a feed of the latest moods. No need for us to worry about how to render a mood in the browser here in the server code. All we have to do is deliver the data on time. That's an excellent way to separate concerns, and separation of concerns leads to more easily debugged and maintained code.

Posting Moods and Searching YouTube: It's Easy With jsonc

Now we can fetch moods, but how do we post moods to the site in the first place? Let's check out the "/mood" route.

The first thing we do is a quick sanity check. Did the browser submit a mood? And is the user logged in? It's the browser's responsibility to redirect to the /auth/twitter route to take care of that before trying to use this API.

All we need to do is fetch the feed with the request module, which consists of one simple and super-useful function for fetching URLs, then parse the JSON response with JSON.parse() and peek at the properties of the object we get back to locate the YouTube ID and thumbnail URL we want.

The request() function takes two parameters: an object specifying what we want to fetch, and a callback function to be invoked with the content that was fetched.

Here's the object we'll pass as the first parameter to fetch the feed. The url property is YouTube's public API for searching for videos. The qs property ("query string") contains the properties we'd like to pass as part of the query string of the URL (the part after the ?). request() is nice enough to take care of that for us:

Three of the query string properties, "q", "v", and "alt," are well-documented. The third, "format," is less well-documented but very handy. In particular, "format: 5" returns only videos that can be embedded on non-YouTube sides like vidmood. That saves us from displaying a lot of unhappy video players. The use of "jsonc" rather than "json" for the "alt" parameter gives us the friendly and useful feed rather than the verbose one.

By the way, you can try the resulting URL out in the browser directly. Just visit:

You'll get back a JSON response with lots and LOTS of information, but it's not formatted in a particularly human-friendly way. jsonprettyprint.com offers a convenient way to view a JSON response with attractive spacing. Just paste the response you got from YouTube into the form on that site.

Let's move on to the second argument to request(), which is our callback function:

request() passes us all the information we might need: the error (if any), a response object (from which we can read the statusCode and contentType if we wish), and of course the body of the response. If we fetched a webpage with request(), the body parameter would contain the HTML for the page. For a JSON API call, it contains the JSON-encoded data we asked for.

Our callback function starts by making sure there was no error at the HTTP level - in other words, it makes sure we got a response from the YouTube server at all - and that the status code was 200 (the HTTP status code for "OK"). Then we use JSON.parse to convert the JSON string to a JavaScript object so we can rifle through its properties.

Before we start examining the results, we'll carry out two more sanity checks: did YouTube report any errors at the API level? And did it return a 'data' property? YouTube's "jsonc" responses always contain an 'error' property if something goes wrong, such as a badly formatted request, and always coantain a 'data' property if things go right. If we hit the video search API too often over a short period of time, YouTube will return an error property as part of their rate limit policy. We should recognize this condition and return a 500 error to the browser, rather than crashing in an attempt to look at properties that aren't there. (If vidmood were wildly popular, we'd want to cache respones from YouTube for popular moods like happy and sad, so that it wouldn't be necessary to hit the API quite so often.)

Now we're ready to pick a video. The returned feed actually contains up to 25 results (the first page of a much larger pool of possible results, but we don't need the rest). Since it would be boring if everyone got the #1 video matching a search for "happy," we'll pick one of the results at random. First we'll double-check that we received at least one video in the results:

You're probably familiar with Math.random(), which returns a random fractional number between 0 and 1 (but never exactly 1). By multiplying that number by the total number of entries and then calling Math.floor() to round down, we obtain a number between 0 and (count - 1), inclusive. This is a very common JavaScript snippet. If we used it again, it'd be worth tucking it into a dice_roll() function.

Our next task is to pull out the YouTube ID in the response. Thanks to "jsonc" this is very easy:

var id = entries[i].id;

Now we've got the id string. All we need to finish the job is the thumbnail URL. (Recall that our browser-side JavaScript app uses the thumbnail rather than an actual player until the user actually clicks the "play" button.)

Every YouTube video has at least one thumbnail. Fetching the thumbnail is almost as easy:

var thumbnail = entries[i].thumbnail.sqDefault;

Pretty straightforward. But what's this "sqDefault" business? YouTube returns two thumbnail URLs, sqDefault and hqDefault. As near as I can tell, sq is "standard quality" and hq is "high quality." My educated guess is that all videos are guaranteed to have "standard quality" thumbnails available, but "high quality" might depend on the quality of the original. So let's use the "standard quality" thumbnail as a common denominator. (I did search for a definition of these properties, both with regular Google search and in YouTube's API documentation. Dear reader, please let me know if you find an official explanation.)

In a previous version of this article I also pulled out the width and height of the thumbnail. Those properties are not part of the "jsonc" feed, but as it turns out we never used them in the browser-side app, so I've removed them from the schema. If you really want them, you can parse your way through the "json" feed instead.

(We save both the Twitter username and the Twitter id. Saving the username is convenient for display, while saving the id is guaranteed to find that user's moods again someday if they change their username. We can initialize the date property to the current date and time simply by creating a new Date object with no parameters.)

Great, let's save it to the database and give the browser the good news (assuming things go well):

The Mongoose save() method takes a callback to be invoked as soon as the object has been saved. (Actually, since we didn't specify the strict option, MongoDB assumes the save will work out fine and returns right away for a small performance boost.)

The first argument, as usual for callbacks, receives any error that may have occurred. The second receives the MongoDB object that was saved. So we'll call JSON.stringify(mood) to turn that into a JSON string, then send it to the browser.

Mischief managed! We can save moods to the MongoDB database, with suitable YouTube videos attached, and let the browser know the outcome.

It's all over but the shouting. But there's one more important route: the browser-side JavaScript application itself! We do have to deliver the one page that makes up our one-page app at some point, you know.

We took a look at this code in the previous installment, so a quick recap will do:

Our "/" route delivers the home page, aka the only page, aka the single-page JavaScript application. That page is delivered from the file /templates/app.html. We fetch it from the filesystem with the fs.readFileSync() method - a rare case of not doing something asynchronously in Node, but since we only fetch it the first time it is asked for, there's no performance impact.

Once we've fetched the content, we set the appropriate content type and send the HTML... after first replacing the placeholder "%USER%" with a JSON object containing information about the currently logged-in user. Logging the user in requires redirecting them off to Twitter-land, so when they come back from Twitter and load the app again it's a good opportunity to just stuff that data right into the JavaScript code, as in this snippet from app.html:

var user = %USER%;

It's not uncommon to take this sort of thing further. We could pass in the initial feed in the same way rather than using a separate HTTP request. In production, once the load started going up and the system administrators started getting nervous, we probably would (although at the speed of Node, it might take a while for anyone to become overly concerned). For now it's smarter to keep things simple and understandable and fetch the feed in one consistent way.

What's left? Just listening on the appropriate port! As usual we listen on port 3000 by default. However, we'll also check for a data/port file. In production, that file is created by stagecoach, a handy open-source tool we built at P'unk Avenue. Both at work and for justjs.com, I use Stagecoach to run many Node apps on a single Linux VPS or dedicated server. Stagecoach assigns ports to each one automatically and allows each app to respond under a different hostname without a clunky port number in the URL. That's accomplished by a proxy server based on Nodejitsu's great node-http-proxy module. stagecoach also provides deployment scripts and optional Ubuntu "upstart" scripts to start up and shut down the entire collection of apps automatically. We'll check out stagecoach in detail in a future installment.

Atomic Turbines to Speed!

That's it - we've covered both the browser side and the server side of the vidmood app! You can launch it just as you launched the blog examples in previous installments. Make sure you added vidmood-dev.justjs.com to your /etc/hosts file so that you can visit http://vidmood-dev.justjs.com:3000/ rather than http://localhost:3000/. That's especially important when logging in with Twitter to share your mood. Also make sure you registered a Twitter application for your tests as described above.

"Yeah, but how do these apps ever make it to a production server?" That's a good question. So next up I'll take a look at deploying Node applications with stagecoach. And then we'll examine the benefits of rewriting the browser-side vidmood application in Bootstrap. Or maybe AngularJS. It depends on my mood.

]]>http://justjs.com/posts/vidmood-part-ii-behind-every-great-browser-app-there-s-a-great-node-servicevidmood-part-ii-behind-every-great-browser-app-there-s-a-great-node-service
Earlier installments of justjs focused on building a traditional model-view-controller web application, in which the server renders HTML pages via ejs templates and sends the finished product to the browser. JavaScript was sprinkled in, here and there, but essentially each page was static.

Many web applications take things a step further by empowering the webpage to refresh part of itself, or to submit a form in such a way that it only refreshes the part of the page that needs to change. But the server is still responsible for assembling the page and returning fragments of HTML that replace parts of it. This approach was popularized by Ruby on Rails, which provided "helper" functions to dynamically update pages in a few standardized ways, without asking the programmer to actually learn JavaScript. But this approach is limited, and based on outdated assumptions about the limitations of web browsers. These days, we can finally count on a decent degree of smarts in a web browser, and call upon it to do more interesting things.

New Wave Web Apps: it's all in the browser (almost)

That brings us to the new wave approach: where the webpage becomes the web application. Rather than asking the server to generate many different pages, we send a single page which contains all - well, almost all - of the application in the form of JavaScript code.

What does that "almost all" refer to? The application still needs to be able to retrieve, save and share data. And in most cases that data needs to be shared with other users. So storing it entirely in the web browser doesn't really help much.

In a nutshell, the model layer still needs to "live" on the webserver. But since the web app in the browser is smart, our node code can be drastically simplified. Rather than rendering page templates, all node.js needs to do is give an appropriate JSON response. This is easy to code, easy to test, and easy to reuse in other applications.

But there's one more thing the webserver still needs to do: decide who to trust. You can code as much validation as you want into your browser-side JavaScript, making sure everything the user inputs looks safe and legal. But the webserver has no way of knowing whether it's talking to your original JavaScript or a five-line shell script that uses the curl command to send nasty commands to take over the world. So while your node code can be simplified, it still needs to concern itself with who can do what.

That's enough theory. Let's talk about a fun new app, and examine how to build it.

vidmood: social video sharing for moody people

vidmood is an entertaining and social web app, just the sort of thing people like to play with these days. The idea is simple: you tell vidmood how you're feeling, and vidmood pulls up an appropriate YouTube video and adds that to a feed that everyone can see. In order to learn your name, vidmood asks you to log in via Twitter. Later, we'll teach it to tweet out your video moods as well, for an extra kick of viral marketing.

vidmood is split into just two important files: server.js and app.html. server.js contains the node code that responds to requests like "please send me a feed of recent video moods," while app.html is the JavaScript-powered webpage that makes those requests. We'll start with app.html because it feels more natural to start with what the user is doing and seeing. In the next installment we'll move on to server.js.

The skeleton: looking at the markup

app.html begins with the usual Twitter bootstrap stuff and a style element to fix the appearnce of a few things. Then we come to the form used to share your mood:

This looks like an ordinary HTML form, the kind that causes an entirely new page to load. But we'll change that with JavaScript as you'll soon see.

Notice that there's a "spinner" GIF present in the form. The spinner is initially hidden by CSS properties in the style element. We'll reveal it with JavaScript at the right time.

Below the form, you'll find a div with the "feed" class, just waiting to be stuffed with moods:

<div class="span12 feed">
</div>

Great, but where's the content? This is where JavaScript comes in. The initial structure of the page is just a vessel waiting to be filled. We'll use AJAX requests (Asynchronous Javascript And hardly ever XML) to get the data we need. Then we'll render it using underscore templates, which are compatible with the EJS templates we've seen previously. They just happen to come with underscore "out of the box," which means our app is up and running faster. They aren't as convenient to debug as EJS templates, though. If that bothers you, you can use EJS in the browser as well.

Naturally, There's JavaScript Inside

Let's check out the script tag that drives this puppy. The first line is special:

var user = %USER%;

"What's this %USER% business?" For the most part our application will phone up the server and ask for any information it needs. But we can also pass information as part of the page when we first transmit it to the server. Since we're using Twitter logins for this site, any change in the identity of the user always involves loading a new page. So when we transmit the browser-side application code, we'll replace this %USER% placeholder with the user's identity. Here's what it looks like in server.js:

"Why didn't you use EJS or underscore on the server too?" Because app.html also contains Underscore templates, which the server would try to interpret. There are ways around that: it's possible to specify different start and end tags for Underscore. But since all we really want to do here is pass some JSON data directly in the page, I found it simpler to just use replace().

We'll look at the rest of server.js in the next installment. Let's return our attention to app.html, in browser land:

Here we establish global variables to hold compiled Underscore templates for our mood video players. Then we use jQuery to reach right into the page and fetch the markup for those templates, and pass it to _.template to compile it.

Notice that we don't try to do anything with the DOM (the Document Object Model) of the browser until we're inside this block:

$(function() {
...
});

This is a convenient feature of jQuery: pass a callback to the "$" function and it won't be called until the DOM is ready - that is, until the page has loaded sufficiently for jQuery to be sure of "seeing" all the elements in it (although it doesn't wait for really slow things, like loading images).

"OK, you've got HTML markup inside a script tag. I'm lost." Ah, but take a closer look at that script tag! Where you're used to seeing:

type="text/javascript"

This tag says:

type="text/template"

What does that do for us? It turns out that all web browsers sensibly ignore a script tag with a type they don't recognize. If you specify type and don't set it to text/javascript, the browser will leave your script tag alone - but you can still access its markup later via jQuery's .html() method. So we give these script blocks ids and fetch them with jQuery in the usual way to get their source code. Flawless victory!

This particular template displays a player for a video mood - or rather, a placeholder for a player, displaying a thumbnail image until the user clicks the "play" button. That avoids the performance hit of loading dozens of YouTube videos on the same page - something you don't want, trust me, especially in older browsers.

"What's up with that little play button arrow?" It's a Unicode black right-pointing triangle character, standard in web fonts like Helvetica, rather than an image. I pump it up and add a border with a bit of CSS in the style tag, just because I can. Feel free to use an image element instead.

There's a second Underscore template for live players - video players that the user has actually interacted with. Whenever the user clicks "play," that particular player switches to the live version. We'll see how in a moment:

Notice that the template inserts a YouTube ID. Similarly, the previous template inserted a thumbnail URL. Where does that information come from? We'll find out when we check out server.js. Here in app.html all we need to know is how to ask the server for the resources we need.

Let's return to the web application JavaScript code:

if (!user)
{
$('.login-required').show();
}

Here we display a prompt explaining that the application will ask you to log into Twitter before sharing your mood for the first time. Earlier we saw how the user object is inserted into the script block when the page is transmitted.

Scheduling Feed Updates

The updateFeedSoon() and updateFeed() functions take care of retrieving the latest moods from the server at regular intervals, and also on demand when the user does something that ought to feel instantaneous, like loading the page for the first time or sharing a new mood. It's time to make that first call to updateFeed():

updateFeed();

This call kicks off the first update of the feed. We only call this function directly once. Anytime we want an update in the future, we'll call updateFeedSoon() to bump the schedule for the next update. If we called updateFeed directly a lot, we'd eventually get two copies of the same mood, which is known as a race condition. Redundant mood feeds are very emo.

It's time to look at the updateFeedSoon() function, as well as the global variables that keep track of whether a timeout has already been set for the next update:

What's happening here? You are probably familiar with JavaScript's setTimeout() function. setTimeout() schedules another function to be called after a certain amount of time. If you specify zero milliseconds, what happens is that the function is called immediately after the browser takes a deep breath to allow the user to interact with the page.

But the updateFeedSoon() function can't assume that we haven't scheduled an update already. Indeed, we normally update every five seconds. Instead it checks for an existing timeout and clears it before scheduling the next update to happen as soon as possible.

What if feedUpdateTimeout is null? The updateFeed() function sets that variable back to null each time it starts fetching an update. So a value of null means an update is already in progress. We don't want two updates to happen at once. That's why, in the else {} clause, we set the updateRequested flag. The updateFeed() function will check that flag later to determine that another update is desired as soon as possible.

That leads us to the updateFeed() function that does the actual work of talking to the node server:

Here we use jQuery to fetch a reference to the div with the feed class. Then we build up an array of parameters to send to the /feed route of the node application. Specifically, if we have already loaded the feed at least once, we send the timestamp of the most recent mood we've already seen. That allows the node application to send us just the new content when we call jQuery.getJSON.

jQuery's getJSON method is a convenient way to request a URL and return the JSON-encoded results as ready-to-use data. The first parameter is the URL, the second parameter contains parameters to be stuffed into the query string (exactly like any form submitted via the GET method), and the third is a callback function that will be called with the data when it's ready.

So what does that callback function do when the response arrives from the server?

First we take note of the most recent mood we've received, so that the next update doesn't contain redundant entries:

if (data.length)
{
lastMood = data[0].date;
}

Then we're ready to loop over the moods we received from the server, rendering an Underscore template to display a video player for each new mood. In a moment we'll look at the playerForMood function that takes care of the Underscore template rendering:

"Hang on. Why are you reversing the order of the array and then prepending each player to the feed div? Isn't that the same as appending them in the original order?" It is at first. But after the first update, you'll have lots of moods in the feed div already, and you don't want to append new updates at the end. You want them at the beginning, but in their proper order. Prepending the new players in reverse order gives us the order we want every time.

Creating Players With Closures

So what does the playerForMood function do exactly? The first part is simple:

var player = $(playerTemplate({ mood: mood }));

Rendering an underscore tempalte is pretty easy, as I mentioned earlier. We've already compiled them, so we can just pass the mood object to the template function and pass the result to jQuery's $ function. When you pass HTML markup to the $ function, you get back a jQuery object that's ready to insert into the document, as we saw above:

feedDiv.prepend(playerForMood(mood));

But before we append the player, we need to teach it a trick: how to respond to mouse clicks, replacing the thumbnail and play button with an actual YouTube player. Here's the code:

Notice that we can add event handlers to our brand new player even before it is added to the feed div. And since this code is nested inside the playerForMood function, we have access to the mood variable. In computer science terms, we've created a closure in which this variable is still available even though the playerForMood function has already returned.

In jQuery event handlers, "this" always refers to the DOM element that responded to the event. But it's not wrapped in a jQuery object yet. So we do that here:

var thumbnail = $(this);

Now we can invoke the Underscore template that generates a live player:

var livePlayer = $(livePlayerTemplate({ mood: mood }));

(Recall that we looked at the source code for the template itself earlier.)

Next we replace the thumbnail with the live player:

thumbnail.replaceWith(livePlayer);

Finally our jQuery event handler returns false, which is customary when you've done everything that needs doing and don't want the browser to take any default action.

The playerForMood function wraps things up by returning the player:

return player;

After prepending each player, the updateFeed function has one more task: scheduling the next update via the setTimeout function. A 5000 millisecond timeout gives us reasonably frequent updates without swamping the server. The updateFeed function simply schedules itself to be run again. If the updateRequested flag was set by updateFeedSoon(), we schedule another update right away, otherwise we wait 5 seconds (5000 milliseconds):

So far, so great! We have a feed of moods that updates itself without refreshing the page, and it's all happening through simple API requests to the node server.

But what about posting new moods? Empty feeds aren't very interesting. We saw a form in the markup, but we haven't yet looked at the JavaScript that submits the form. Let's check out that code.

Posting a new mood to the server can turn out one of two ways. If the user is already logged in, it's simple: we use $.post to send the data, then we invoke updateFeedSoon() to update the feed as soon as possible so that it feels immediate.

Logging In With Twitter

If the user isn't logged in yet, we can't post their mood right away. We need to redirect the user away from the JavaScript application, to the /auth/twitter route, which takes care of sending the user to Twitter to log in. We'll examine that code in server.js in the next installment. But first, we save the mood they are submitting in a cookie, using the jquery.cookie plugin, so that we can try the form submission again later when Twitter returns the user to the page:

$.post works a lot like $.getJSON. The first argument is the URL we're POSTing data to. The second is the data we want to POST. jQuery's serialize() function examines all of the form elements inside the form and extracts them into a nice JavaScript object with properties and values, ready to pass to the $.post function.

The third argument is a callback function that gets called with the JSON response from the server. Right now, we don't look at that response directly; we just request an update of the feed as soon as convenient, and stop the progress spinner. The fourth argument tells jQuery that we want the response to be parsed as JSON data. (There is no $.postJSON shorthand function, but adding an extra argument isn't much of a burden.)

The complete form submission event handler for the mood form looks like this:

We start out by calling startSpinner(), process the form submission or nudge the user to log in, and finally return false - especially important in a form submission event handler, because otherwise the form will be submitted normally and the page will refresh completely. Not what we want. We're much happier updating just the feed div, and only with the latest data from the server.

Saving the Day: Posting a Mood from a Cookie

One more detail: when the user tries to share their mood but hasn't logged in yet, we stash the mood in a cookie and then redirect to Twitter authentication. But for this to be effective we have to do something with that cookie when the user comes back. Here's the logic we need:

Our strategy is simple: if there's a mood cookie, stuff it into the mood element of the form, then submit the form. This way we can reuse the submit handler we already have rather than duplicating effort. Of course we also have to clear the cookie so it doesn't happen over and over.

We're just about finished! startSpinner() and stopSpinner() are worth a quick peek. These functions swap the submit button for the animated spinner GIF and vice versa:

When Next We Meet

We've seen how to build a JavaScript web application in a single HTML file. And that's very cool. But what about the node server that manipulates all those moods for us? And what about the YouTube API calls that search for appropriate videos? In my next installment I'll explore the server.js code that implements the back end, aka the model layer, of the vidmood application.

Soon after, we'll check out how to deploy the application to a real server - just in case you're tired of sharing moods with yourself.

After that, I suspect we'll refactor the front end to use Backbone. Or AngularJS. Or whichever front end MVC framework is looking especially great that day.

]]>http://justjs.com/posts/vidmood-part-i-when-the-web-page-becomes-the-web-appvidmood-part-i-when-the-web-page-becomes-the-web-app
After a long wait, we have a shiny new sample project: vidmood. vidmood lets you share your moods in the form of wacky YouTube videos. It is (or will be) social, it's fun, and most importantly it demonstrates a different approach to building web apps with node.

The blog application was a traditional web app, in the style of Ruby on Rails or Symfony 1: a big web application server rendering lots of HTML for each individual URL the user can access.

vidmood is different. vidmood follows the new wave approach taken by web apps like twitter and gmail. When you visit the home page, you receive a web page... and that's the last time you receive a new web page (except for the Twitter login process).

Instead, that web page contains a JavaScript application in its own right, perfectly capable of asking the node app server for the resources it needs. When the JavaScript code in the browser wants to post a new "mood" to the site, it does that via an AJAX request. Updates to the live feed of "moods" are also received via AJAX. And rather than sending "fully baked" HTML for these AJAX updates like early AJAX applications did, the server just responds with data in JSON format.

What does all this mean? It means that vidmood is fast, and easy to scale. The server does as little work as possible. The browser does all that heavy lifting shifting HTML elements around. All the server has to do is provide access to data. In a nutshell, the server is the model layer.

]]>http://justjs.com/posts/vidmood-a-new-wave-web-appvidmood-a-new-wave-web-app
I'm back from my honeymoon, newly espoused and ready to write about node.

I was surprised to discover that justjs went down during my absence. My VPS was rebooted, 11 days ago. And it seems mongodb won't restart from an unexpected shutdown without a little manual intervention when journaling is not in the mix. Speaking of which, I really should be running MongoDB on a 64-bit system; but for the justjs app it isn't really necessary, so long as I understand its quirks (like this one).

]]>http://justjs.com/posts/tanned-rested-and-readytanned-rested-and-ready
Just checking in to give folks an update on what's happening with justjs.

Three things are going on, in descending order of importance:

I'm getting married on Saturday! I'm busy getting ready for that long-awaited occasion and the honeymoon to follow. So justjs is running a bit behind schedule.

The next installment will use Backbone... which means a complete rewrite of the blog. This is exciting. It also means it's taken a bit longer than the average installment.

I am ready to throw CKEditor out the window of a very, very tall building. If I never see the words "Cannot call method setHtml of null" again, it will be much too soon.

So when I get back from snorkeling in Ambergris Caye, I will be ditching CKEditor, probably in favor of jWYSIWYG, and wrapping up the next installment of justjs.

In the meantime, I suggest you check out backbone.js. Because as fast as node is, there's still no reason to make your users wait for the browser itself to create a new universe on every page of your application. Backbone is where sanity and AJAX meet. I'm excited. But not as excited as I am about Roberta Susan Petusky. No offense, folks.

]]>http://justjs.com/posts/hey-where-did-you-gohey-where-did-you-go
We've got a blog... if you call being able to type in a single run-on paragraph with no line breaks a blog. Let's fix it so we can style blog posts any way we like.

We could tackle this the old-fashioned way... one of several old-fashioned ways, actually:

We could let users type in HTML, and just trust it.

We could let users type in HTML, and validate it to remove scary stuff like XSS attacks.

We could offer users a more convenient alternative markup language, like Markdown.

We could just convert newlines to <br /> tags and URLs to links.

We've all seen sites that use each of these strategies. And honestly they have their place. For comments and status updates, the last one can be a good choice; fancy formatting is probably just a distraction when folks are writing short comments. And Markdown is pretty awesome if you know what you're doing. Making people type HTML just to get line breaks is obnoxious, but it's not uncommon to mix that strategy with the "convert newlines to <br /> tags" strategy.

So which one are we going to pick? None of them! We're going to give the user a full-blown rich text editor, just like Tumblr or Google Docs or Gmail. Because there's no reason why you should have to learn a markup language just to use italics.

The Rich Text Editor: Already Built Into Your Browser...

There are many rich text editors for the web. No, wait, let me rephrase that: there is exactly one rich text editor for the web... the one that's baked right into your web browser. Adding rich text editing can be as easy as this:

<div contenteditable="true">
This text is editable.
</div>

Every "rich text editor" you can add to a website is nothing but a collection of buttons to trigger functions in this built-in rich text editor, accumulated workarounds for frequently encountered problems with it, and extensions to make it do just a little more.

... But Making It Work Well Is Another Story

Sounds like the Emperor has no clothes! So why doesn't everyone just take advantage of "contenteditable" for themselves? Because while it is easy to make an element editable in modern browsers, the number of workarounds necessary to really make this cooperate nicely in every browser is pretty scary. So much so that the difference between a well-written wrapper for "contenteditable" and a naively coded one can be enormous.

That's why there are zillions of rich text editor projects out there, but only a tiny handful that can be trusted to work across Internet Explorer, Safari, Opera, Firefox and Chrome in a reliable and predictable fashion. Subtract the editors that have problematic or non-free licenses or dependencies on JavaScript frameworks most people don't use, and you're left with two options that are pretty much guaranteed to work.

Those two options are TinyMCE and CKEditor. Both have been around for a long time. Both are open source with permissive licenses.

CKEditor is a complete rewrite of the rich text editor formerly known as FCKEditor. I recommend CKEditor because it has a larger development community behind it than TinyMCE, and when things go wrong, a larger open source community equals quicker fixes.

"Isn't CKEditor slow?" You're probably thinking of FCKEditor, which had a well-deserved reputation for being "heavier" than TinyMCE. CKEditor is a successful rewrite with much better performance. Even so, we will load CKEditor only on the pages that really need it.

"If we're going to use a rich text editor, what does this really have to do with node? The rich text editor runs on the browser." Good question. First, this blog is primarily about Node, but also the entire web development stack around Node applications. The goal is to learn how to build complete Node apps, and also to see how JavaScript techniques that matter for Node also matter for jQuery and other client-side work. So this isn't the last time we'll look at browser-side code. Second, we'll be looking at the server side when it comes time to actually save the rich text without allowing unwelcome things like XSS attacks to sneak through. And we'll also be looking at how to integrate CKEditor with our view layer effectively.

Installing CKEditor

You can download CKEditor from the CKEditor site. It doesn't really matter whether you download the .zip file or the .tar.gz file. Your browser will invite you to open either one with your Mac's Archive Utility app. This will extract the file to a subdirectory of your Downloads folder called ckeditor.

Copy your node-apps/blog-5 project to node-apps/blog-6. Then move the ckeditor folder to node-apps/blog-6/static/ckeditor (that is, make it a child of the static folder in your blog-6 directory).

If you take a peek at the ckeditor folder, you'll notice three things:

1. It's pretty big! You can delete the _samples and _source subdirectories. You can also remove every subdirectory of "lang" that starts with a two-letter language code, except for "en" (English). Feel free of course to leave the other languages in place.

2. CKEditor is about 90% JavaScript and 10% HTML and CSS for the various dialogs it presents. But it comes with example code to integrate it with server-side languages we're not using, like PHP and ASP. You can remove these, but since our server is based on Node and wouldn't know how to run an ASP file if you begged it to, there's no real security risk in leaving them in place.

3. There's a config.js file. You can edit this and uncomment the "config.language" line if you want to set a default language for the interface other than English.

Integrating CKEditor

CKEditor is remarkably easy to integrate. CKEditor is designed to "progressively enhance" the good old textarea HTML element. The idea is that server-side code can receive the form submission just as it always did.

Of course, what CKEditor submits to the server is rich text, not plaintext. In practice, this means CKEditor submits HTML tags. It is exactly as if the user had manually typed:

<p>This is a paragraph with some <b>bold</b> text</p>

... In an ordinary textarea.

So how do we ask CKEditor to enhance textareas? And how do we prevent it from enhancing textareas we'd rather leave as-is?

A good strategy is to assign a CSS class to the textareas that should be enhanced. And that's exactly what we'll do, in views/new.ejs:

<textarea name="body" class="post-body rich-text-editor"></textarea>

Note the addition of the rich-text-editor class.

That's great, but how do we take advantage of the class? Simple: in views/layout.ejs, we'll ask jQuery for all of the textareas that have that class. Then we'll enhance each one.

This code loads CKEditor, then calls CKEditor's replace() function with each element that has the .rich-text-editor class. We do it just before the </body> tag:

jQuery and Node: Not So Different

Many programmers use jQuery without really understanding it. A good grasp of Node really helps here, because like Node jQuery relies on callbacks and nested functions.

jQuery's famous "$" is just a JavaScript function named "$." The "$" function is jQuery's main point of entry. If you pass a callback function to $, it is called when the DOM (Document Object Model) is ready- in other words, when all of the HTML elements in the page are ready to be safely manipulated. This is different from using an "onLoad" handler on the body element because it does not require waiting for every image to load.

When you pass a CSS-style selector like .rich-text-editor to the $ function, you get back a collection of all of the elements in the page that match that selector (in this case, the textareas we want to enhance). These collections support lots of useful functions that can be chained together, including each(), which is similar but not identical to Underscore's each() function; we pass a callback function to it and it is invoked once for each matching element, with "this" set to the matching element. Note that jQuery also has many methods that don't require using each(); you can call hide(), for instance, to hide every matching element in the collection without the need for a callback function for each one. We use each() when we want to do something to each element that is not included in jQuery's built-in set of functions.

Thanks to each(), we're ready to use CKEditor's API to finish the job. A call to CKEDITOR.replace(this); replaces the element we pass to it with a rich text editor that behaves identically.

CKEditor's ability to make its rich text editors behave exactly like the textareas theyreplace isn't perfect. If we were submitting the form via AJAX, we would have to nudge the rich text editor to update its underlying, hidden textarea first. We'll touch on that in a later installment. But for a normal form submission like ours the progressive enhancement works with no extra effort.

Displaying Rich Text

If you fire up the site now, you'll see a rich text editor with a toolbar not unlike Google Docs or Microsoft Word, in place of the plain-vanilla textarea. This is great, but once you save the post and go to view it, you'll see... escaped HTML tags. Not so awesome.

To fix that, all we have to do is edit views/post.js and change:

<%= post.body %>

To:

<%- post.body %>

While <%= is great for deliberately escaping things so that unwanted HTML doesn't creep in, <%- is perfect for situations where we actually want the HTML tags to take effect, because it does not escape what it outputs.

Sanitizing the Rich Text

"Rich text is great, but doesn't this mean that an irresponsible user could write a post containing malicious markup?" Yup! To be exact, if we decide to allow someone to post but don't trust them 100%, that person would be able to carry out an XSS (cross-site scripting) attack by inserting HTML that includes JavaScript code, altering the behavior of the site to mislead other users (although they would not be able to compromise the server itself).

Fortunately there is a handy node module to clean this up for us. The validator module, by Chris O'Hara, includes a variety of useful functions for validating and sanitizing form fields. With validator you can easily verify valid email addresses, check that numbers are in a certain range, and carry out a variety of other validation tasks. And the validator module also includes an xss() function which can be used to eliminate XSS attack vectors from HTML markup without discarding the rest of the tags.

To take advantage of validator, first install it:

cd node-apps/blog-6
npm install validator

Then add a line to the top of app.js to require it and save a reference to the "sanitize" function:

var sanitize = require('validator').sanitize;

The validator module includes two different kinds of functionality: validation and sanitization. Validation checks to be sure input meets certain expectations or rejects it, while sanitization actually cleans up the data. The xss() function is in the latter category.

Now let's modify the app.post('/new') route to take advantage of xss() by adding a new line just before our call to context.db.posts.insert():

post.body = sanitize(post.body).xss().trim();

Notice that we must call the sanitize() function first, passing in the body of the post. After that we can call as many sanitization methods as we want, chaining them together to get the desired result - in this case, removing XSS attacks, then trimming off any leading and trailing whitespace.

Any new posts you add after this point should automatically discard any links you add via CKEditor's link icon that attempt to embed JavaScript, such as "javasscript:alert('test')".

"Can't I just remove links from the toolbar by editing ckeditor's configuration file?" Yes, but remember this: never trust the web browser! Any attempt to secure your website with JavaScript on the browser side is doomed to failure because any skilled hacker can just rip out your JavaScript in a copy of the webpage and substitute their own code to submit whatever they please to the web server. The only validation that really protects you is validation on the server side, in your node code. Validation on the browser side is only appropriate as a convenient way to give honest users feedback. Don't trust it.

Don't Slow That Site Down!

Launch the server and visit the site. Add a new post with some styling via the rich text editor. Lo and behold, you get rich styling. Victory!

Just one problem: we're loading CKEditor all the time. And that's slow. Slow for the user, because the browser does extra work. And slow for the server, because it must deliver extra files to every single site visitor. Which is asking for trouble as your traffic grows.

We can fix it by loading CKEditor only when we really need it.

Loading CKEditor in the layout is still a good idea. But we need to do it on demand. We can do that by taking advantage of slots in the new.ejs template.

Just add this line to views/new.ejs:

<% slots.richTextEditor = true %>

Now we can look for this flag in the layout to decide whether to go to to the trouble of firing up CKEditor, like so:

Launch the site again and take advantage of the "view source" feature of the browser. You'll see that CKEditor is only loaded on the one page that currently needs it: the "new post" form. If we need it on other pages, we can just add the same slots.richTextEditor = true code to those pages as well.

Next Up: Summaries and Caching

Rich text posts are all very well, but right now, visitors to our blog's home page don't see much. All they get are the titles of posts. Most users appreciate a bigger clue of what's inside each post. In the next installment we'll look at how to summarize the individual blog posts on the home page without displaying the full text. And since accessing and summarizing every blog post on every visit to the home page threatens to slow the server down a bit, we'll look at how to cache the result so that the home page is not rebuilt until a new post is added. Caching can lead to unpredictable behavior if it is not done well. But it is also an important tactic for popular sites running in production. We'll see how Node's single-threaded nature makes caching dramatically easier than in most environments.

]]>http://justjs.com/posts/let-s-be-bold-rich-text-editing-and-validationlet-s-be-bold-rich-text-editing-and-validationSince the last installment, our blog is looking a whole lot nicer. But right now, anyone can post. They don't need jedi mind tricks. We have no security whatsoever.

Let's fix that by taking advantage of the passport module and gmail authentication. We can easily lock the "New Post" feature down either to one person or to anyone with an email address in an entire domain, as long as that domain is hosted with Google Apps for Business (aka "gmail for domains"). If you're already logged into gmail, all you have to do is check a box to grant permission. Logging in separately is not required.

"Hang on there. Are you telling me I can only use gmail authentication? I don't even use gmail." No, not at all. Passport actually supports many, many authentication methods, including Facebook, Twitter, and yes, an old fashioned local database of accounts just for your site. Using gmail as an authentication provider is just one example.

So why did I pick gmail for this demo? Because so many individuals and companies do use gmail accounts or host entire domains with gmail. And because in the latter case we can easily grant access to everyone at our domain. Trust me, when your title is "web developer" but your responsibilities also include system administration, you want to do this. Managing accounts yourself when your users already have them via Google is just an unnecessary hassle and a security risk.

So if you want to use Facebook, Twitter or another method, don't worry. The technique I demonstrate here is still 100% relevant. Passport keeps these things as consistent as possible between authentication methods. You can find the details for configuring other authentication strategies on the passport site.

To match all users in that domain - more convenient and less error-prone than using regular expressions.

To use Passport we'll need to turn on a few other features in Express. Passport needs to remember that the user is logged in, which requires a user "session" that retains information about this particular user from one page view to the next. In order for user sessions to work, an HTTP cookie has to be sent to the browser containing a session identifier.

To enable these things we'll activate the standard express.cookieParser() middleware, which makes cookies available and updateable as properties of a req.cookies object, and also enable sessions in Express. Enabling sessions requires us to set a session secret. The session secret is used to cryptographically hash the session identifiers to verify that they really came from our app, so that they can't be easily guessed and faked. Just add these lines to the init() method of app.js, just before the express.static middleware is added:

You'll need to add a value for the sessionSecret property to your settings.js file. Any secure password will do.

The rest of our code relating to passport could follow right here, but that would clutter up the init() method. So instead we'll call:

configurePassport()

Right after adding the express.static middleware.

Now let's check out the configurePassport() function, which we'll add at the end of the init() method. Keeping it nested inside init() allows us to see convenient local variables like "app".

The configurePassport function begins by setting up GoogleStrategy, a module included with passport that implements gmail authentication. We'll pass in appropriate settings and a callback function that creates a simple user object from the profile information that gmail gives back to us. This user object will become available as "req.user" once a user logs in:

Our callback function ends by calling done(null, user) to pass the new user object back to passport. null signifies that no error took place.

So far, so good. But what should context.settings.google be?

In your settings.js file you'll need to add a "google" property. This contains two properties of its own, "returnURL" and "realm". returnURL is the URL we want the user to be sent back to after they successfully authenticate with Google. "realm" represents the domain for our blog (not the user's email domain). If you're currently testing the site as http://localhost:3000/, then reasonable settings would be:

Remembering the User in the Session

I mentioned that passport doesn't force us to use a particular authentication module. Actually, passport doesn't force us to do much of anything. In particular, it doesn't force us to store the current user object in the session in a particular way. So we need to provide our own code to store the user in the session and to pull them out again.

We do this by calling the passport.serializeUser and passport.deserializeUser functions and passing in callback functions (yes! However did you guess?) that turn the user object into a string and back to a user object. For some projects, you might choose to do this by storing user objects in a database, and returning just the id of the user object. But since an email address and a display name don't occupy much space, we'll just turn them into a JSON version of themselves (that is, we'll turn them into simple JavaScript data, not unlike our settings.js file). This guarantees that they can be stored in a database if we choose to keep sessions in a database later.

Here's the code to serialize and deserialize our users using the JSON.stringify and JSON.parse methods. These calls are short, and they'd be even shorter if I didn't bother to check for bad data in the session (hey, you never know):

Hooking Up to Google

Now we're ready to initialize passport and turn on session support:

app.use(passport.initialize());
app.use(passport.session());

We're making good progress! The next step is to provide a route that forces the user to log in, and a route that actually accepts a successful login when authentication information is returned from Google. The first route, /auth/google, is convenient because we can redirect to it in order to force users to log in to continue. The second route, /auth/google/callback, sounds scary but passport does all the hard work for us.

In both cases, the passport.authenticate() function does the labor. When we pass it the name of a valid strategy (such as 'google'), it redirects the user to Google where they may choose whether to log into the blog. When we pass it an object containing successRedirect and failureRedirect URLs, it processes the authentication information sent back by Google and redirects appropriately.

You Can Check In Any Time You Like, But You Can't Never Leave

Passport adds a logOut method to the request object, so we can just call that method and then redirect the user back to the home page.

You're logged in! Oh, you wanted to SEE that?

If you fire up your blog right now and visit:

http://localhost:3000/auth/google

You'll experience the Google authentication process. If you approve the authentication you'll be returned to the site successfully... but it won't make a whit of difference, because we haven't started checking whether you have permission to post yet!

We'll move on to that in a moment. But first, we'll need a way for page templates to distinguish between logged-in users and logged-out users, and also to display the user's name.

The simplest way to do that is to inject the 'user' object into the 'slots' property of the data object that we pass to the view.page method. Then we can easily write code like:

<% if (slots.user) { %>
<%= slots.user.displayName %>
<% } %>

We could tackle this by passing this data in every call to view.page. But that would become very tedious very quickly. We could also alter the view module to do it. But that would require marrying it to the app module more closely by passing in the req object.

A better way, one that saves work rather than making our routes more complicated, is to refactor a little bit by creating a "page" function in the app module. Here's is that function:

This function accepts the req and res objects, a template name, and a data object to be passed to the page template. Then it adds both the user object (as slots.user) and the session object (as slots.session) to the data object. And then it calls view.page for us and passes the result to res.send() to finish the job.

Now we can rewrite each of our existing calls that look like this:

res.send(view.page(template, {posts: posts}))

To look like this:

page(req, res, 'index', {posts: posts});

The result is a bit shorter, and also better because each page template (and the partials invoked by it) now has access to slots.user and slots.session.

It's time to add a login button! Let's make an addition to the section of layout.js that currently displays the "brand" at the top of the page:

Notice I've also added the "req" parameter to the "newPost" function. That was necessary to make sure it can call the new page() function properly.

The purpose of the validPoster function is to force the user to log in before they continue, and then to make sure they actually have permission to post (every gmail user in the world should not be allowed to post to your blog).

But where is the validPoster function? We'll add that new function at the end of init(), after configurePassport().

Our strategy here is simple:

1. If the user is not logged in (req.user is not defined), then redirect to /auth/google.

2. If the user is logged in, use the minimatch module to check whether they match our pattern for allowed email addresses of posters (found in context.settings.posters). If they do, great! If not, set a session property called req.session.error to a message explaining that although the user is logged in, they are not awesome enough to post to the blog.

In a few moments we'll see how to display this error message attractively using Bootstrap's excellent alert feature.

For this code to work you'll need to add a "posters" property to settings.js. If you set it like this:

posters: 'myemail@gmail.com'

Then only the user myemail@gmail.com is allowed to post. If you set it like this:

posters: '*@punkave.com'

Then anyone in my office can post - a trick which only works if that company's email is handled by Google Apps for Business, of course. (Note: that service has a free tier as well as an educational version which should also work.)

"Why didn't that work?" "Who knows?"

Go ahead, fire up the blog and take a peek. You should be able to log in and out as before, and more importantly, you won't be able to post unless you log in properly. For now though, there is no error message when you try to post and don't have posting privileges - you just get sent back to the home page. Let's fix that.

We stashed the error message for logged-in gmail users who are not quite awesome enough to post to the blog in the session. We'd like to be able to display an error, if there is one, on the next page - whatever that page might be. So let's add a new partial() call right after the breadcrumb trail:

<%- partial('crumbs', {}) %>
<%- partial('error', {}) %>

The "error" partial can take advantage of the fact that the session is available in req.session. So we'll use Bootstrap's nifty alert component to display an attractive error message that users recognize. Here's the code for error.ejs:

This code makes certain there is an error, then uses standard Bootstrap layout CSS and a Bootstrap alert box to present the error message nicely.

"MVC VIOLATION IN SECTOR TWELVE! CONTAIN AND DEACTIVATE MR. BOUTELL!"

We also do one sneaky thing here: we remove the error from the session. Since JavaScript objects are always passed by reference, we can modify them anywhere, including in a partial. So when we write:

<% slots.session.error = undefined %>

We zap the error right out of the session. Refresh the page and you won't see it again.

Many would say that you should not modify the session in a template, because it violates the separation of the controller layer and the view layer. And honestly, those folks are right in the long run. It would be better if clearing the error message by clicking the "x" sent an AJAX request to the webserver, asking nicely to remove the error. That way the error would keep displaying as the user moved from page to page until they paid attention to it, read it, and cleared it. We'll look at how to implement this in a future installment. For the time being, though, this is a very handy hack.

Making that "x" do something

Speaking of which, clicking that "x" doesn't do anything yet. We can fix that easily, thanks to Bootstrap. All we have to do is load Bootstrap's standard JavaScript in our layout. Then Bootstrap will spot the data-dismiss="alert" attribute on the "x" button and arrange for the alert to disappear as soon as the "x" is clicked.

Just add bootstrap's JavaScript right at the end of the body element in layout.ejs, right after jQuery:

Restart the app and behold! Users who are logged in but not awesome enough to post get a reasonable error message that they can dismiss. Nothing left to complain about, right?

Why Don't My Sessions Persist?

"DUDE. I definitely have a complaint. I have to log in again every time I restart the app. This is getting old fast."

Yes, that is annoying. It happens because we're using passport's default session storage.

By default, passport just stores sessions in memory in our node process. This is convenient because we don't have to lift a finger to store sessions in the database, encrypt them into a cookie or attach them to messenger pigeons. And it's fast, of course, as storing things right in RAM always is.

But there are drawbacks. You have just discovered the most obvious drawback: restarting our node app discards our sessions. Another, more important drawback: when an app gets popular (and I mean seriously POPULAR, not just "this is a neat blog" popular) a single CPU just won't do. For a truly large-scale app, you will someday need to run multiple Node processes (one to take advantage of each CPU core in the server), or even multiple Node servers. And when that happens, you'll discover that these processes can't see each other's sessions at all. You'll be logged in on one CPU, logged out on another. Not a pretty picture.

Fortunately Express doesn't limit us to storing sessions in memory. We have lots of other options: storing them in a Redis database, storing them in our MongoDB database, and so on. We'll look into it in a future installment, I promise. If you're in a hurry, check out the session documentation on the Express site-

"No, seriously, this is REALLY annoying. Can't we just fix it now?"

Oh all right. Here's how to store our sessions in MongoDB so that your login persists when the app is restarted.

The express module is built on connect, and connect makes it really easy to write custom session storage handlers. Easy enough that there are at least three for MongoDB. I'll save you some time: connect-mongodb is the right choice, because it lets you reuse the database connection you already have rather than creating a separate one.

You can install it with npm as usual:

npm install connect-mongodb

To hook it up we'll need access to our MongoDB connection. That object is created in our db.js module and, so far, we don't make it available in the context. Let's fix that by adding this line right after we call "new mongo.Db":

What's going on here? First we use require to load the connect-mongodb module. This returns a constructor function we can use to make session storage handlers. So our next step is to create one, passing in the existing database connection that we saved in the context object. FInally we call express.session much as before, but with the addition of a "store" property pointing to the storage handler we just created.

Stop the app and restart it, then log in just one more time. Now restart the app again, click refresh and... yes! You are still logged in.

Oh I'm Sorry, Did You Want Paragraphs?

We've brought the blog a long way. It looks good, it stores posts in a proper database, and we have access control. But the actual blog posts are... well... unconsidered, as the designers in my office would put it. Right now all you can type is an amorphous blob of text that shows up as a single run-on paragraph. We can do better than that. In the next installment, we'll do a lot better by integrating a rich text editor into the app while also validating the markup so that no funny stuff sneaks through.

Yeah, not so marvelous. It's too much work, it's tough to maintain, and it's scattered all over our "app" module. Which really ought to be our "controller" layer. Not our controller and view layers all mashed up like a Reese's Peanut Butter Cup. Except not yummy.

Clearly, we need page templates! Folks who have programmed in almost any web framework have used templates of one sort or another. What they all have in common is that they put HTML first and code second, allowing you to write HTML with some code in it, as opposed to the other way around.

Really good template systems also let you have layouts. Sure, you can write a complete HTML page - starting with the DOCTYPE, continuing through <html> and <head>, plowing through <body> and going on to that final </html> - for every single interaction. But that's a waste of effort, and it pretty much guarantees your pages will get out of sync with each other as you improve your design for one page and fail to apply it to 600 others.

The best template systems also permit you to have partials. Partials are snippets of HTML usually not intended to render an entire page. Instead they are responsible for some part of the page - the tabs, the breadcrumb trail, the side navigation, or a single blog post in a pageful of post summaries.

Good template systems also let you pass data into a template, but don't automatically assume every variable should be visible to the template. That's because good separation of concerns is a virtue in every part of your code, whether it's part of the model layer (the database), the view layer (the templates) or the controller layer (the routes). Many developers have come to feel that it's better to be explicit rather than doing things "by magic." Things that happen by magic are also things that lead to a lot of "WTFs per minute" and "spooky action at a distance" - situations in which you can't figure out why something is happening. Usually something bad.

At the same time, it's important that we be able to communicate things from a page template back "up" to the layout. In particular, the page title, the breadcrumb trail and the subnav (just to take common examples found on many sites) usually have consistent markup that can profitably be shared amongst pages via a shared layout template. But the actual text of the page title or list of breadcrumbs is most intuitively set in the page template. That's why great template systems allow us to set "slots" that are intentionally visible to other partials (a concept I learned from the Symfony framework).

Picking a Template System for Node

Which template system for Node.js best exemplifies these virtues? Well... none of them, really. Express has template support built in, which is great, but all variables visible to the page template are (as near as I can tell) shared with partials, which I consider to be a drawback. There's also support for layouts in Express 2.0, but no clear mechanism to share data between partials and layouts on purpose. Express 3.0 (which we're not using yet, as it's not the current stable release) defers the issue of layouts to the template language... and not all template languages have them built in.

But in this confusion lies an opportunity. Node developers tend to want to pick and choose from different systems and build apps their way, for maximum performance and control. So in that spirit we'll write our own view module to wrap our template engine and provide page templates, layouts and partials that work the way we want.

But will we write our own template language? Not on your life, buddy! Creating the right "glue" for our needs is one thing. Totaly reinventing the wheel is another. That way lies madness.

So what template language should we use? One that suits our philosophy of putting JavaScript front and center when possible and not learning additional languages to gain relatively small benefits. And one that builds on our existing knowledge of HTML. So while Jade and Haml have some neat characteristics, we're going to use EJS. In particular, the really sweet implementation of EJS from visionmedia which is available in npm.

EJS is right for us because it makes mixing JavaScript into HTML simple. EJS has just three really important features:

1. To print the value of a JavaScript expression as part of your HTML, while escaping any markup characters like <, >, and & to prevent XSS attacks, just use <%= ... %>, like this:

<h3><%= post.title %></h3>

As a good rule of thumb, this is the safest way to output variables in your markup.

2. To print the value of a JavaScript expression as part of your HTML without escaping markup characters (because you are intentionally printing out HTML markup), use <%- %>, like this:

3. To simply run some JavaScript inside your template, use <% and %>, like this. Notice that we can shift back into "HTML mode" inside the callback function of our call to "_.each()". We'll switch back to "JavaScript mode" to close the _.each call:

The _.each() function, as you've probably guessed, is an Underscore function that acts a lot like jQuery's each() function. It loops over all of our posts and invokes a function (usually written inline with the "function" keyword) once for each post.

EJS has more features, but these are the features we depend on. They provide almost enough power to finish the job.

I say "almost" because we still need a way to get page templates, layouts and partials to behave the way we want. And that's where our own "view.js" module comes in.

Housekeeping: Setting Up the Fourth Version of the Blog

First, make sure you've copied your blog-3 project to a new blog-4 folder.

Then install the ejs npm module:

cd node-apps/blog-4
npm install ejs

Changes to server.js: initializing the new view module

Now, to get started, we'll load view.js as part of our initialization in server.js. Just add one more callback to the async.series() call:

Like our other modules, the view module has an init() method that takes a callback, just in case we want to do time-consuming things. Unlike our other modules, we don't pass the context object into the module. Instead we just pass the options the module really needs. That's because I'm making an effort to keep this module "clean" of justjs-specific features, with an eye to releasing it as an npm module. And the more we can keep our modules independent of each other, the easier it is to debug, refactor and reuse code later.

The call to context.view.init() is interesting because we it deals with directories on the server. The special global variable __dirname is always set to the name of the folder in which the current source code file (server.js) is located. We can take advantage of this fact to reliably locate views (that is, ejs template files) that are kept in a subdirectory of the main project called views.

Introducing view.js: our view layer

Now that we've seen the configuration code, it's time to check out the view module itself.

We need the ejs module for templates, Underscore to fill JavaScript gaps too fundamental to go without even inside a template, and the fs module to load template files.

Next we'll declare two variables that are global to the module:

var options;
var templates = {};

"options" is just a reference to the options we pass to the init() function. We'll use templates to cache each compiled ejs template so that we don't have to load each one from disk and compile it to a JavaScript function again on every page view. If the templates object already has a property matching the name of the template, we don't have to compile it again.

As before, we have a callback so that we have the option of doing time-consuming, asynchronous things in the init() function later.

The "page" method: rendering complete pages with layouts

Now we can check out the good stuff. Let's look at the "page" method. Our "page" method takes two parameters: a template name and a "data" object.

page: function page(template, data)

The "data" object is passed to the ejs template we'll use to render the main body of this particular page. Any properties set on the data object become visible as local variables in the ejs template. So if we write this in a route:

res.send(view.page('post', { post: post }));

Then we'll be able to access the post object in post.ejs:

<h3><%= post.title %></h3>

There is a special property that is always set on the data object. That property is called "slots." This property is special because it is always passed on to any additional partials you call from inside your page template, as we'll see. It is also passed on to the special "layout" partial that renders the outer part of the page (the head element, the title element, the body element and so on). Since variables are always passed by reference in JavaScript, this means that the layout can see any changes you make to the slots property.

One great use for the slots property is creating a breadcrumb trail. The page template might add a "crumb" linking to the current page, and the layout itself might add another "crumb" pointing to the home page. Since this is a common operation, the "slots" object is created with an empty "crumbs" property to save us the trouble of checking whether it exists yet in every partial that touches it. We do the same for "title," which page templates use to pass the title of the page back to the layout, and "bodyClass," which page templates use to set a CSS class on the entire "body" element. Setting defaults here saves us many calls to _.defaults inside our templates:

We've seen _.defaults() before. We use it to provide default values for any properties that are still undefined on the data object and otherwise leave it alone. In this case we use it to provide a slots property if one was not explicitly passed in. We'll use _.defaults() in many places throughout justjs.

Now that we've set up the data object for this page, we'll call the "partial" method to actually render it, and stuff the result into the "body" slot. Notice that a page template is really just a partial like any other, with a thoughtfully set up data object:

data.slots.body = view.partial(template, data);

Wrapping the Page Template in a Layout

Once we've done that, we're ready to tuck the output of the page template inside the layout by rendering the "layout" partial. But first we provide the option to override the layout or shut it off entirely and just return the output of the page template. This is helpful when certain pages require radically different layouts or are being rendered as part of an AJAX response:

Notice that we don't pass the entire data object from the page template to the layout - only the slots object is shared, as we intended. This makes it easier to insulate the layout from unintended side effects.

Of course, the real bottom line of the view module is the "partial" method. This is the all-important method that:

1. Compiles the ejs template, if it hasn't been compiled already.

2. Injects a function into the data object that allows partials to embed other partials. This allows us to split things like the breadcrumb trail and the rendering of a single blog post summary out to separate files, which makes code easier to maintain and read and promotes reuse.

3. Ensures that every partial has access to the shared slots object, and also to Underscore (which I consider essential to responsible JavaScript programming because of core features like _.each and _.defaults).

This code first checks whether we already have a compiled template function for this template name. If not, we call ejs.compile() with the text of the template. In order to do that, we first need to fetch the text of the template from the appropriate template file.

The fs.readFileSync function reads a file from the filesystem and returns a string, provided that we pass the filename and the encoding we want. The path to the template is simple enough: we already know what directory the views are in, so we just add the template name and a .ejs extension. The encoding is...

"Hold up there. What's this encoding business?" Historically there have been many ways to encode text on the Internet so that many different human languages can be represented. The fs module does not assume that everything is in English, and boring "plain ASCII" English at that. The most common way to encode text is the "utf8" encoding, part of the Unicode standard. "utf8" starts out with the plain ASCII character set but can also use multiple bytes to represent characters in just about any language, as long as your fonts cover them. In a nutshell, utf8 is the right encoding choice for the modern web.

"You're calling a 'sync' (synchronous) function. Isn't that against the spirit of Node?" Calls that load small files from the local filesystem occupy a middle zone between "clearly too slow to do without callbacks" and "often too fast to worry about," which is why synchronous versions of them are available. However, in this case, notice that we only load the template file if we haven't loaded it already since the server was launched. So there is really no meaningful performance penalty here. If you're concerned, you can refactor my code to compile everything in the "views" folder in the init() function. Heck, you can do it asynchronously with async.parallel and invoke the callback function of init() when it's all done. Knock yourself out.

Let's look at step two: making a "partial" function available inside the template.

Notice that we don't force the issue. If you really want to pass in a different "partial" function, you can, and that will be communicated through to all of the partials called by your page template.

The function we assign to "data.partial" is a simple wrapper around a recursive call to view.partial (a call that a function makes to itself). Our wrapper function expects a partial name and allows us to also pass a data object to the partial. As before, we set partialData.slots to the same data.slots object that is shared with the calling partial if it has not been explicitly overridden.

Here's step three:

_.defaults(data, { slots: {}, _: _ });

Now our "data" object is guaranteed access to a "slots" property (usually already present by this point unless it was intentionally cleared), and also has an "_" property (which allows us to call Underscore functions from JavaScript code in the template).

Finally, step four:

return templates[template](data);

Once the template has been compiled, rendering it is very easy. A compiled template is just a JavaScript function that expects a data object. We pass in the data and the function returns the result.

Writing EJS Templates

That covers it for the view module. We're ready to render templates on demand. Now let's look at the actual templates.

We'll start with the layout, because the layout provides the basic structure of every page on the site. Here's the beginning of layout.ejs:

What's going on here? The first line is an HTML doctype for HTML5; it tells the browser to expect modern markup.

The next line adds a breadcrumb. Actually, it prepends a breadcrumb to the front of the breadcrumbs array. That makes sense because the "Home" crumb should always be the first one. If you're not familiar, the "unshift" method is a standard JavaScript method that inserts a new element at the beginning of an array (much as push() appends it at the end).

If you're basically familiar with HTML this isn't too new to you. The <meta name="viewport" ... /> line is a little unusual; it's part of the standard boilerplate that comes with Twitter Bootstrap, an amazing library of styles, buttons and functionality that makes brand new web apps look and feel ever so much better fresh out of the box. This particular line tells mobile browsers not to try to automatically "zoom out" like iPhones usually do if they suspect a page was not specifically designed for them.

The next line sets the title element. Totally familiar stuff, but it demonstrates how to use EJS to insert one of our slot properties:

<title>justjs: <%= slots.title %></title>

After that we bring in stylesheets from Bootstrap as well as one of our own. We bring them all in from a directory called "static." Later on I'll show you how to make files in the "static" folder visible much like a regular Apache webserver would.

Our "body" element kicks off by using a slot to optionally set a CSS class on the body. This is very handy for writing CSS rules scoped to entire pages:

<body class="<%- slots.bodyClass %>">

Now a little barrage of HTML to set up the standard Bootstrap navigation bar, which we'll fill with tabs later. For now we just have a "brand" element with a link to the homepage:

Next we'll create a container div to wrap the main content of the page as well as the breadcrumb trail. We'll insert the breadcrumb trail by calling the "partial" function to render the breadcrumb.ejs template. We don't have to pass the crumbs explicitly because they are part of the "slots" object:

The divs here are just Bootstrap structural stuff. The interesting bit is the JavaScript in the middle.

Here we loop over all of the crumbs found in slots.crumbs and create a list of links. Notice that this time when we use _.each() we take advantage of the opportunity to receive the current index in the loop and the list of crumbs itself in the callback function. This is a highly convenient feature of _.each() that makes it a good substitute for a 'for' statement in most situations involving an array.

Each link points to the "href" property of its crumb and takes its label from the "title" property:

<a href="<%= crumb.href %>"><%= crumb.title %></a>

The additional complexity comes from the desire to set a class on the last crumb (the active page), and also to insert a nice divider character between crumbs:

Note the use of the "?" operator, JavaScript's "ternary" operator. This is a handy way to avoid an "if" statement for simple cases. It works like this:

condition ? "value if true" : "value if false"

To set the "last" variable on each pass through the loop, we need to test whether our current index in the array is the last element in the array:

<% var last = ((i + 1) === slots.crumbs.length) %>

Now let's look at the "page templates" (templates that render a page on behalf of various routes). Here's index.ejs. Notice how similar it is to our previous hardcoded HTML. The layout does most of the work. All this template has to do is set the title and worry about its content. The markup has changed slightly to take advantage of Bootstrap's buttons. The button-box CSS class is found in our own justjs.css and just provides a little padding.

The new.ejs template, which presents the form for adding a new blog post, is simple too. The markup has changed a bit more to provide a decent look and feel via Bootstrap. An error message is passed in by the route via the data object parameter, as we'll see:

One interesting thing we do here: prepend a crumb to the breadcrumb trail, just like we did in the layout. Since the layout template executes last, it prepends last, so the Home link winds up first in the breadcrumb trail.

"Hey, you're still not formatting that post body!" Yes, in truth, the way we output post.body here is still half-baked. Escaping <, > and & is good because it means the site won't play host to XSS attacks, but we really need to give consideration to letting the user style what they type. Right now we don't honor newlines or present links, let alone allow for rich text editing. In a future installment we'll see how to use a rich text editor such as CKEditor to add a user-friendly rich text editing experience while still protecting the site against XSS attacks and bad markup. One could also use a module like node-markdown to allow a text-based markup language more convenient than HTML.

Rendering Page Templates in app.js

Almost there! We still need to actually call our lovely new page templates from our routes. That's where app.js comes in.

The code at the end of each route that used to build the entire page with "s +=..." statements goes away. And this call comes in:

res.send(view.page('index', {posts: posts}));

That's all it takes to render a page template for the home page route (app.get('/')), passing all of the posts along to the page template.

Rendering the page template for an individual post (app.get('/posts/:slug')) is quite similar:

res.send(view.page('post', {post: post}));

Both versions, GET and POST, of the /new route still share a "newPost" function that renders the form. That function just got a lot shorter:

We could modify the notFound function to call view.page too rather than just passing a string, if we wanted to create a fancy 404 error page. It would work exactly as it does now, with 404 still being passed as the second argument.

Delivering Static Files (When Node Acts Like Apache)

However, there's one more important change to app.js. We need a way to serve static files, most importantly the Bootstrap CSS and icon files, as well as our own justjs.css.

That's why we add this line to the init() method to leverage the built-in support for delivering static files in Express:

app.use('/static', express.static(__dirname + '/static'));

Again we take advantage of __dirname to locate the folder this source code file is located in, and add '/static' to point to a subdirectory by that name.

As for the contents of /static, I've divvied it up into folders like so:

/static
/bootstrap
/css
/js

/bootstrap should contain the css, img and js folders from the current release of Twitter Bootstrap. If you look at the justjs project on github, I provide a tidy little subset that works with my markup.

Ain't it pretty? (Well... relatively speaking)

That's truly it - we're ready to play with our new view layer. Fire up the site and have a look.

Bootstrap definitely makes a dramatic difference. And that's just a small down payment on what can really be done with Bootstrap by a skilled (or even competent) designer.

More generally speaking, the power of layouts, partials and slots sets developers free from repetitive coding and accidental mistakes and enables much better front end coding practices. Code is code, and all code affects the user experience your application delivers. It pays to take the view layer at least as seriously as the model and controller layers.

Next Steps: Locking the Front Door

The look and feel of the blog has improved a lot, but one big omission is easy to spot: security. There's absolutely nothing to prevent any random visitor from writing a new blog post. In the next installment I'll show you how to fix that by logging users into the site with the flexible Passport module, which supports many types of authentication, from a local database of passwords to Facebook Connect.

1. We created a "db" module that stores and retrieves posts for us. That module happens to use MongoDB, but if we changed our minds, we could write a new module with the same methods and "require" that module instead, and the rest of our code would be none the wiser. That makes it much easier to maintain our code.

2. We learned how to do simple things simply. The async.series function lets us carry out a series of steps one at a time, even though the code that implements those features returns right away and doesn't invoke our callbacks until later.

3. We learned a lot about nested functions and closures, key features of JavaScript that make callback-driven programming (Node's defining characteristic) soooo much more pleasant than in most languages.

But there's one thing we didn't do: run our app! That's because we stopped short of exploring the last new module in this version: app.js.

The app.js module: controlling the action

In previous versions of the blog, our code for displaying blog posts was pretty much the same as our code for fetching them. (To be fair, we didn't have a database yet, so the code for fetching them was awfully simple.) Yesterday we busted out the code for fetching and creating blog posts to a new module.

In the new version, we'll break out the code that responds when a browser accesses a URL - the routes, in Express lingo - to a separate module as well. As we saw yesterday, this makes server.js very short indeed. Now server.js is just responsible for deciding what modules go into the project, initializing them, and starting up our webserver.

So let's take a look at app.js. But first we'll install a new npm module, justly famous in both the Node world and the browser-side [removed] underscore.

Introducing Underscore

Underscore fills many otherwise serious gaps in JavaScript- a beautiful language, but it has its faults. Underscore overcomes many of them by providing truly reliable ways to manipulate arrays and objects, especially when we're using objects like associative arrays in other languages.

We'll use Underscore for many things in this blog, but today we want it for the ability to easily pick just the properties we want from an object.

Use these commands at the terminal prompt to pull underscore into your project:

cd node-apps/blog-3
npm install underscore

Building the app module

Now let's see some code! The first two lines of app.js just summon the underscore and express modules:

var _ = require('underscore');
var express = require('express');

Now let's look at the way module.exports is structured in this module:

"Hey, this could be simpler. Can't you just set module.exports to the init function and be done with it? And why have a callback at all if we're just going to call it right away?"

Good eye. Yes, technically I could just write:

module.exports = function(context) { ... }

And expect server.js to know that requiring this module just returns a function, which I could call like this:

// Require the module and immediately invoke the function it returns
require('app.js')(context);

But there are two problems with this:

1. I'll probably want to add more methods to this module at some point. Once I do that, the cute trick of just returning the init() function doesn't work so well.

2. Sooner or later, init() will probably wind up calling something that returns immediately and invokes a callback later. By planning in advance for a callback function, I can avoid aggravation down the road.

Still, it's good that you realize modules won't always follow the same pattern I've shown here.

Let's dig into the init() function. app.init() starts by creating our Express server object and saving it in the context for use elsewhere. We also save a reference to it in a local variable for convenience:

var app = context.app = express.createServer();

We're almost ready to set up routes. But first, since we're going to want to let the user write new posts, we need to be ready to deal with the POST HTTP method. As you know if you have written HTML forms before, a basic HTML form can submit to the server via one of two methods: GET and POST.

GET is handy for search engines and the like because the submitted fields wind up right in the URL. But since URLs have a limited length, and ideally should not be used as "verbs" that change things, we normally write:

<form method="POST" action="/new"> ... </form>

For a form that submits something big like a blog post.

"So wait, you're telling me Node can't handle forms?" Not at all- of course Node can handle POST. But Node doesn't automatically assume you want the form submission parsed for you in advance. For instance, if we were handling giant file uploads, we might prefer to handle it ourselves with a callback and display a progress meter while the upload is still going on.

For a form like this, though, parsing the form submission in advance sounds great. So let's add some middleware.

Express routes are very extensible. You can add "middleware" functions to any route, just by passing them as additional arguments before the final callback function.

Express includes a number of middleware functions, via a module called Connect that is baked into Express. One of these is express.bodyParser(), which returns a middleware function that can automatically parse POST requests and pop them into properties of the req.body object for us.

That lets us write:

req.body.title

We could install the body parser middleware for each route that requires it:

But while this is handy, we have the even more convenient option of installing it for every route automatically. So we'll do it that way:

app.use(express.bodyParser());

"If express.bodyParser is a middleware function that the route calls later, why do we follow it with ()? Doesn't that call it now?" You're right. express.bodyParser is actually a function generator - a function that takes the options we pass as arguments and then returns an inline function that can still see those options in its scope later. In other words, it returns a closure. But don't worry, you don't have to worry about that today if your brain is already full.

Now that we've added the body parser, let's look at the "/" route that displays all the blog posts. Most of this function has not changed from the previous version. The big change is that we call context.db.posts.findAll() to fetch our posts, and we pass an inline function that receives the posts back when the database is ready.

Here's the entire function. Notice that the previous version of this route is pretty much intact, nested in the callback that receives the posts:

Even though we now fetch real posts from a real database, our code didn't get much more complicated, apart from introducing a callback. And we're not dependent on MongoDB at all - as long as there's a context.db.posts object to talk to, we're happy. This simplicity is the big payoff of moving the database code into its own module.

We're still rocking the req.params object to access the named parameters in the route. The only change, again, is the introduction of a callback to receive the post from the server. If we don't actually get a post, we call notFound(), just as in the previous version.

Now let's move on to the cool bit: creating and saving new blog posts. For this, we have two different routes for the same URL: /new. The difference is that one responds to GET requests and the other to POST requests. We'll use the GET method version to display the form, and the POST method version to save it... or, when things don't go so well, to display it again with an error message.

The code for the actual form will be in a newPost() function shared by both routes. So the app.get('/new') route is very simple:

app.get('/new', function(req, res) {
newPost(res);
});

The app.post('/new') route is only a little more complicated, because we already wrote a sweet context.db.posts.inert() method that takes are of generating a reasonable slug from the title and then saving the post. All we have to do is make sure we fetch the right properties from the user's form submission. And that's where underscore's _.pick() function comes in:

Here we are interested in the "title" and "body" fields in the form submission. If any other fields were submitted, we're not interested, and we don't want them to clutter up our database. So we use _.pick(), a function that accepts an object and a series of property names, and returns a new object containing only the properties we asked for.

A note on security and user-submitted data

Our use of the pick() function here is a good first step toward validating the user's data. But it is only a baby step. It is important to realize that web browsers can submit anything... anything at all... no matter what our HTML and JavaScript code might say. That's because it is trivial for any halfway skilled hacker (or even your little brother) to gin up a script that ignores your markup and submits things you do not want on your website.

In the production version of justjs.com, I do more to validate form submissions (even though I'm currently the only poster to the blog). In future installments we'll explore tools like the _.defaults function, which is great for ensuring that no fields were skipped, and the validator npm module, which can be used to escape any HTML markup in submitted text, or just to validate submitted HTML to make sure it is not malicious.

On the other hand, there's a big area of security we don't have to worry so much about. Since we're using MongoDB's JavaScript-based API, not SQL, we don't have to worry about escaping queries to prevent SQL injection attacks. MongoDB's API was designed from the beginning to keep commands and data separate so the issue doesn't arise.

Inserting the Post

Once we have the post object the way we want it, we just call db.posts.insert() to insert it. If something goes wrong, we assume a conflict with another post that has the same title and call the newPost() function to redisplay the form. (Currently, we don't bother to redisplay it with the previous submission intact. We'll do more of that in future installments, in which we'll also introduce a nicer way of templating HTML markup.)

If everything works out great, we use res.redirect() to send the user to the URL of their shiny new post, just so they can have a special moment. Redirecting after a form submission is a good idea to prevent accidental resubmission of the same form if the user refreshes the page.

Displaying the Form

Finally, here's the newPost function that displays the form. I've kept the markup basic for now, since we'll be introducing nicer ways to handle HTML in future installments.

Fire it up!

Then connect to http://localhost:3000/ and enjoy. Lo and behold, the "New Post" link takes us to a form that successfully adds new posts to the blog. And they sort in newest-first order, as blog posts should.

Coming Up

So that's it right? Oh, not hardly. There are so many holes in this blog it ain't funny.

Things we haven't addressed yet (a partial list):

Templating. Currently we're writing HTML by concatenating JavaScript strings. This is not the easy-to-maintain way to go. It also represents a failure to separate concerns. Right now we have a nice "model" layer (the db module) and a nice "controller" layer (the app module). But the "view" layer is also shoved into the app module, as an afterthought.

Layout. Right now we have crummy bare-bones HTML pages that aren't very consistent. We need a way to wrap the markup for each route in an easily updated, attractive site-wide layout.

Access control. Everyone can post! That's how your blog works, right? Yeah, not so much. We need to log the user in, and decide what the user is allowed to do on the basis of who they are.

Session management. Part of logging people in is remembering that they are logged in.

Editing. We can post. But we can't edit or delete existing posts. We need to round out the feature set.

Better performance through web services. So far we've been assuming the web browser is a pretty dumb device, capable only of displaying web pages and filling out forms at the most. But in case you haven't noticed, it's 2012 and even Internet Explorer is pretty bright these days. gmail arrived a long time ago. We should be able to leverage the browser to speed up the user's experience and do more of the work on the browser side. At the same time, we need to do a rock-solid job of delivering the same content to dumb devices- and to Google and other search engines that are effectively equal to a web browser with JavaScript turned off. But we don't have all day to write duplicate code for two different devices. Fortunately, we don't have to - it's all JavaScript. We'll explore ways to leverage JavaScript's ubiquity to achieve good SEO and a modern user experience at the same time.

]]>http://justjs.com/posts/let-s-post-something-in-which-our-blog-becomes-a-blog-just-barelylet-s-post-something-in-which-our-blog-becomes-a-blog-just-barelyPlease note: "oh my" should be read in George Takei's voice for maximum effect.

Separation of Concerns: Breaking Things Up Into Modules

Our app is getting bigger. And as it gets bigger, the code gets harder to grapple with in a single file. It's time to break it up into several Node modules.

Readability isn't the only reason to split code up into modules. An even better reason: code for unrelated purposes shouldn't be jumbled together. Ideally, code that performs a particular function is best off in a separate module, relying as little as possible on knowledge of the outside world.

One can take this too far, but it's a good guiding principle. For instance, the route callback that displays a post to the user needs to fetch posts by their slugs. But it doesn't have to contain all the agonizing details of how to do that with MongoDB in particular. And if it doesn't - if you factor out that code to a separate module - then you can replace MongoDB with another database at any time, without searching your source code from top to bottom for things that might break. All you have to do is consistently provide the same set of functions.

Our First Module: settings.js

We need to start breaking up our code somewhere. Handily, since we're bringing the database into the picture, we need a place to keep our database settings. And a module to contain our settings is just about the simplest module possible.

Modules in node.js are very straightforward. A module is just a .js file and can contain any JavaScript code you wish. Any variables you create in that file are completely invisible to all other modules - with one especially important exception: module.exports.

When you write:

var settings = require('./settings');

In your server.js file, the require function returns whatever was assigned to module.exports in the settings.js file.

This simple rule means that you can party down with as many global variables as you want in each module without worrying about the impact on other modules. Just remember that if you want to make something available to other modules, you should make it a property of the module.exports object.

So let's get started! Create a blog-3 folder in your node-apps folder. Inside blog-3, create a new file called settings.js and paste in:

We've just created a module that exports two properties: db and http. Each of these has properties of its own. One set are handy for setting up a MongoDB database connection, while the other tells Express what port number to listen on.

At the top of server.js, we could now write:

var settings = require('./settings');

Note the use of ./ at the start of the filename. This tells Node not to look for a module that was installed in npm, but rather to look in the current directory.

However, I'm going to start off by first creating a container called "context" for all of the important objects that make up the application. That way we can easily pass both the settings and various objects like the Express application object to modules that have a need for them. We also get the option of being pickier and passing only the parts we deem important to a particular module. The latter is a smart choice for modules you might want to reuse in other projects.

So let's set up our context, beginning with adding the project-wide settings as a property of the context:

var context = {};
context.settings = require('./settings');

Now that we have our project settings, we're ready to configure the database:

Notice that the setupDb function first requires the db.js file, which we'll examine in a moment, and then calls the init method of the object we've just assigned to context.db. By passing the context to context.db.init, we make our database code aware of the project-wide settings. And by passing a callback function, setupApp, we're able to proceed to the next step of starting up our application: creating the Express application object, much as we did last week. But to make the code more maintainable, we'll migrate the actual route code to a separate module, just as we did with the settings and the database code.

Notice that setupApp expects an argument called err. If we've just asked for a database connection, and the operation fails, then we need a way to know about it. The standard way to do this in Node is to always pass an error object as the first argument of a callback - or pass null if everything is just fine. If other arguments make sense for the callback, they come after err.

If we want to handle errors in a particular way- if, for instance, we know how to work around it and make everything OK, or we want to log the error somewhere special - we can examine the err object. In this case, we just stop the server by throwing an exception, since we can't have a blog without a database:

if (err)
{
throw err;
}

async.series: you CAN carry out a simple list of steps

"Hey, wait a minute! This callback business is tedious and confusing."

Hey, you're right! This callback stuff is great when you want to let 500 things happen at once and not be stuck waiting for the first one to complete. And it also avoids the dangers of programming with threads. Callbacks help us stay out of the server's way and keep things very, very fast.

But when you really do need things to happen in a simple, specific order, it gets annoying real quick. Plus it's hard to follow what's going on, and tough to maintain the code.

Fortunately there's a better way. We can take advantage of the async module.

The async module provides a convenient way to call a series of callback functions in a specific order. Which gives us back the ability to do simple, sequential things, as we might in other frameworks and languages.

Actually, the async module gives us lots of cool toys. But the one we need right now is async.series.

The async.series function lets us specify a list of functions to be invoked in order and a final callback that gets invoked with an error if any of those functions fail.

First, don't forget to use the npm command to add the async module to your project:

We still write our setupDb, setupApp and listen functions, almost as before. The big difference is that they don't have to worry about receiving an err parameter, and they don't have to be hard-coded to call the next step. Instead they receive a single callback function as a parameter, and their sole responsibility is to call that callback when they're done, passing either null (when everything is great) or an error object. We also write a "ready" function that gets called last, to take care of any errors or simply declare victory.

Notice that we pass on the callback function to context.db.init. Since that function also expects a callback and calls it in the usual way, it's fine to delegate responsibility for invoking the callback.

This one is a little bit different because the context.app.listen function works immediately. So there's no need to pass the callback function on to some other function to be invoked later. Instead we just call it ourselves with a null argument, meaning that everything went fine.

These functions are much more maintainable now that we don't have to explicitly code each one to call the next one and worry about errors from the last one! We get to do that in one much more readable place when we call async.series.

But let's not forget the second argument to async.series: the ready function. This one does take an err argument, which will be present if any of the functions in our list failed:

The db module: talking to MongoDB

We've broken our application up into several modules. One of them is the "db" module. This module provides simple functions for fetching and storing blog posts, without worrying about the details of the database being used. Worrying about that is the db module's job. In this way, we create an interface that can easily be satisfied by a completely different database without changing code anywhere else in the project.

Folks who develop in traditional "MVC" (Model, View, Controller) frameworks would refer to our db module as the "model layer."

Start by creating a new file, db.js, in your blog-3 folder.

We'll kick things off by requiring the mongodb module:

var mongo = require('mongodb');

Now we can use the mongo object to work with mongodb.

Next we'll declare some variables that are visible only in this module. Remember, if it's not part of module.exports, it's nobody's business but your own. These are for our own private use in implementing the stuff we do export:

var db;
var postCollection;
var context;
var settings;

Now let's get things moving by implementing our "init" function. The "init" function provides a way to pass in the context as well as a callback function to be invoked once the database is ready.

We'll follow a common pattern by setting up the module.exports object pretty much the same way we did in settings.js, using the syntax:

{ property: value, .. }

Just keep in mind this works for functions exactly the same way it works for other properties. They become methods of the object, accessed with the "." operator just like other properties.

In addition, we'll set the variable "db" to refer to the same thing as module.exports. This is a common convenience to let us refer to the object we're creating with a lot less typing.

Notice that we passed in settings we received via the context object. We need to know the host and port the MongoDB server is running on, as well as the name we want for our database.

Creating the connection just makes an object; it doesn't connect right away. So let's call dbConnection.open. As I'm sure you have already guessed, it takes a callback function that is invoked once the connection is ready:

dbConnection.open(function(err) { ... }

Inside the callback function, we start by checking for an error. If there is one, we invoke our own callback (the one passed to the init function by the dbSetup function in server.js) with the error:

if (err)
{
callback(err);
}

If not, we're ready to get a collection object. MongoDB "collections" are similar to "tables" in SQL databases, but we don't have to explicitly create them; the first time we use them, they just come zinging right into existence:

postCollection = dbConnection.collection('post');

We could use our collection right away at this point. But there's one more thing we'd like to do: make sure that slugs are unique, and that we can find posts by their slugs very quickly.

If someone tries to create a post with the same title as another post, we'll wind up with two posts that have the same URL and no way to distinguish them. So to prevent that, we use the ensureIndex function of our collection to index the "slug" field, and we specify the unique: true option to enforce that no two posts can have the same slug.

Needless to mention, ensureIndex takes a callback function too. Once that function is called, we're ready to invoke our own callback, signaling that the database is ready to use:

Take care to balance your }'s and )'s. JavaScript may look like Java, PHP and friends, but it's really much more like LISP, Scheme and other languages that positively live for the joy of nesting things. Functions can go inside statements, and you still have to close those statements properly. Property lists can include functions, and you still have to separate them with commas. It takes a little getting used to.

Callbacks, Functions and Closures

You may notice that in this case I chose to write all of the callback functions "inline," rather than declaring them as separate, named functions. Node programmers tend to favor writing callback functions inline. My preference depends on readability, and whether I want to take advantage of the ability to see variables that are declared in the enclosing function - something JavaScript allows nested functions to do.

If you're paying close attention, you'll notice that JavaScript also allows my nested callback function to access the dbConnection variable even though the original init() function has already returned. The callback functions are written inside it, but they don't actually get called until later, when the MongoDB module acknowledges that the database is ready. Yet dbCollection still has its value in those callback functions. This extremely powerful feature of JavaScript is called a closure.

The ability to create closures is a key reason why callback-driven web development is so much more popular in JavaScript than it has been in other languages. Callbacks often yield fast, flexible code, regardless of the programming language. But in most languages, programmers tend to move away from callbacks (and events, a close cousin) whenever circumstances allow.

That's because in most languages it is much harder to grasp the context of each callback function and understand the flow of event-driven code. But in a functional language like JavaScript, we can use nested functions, closures, and tools like async.series to get the performance of callbacks without making code too hard to understand and maintain.

Enough Talk! Let's Store Something!

We'd like functions to find all posts, find one specific post by its slug, and create a new post. (Later we'll need more.) All of these functions have something in common: they deal specifically with posts, not with anything else we might add to the database later.

So let's group them together in a "posts" object, nested inside the db object:

The "posts" object is simply the next property after the "init" function in our "db" object (don't forget the comma after that closing "}" for the init function). This allows us to add more nested objects later to work with other types of data.

Now let's look at the inside of the findAll method. This method asks MongoDB to fetch all of the posts, in "blog order" (reverse chronological), then invoke a callback function with the results:

To fetch all of the posts in the collection we call the find() method without any arguments. find() returns a "cursor" on which we can call more methods to clarify exactly what we want MongoDB to do. Calling the sort() method with { created: -1 } indicates that we want to sort on the "created" field of each post, in reverse order (since blogs traditionally show the newest post first). And calling the toArray function with a callback finishes the job, asking MongoDB to execute the query we've described up to this point and then invoke our callback.

The findOneBySlug method is similar, using the findOne() method, a convenient variation on find() that retrieves just one object. In addition, we specify a particular field to search for and a specific value to be matched:

By specifying { slug: slug } we tell MongoDB that we are only interested in the post whose "slug" field is equal to the "slug" argument passed to the findOneBySlug method.

The insert method does a little bit more work than the others. In addition to passing on a request to MongoDB, insert also takes care of converting the title of the post to a reasonable "slug," or URL. "insert" also has to worry about the very real possibility of an error, since the user might try to insert two posts with the same title:

db.slugify is a simple method that converts a title to a plausible slug. It doesn't check whether someone else is already using it; the database index will do that (and I'll demonstrate a way to automatically fix it in a future installment). I'll look at db.slugify in a moment.

postColleciton.insert is where the real magic happens. When we call postCollection.insert, we are asking mongodb to insert a new document into the database.

MongoDB: safe at one speed, not so safe at another

MongoDB is designed to run very, very fast, sometimes even at the expense of safety. In some situations, it might be okay to just keep right on zooming along without worrying about the occasional unreported error. In others... not so much.

Since our blog is in the "not so much" category - users take a dim view of lost posts - it's important that we pass the { safe: true } option to the insert method of our postCollection object. This ensures that MongoDB will actually wait to be sure the insert was successful before invoking the callback function.

Don't worry, though. Even with safe: true, MongoDB is still very fast indeed.

Making Reasonable Slugs

One last method of the db object to look at: slugify. slugify converts titles to slugs by using a series of regular expressions. And it does a good job, as long as you don't have non-Latin characters that you want to preserve in slugs. If you do, you might want to check out the xregexp module in npm.

Tomorrow, We Ride!

So let's take a moment to breathe. Tomorrow we'll look at the app.js module, which contains our route functions for the latest version of the blog, including the "New Post" form that allows us to create and save blog posts.

]]>http://justjs.com/posts/models-mongodb-and-modules-oh-mymodels-mongodb-and-modules-oh-myEdit: brought back the notFound() function so we can use it when a reasonable-looking URL doesn't match an actual post.

When We Last Saw Our Heroes...

In the last installment, we built our first node app. And we learned that every node app is a webserver, deciding what to do with every incoming request from a browser entirely under its own power. Which is pretty cool actually.

Here we have the beginnings of what could easily become a very, very long "if... else if... else if... else if... else if... [deep breath]... else" block. Those can easily become difficult to maintain. What happens when our app is a little less trivial and we have 100 different actions to deal with?

Fortunately, there's Express! Express is an optional Node module that bills itself as a web framework, and it offers many things: user sessions, templates, file downloads, parsing of form submissions, routes (which will solve our problem today), and more. I'm not 100% sold on the way Express solves every problem. But this is Node, and in Node development you are rarely forced to use anything you don't want. So we'll pick and choose the parts of Express that help us.

Adding Express to your Project

Let's get started on our second version of the blog. To begin, copy your blog-1 folder to blog-2. Then, in the Terminal window, use the cd command to enter the blog-2 folder. Next use the npm package manager (the "Node package manager") to install Express in that project:

cd blog-2
npm install express

When this command completes, you should be able to type:

ls node_modules

And see:

express

The npm package manager does a great job of installing optional packages for you and knows how to install the additional components each package depends on. Later we'll see how to create a packages.json file so that the requirements of our project can be automatically installed.

Now fire up your text editor. Create a server.js file in the blog-2 folder, then copy and paste your previous version from the blog-1 folder to get started.

Express is a Node module. As we saw previously, Node modules can be loaded with the require function.

Express extends Node's concept of a webserver, so we don't have to require the http module anymore. So we'll replace this line:

var http = require('http');

With this one:

var express = require('express');

Our Express web application serves the same purpose that our HTTP server object formerly did. So we can remove this line:

... And replace it with this line, which creates an Express "application:"

var app = require('express').createServer();

But what about all that code inside our callback function? Don't worry, we'll bring that back! Rather than writing a giant if/else if/else if/else block, Express lets us register "routes" for each action of the web application. For every distinct action the user can take on the website, we'll just register a route.

Here's what happens when we take the index() function from the first version of the blog and turn it into an Express route:

1. Our index() function turned into a callback function, passed to the route function of the app object. We could also have just passed the function by name, but now that we're registering each route separately it doesn't really help the readability of the code as much as it did before to give each function a name.

2. We don't have to say "if req.url equals blah blah blah..." to figure out what to do. Instead we just tell Express we want a route that matches '/'.

3. We don't need our own sendBody function to set the content type to text/html before sending the page. Express has enhanced the res object for us with a shiny new send method that does the same thing.

4. I changed the link to each individual post a little bit, prefixing it with /posts/. I did that because otherwise we've painted ourselves into a corner: if someone writes a post called "last", we can't add an action called "/last" later without a conflict. It's always a good idea to keep your URL "namespace" clean so that conflicts like this won't bite you later.

"Hang on a minute! You wrote app.get(). But you're not getting something, you're setting up a route. This is very confusing."

Yeah, it is at first. Just keep in mind that what you're really doing is setting a route that responds to the GET HTTP method (the method used for plain old ordinary visits to webpages), as opposed to the POST HTTP method (used for submitting forms). I agree that "app.get actually SETS stuff" is confusing, but it will feel more natural to you after the first time you write"app.post".

Matching Parameters in Routes

So far, so good. But this first route is pretty trivial. What about delivering individual posts?

Now this is pretty nifty. Express provides us with a handy way to match all URLs that begin with /posts/ and are followed by additional characters corresponding to the placeholder :slug (the slug, or unique URL, of the post). And those additional characters automatically appear to our code as req.params.slug.

Did you catch that? Any component of the URL in your route that you prefix with a : will be captured and automatically become available in the req.params variable. That is very cool, and saves us a lot of monkeying around.

For this route to work we do need to make a small change to our data. In the first version of the app, the slug of every post started with a /. Now the / is part of what has already been matched by the route:

/posts/:slug

See that second /?

So let's clean up our posts array, removing the leading slashes from the properties:

We're almost finished! Two more things. First, let's create a route to catch nonexistent URLs and call the notFound function. We'll use the * wildcard, which matches anything after that point in the URL.

app.get('*', function(req, res) {
notFound(res);
});

It's very important to register this route last. The wildcard matches everything, so if you put it first, everything on the site yields a 404 error.

We'll also touch up our notFound function a little bit to send the right status code while still taking advantage of the simplicity of res.send:

function notFound(res)
{
res.send('<h1>Page not found.</h1>', 404);
}

Notice that we have added a second argument when calling res.send to signify that we are sending a 404 error. This is important because it allows Google and other search engines to distinguish this from an interesting page that should be indexed.

One last detail: listening for connections. Previously we did this with the server variable, but we've replaced that with the app variable. So we write:

app.listen(3000);

Now you're ready to fire up the app again. Make sure you are in the blog-2 directory (via the cd command), then type:

node server.js

Now you can connect to your app:

http://localhost:3000/

You'll see the same functionality as before, except that the URLs for posts have changed a bit.

]]>http://justjs.com/posts/classy-routes-with-expressclassy-routes-with-express
And It Really Does Something!

For our very first node app, we'll create the simplest blog ever. And I do mean simple! There's no database, no layout, no stylesheet, no template language to make it easier to render things... and all of that will change in later examples. But for now, this is a great way to explore the basics of creating a web app in Node, without resorting to a meaningless "Hello World" application. I really hate those.

To get started, create a folder inside your home directory called "node-apps", create a second direcory inside that called "blog-1", and finally create a new text file there called "server.js". And if you want, cheat: pop over to github to copy and paste the code. Or just follow along as I explain.

By convention, Node applications begin life in a file called server.js and starts executing at the top of the file. And the first thing we usually do in a Node application is pull in a few other JavaScript files that do useful things.

Unlike PHP, Node isn't already baked into a webserver like Apache. Instead, every node web application is a webserver, listening for connections from web browsers on its very own port. So the first thing we'll do is summon a module that implements the HTTP protocol for us. (HTTP, of course, is the protocol that web browsers and servers speak to one another.) Here's that first line:

var http = require('http');

This line finds the http module that came with Node, executes that code, and returns an object with lots of useful functions for us. We'll learn how to write our own modules later; it's a great way to organize and share code.

We're almost ready to create our webserver. But first, before we can have a blog, we need some posts to share on that blog. Later we'll fetch them from MongoDB, but for this first example, let's just create a little bit of data right in server.js:

We've created an object and assigned it to the posts variable. Each post is a property of the object, and the name of the property is the URL or "slug" of the individual post, a nice friendly URL based on the title. Every post gets its own friendly link. (Later on we'll see how to generate these automatically and make sure they are unique.)

Each post, in turn, is an object as well, with title and body properties. This gives us a convenient set of data to play with.

The createServer function creates a webserver. The createServer function expects one function as its argument. In this case I've chosen to define the function right inside the parentheses. If I wanted, I could just give the name of another function that resides elsewhere.

Functional Programming and Node

"Hang on, a function wants me to give another function to it?" Yep! Those of you who have programmed with jQuery are nodding along with me. Those who have not might be a little confused.

JavaScript is a functional programming language. That means we can define a function just about anywhere. And we can assign functions to variables and even pass them to other functions, just like any other data.

Node, like jQuery, takes advantage of this in a big way. Almost all important functions in Node accept a "callback function" to be called when something is ready, complete, broken, on fire, or otherwise in need of attention. And until one of those things happens, Node just keeps on trucking - delivering content you already queued up to be sent, accepting more connections, and calling other callback functions elsewhere in your code.

Node and jQuery share another important trait: neither one of them ever, ever, ever makes the user wait for JavaScript code to finish running. All JavaScript programmers soon discover what happens if your code takes a long time to run: the user can't do anything at all with that webpage until your code finishes. Eventually the web browser pops up a dialog box that says something like:

"This webpage really stinks. Do you want to make the pain stop?"

You don't want to be that guy (or gal, depending). So you learn quickly to use features like window.setTimeout() to do things in small portions, a little at a time. And you also learn to create functions that respond to events, like this slightly contrived jQuery example:

These are healthy habits you probably already have, and they will serve you well with Node too.

Let's take a look at our callback function: the code between the curlies that responds to each request.

The callback function takes two arguments: req and res. req (short for "request") represents what the web browser asked us for, and contains important stuff like req.url. res (short for "response") is our response to the web browser.

Did they ask for the home page ("/") of the site? Then we call index() to deliver it.

Did they ask for an individual post? Then we check whether there is a post with that URL, and then we call post(req.url) to deliver it. All we have to do is check whether posts[req.url] exists.

Did they ask for something ridiculous? Then we call notFound() to break the bad news gently.

Nested Functions

So far, so good. But where are the individual functions like index()? They are nested right inside the callback function.

"Whoa there, I can put functions inside functions?" Oh, yes. This feature is common to all functional programming languages, and notably absent from a number of other languages. It has many benefits. One of the most important is that your nested functions can "see" variables that are defined in the outer, enclosing function. Like the "req" and "res" variables.

Here's the index() function that renders the home page of the blog, with links to all the posts:

All this function really does is build a big string, 's', that contains the HTML of the home page. By adding to this string we build up the markup for the page. First the title and heading, then an opening 'ul' list tag, and then the 'li' tags and links for the individual posts.

This line loops over the slugs (URLs) of the individual posts in the posts object:

for (var slug in posts) { ... }

Then we can access each individual post object as posts[slug] in order to build up a list of links.

"Hey, isn't this a tedious way to build HTML?" Sure. I've seen worse, but it's not much fun to maintain this way. Fortunately there are alternatives, and we'll use them. In an upcoming post we'll look at Underscore templates, a great way to have our JavaScript cake and convenient templates, too. But at the end of the day, we're still building a response to send to the user. So in this post I'm emphasizing that simple idea.

Responding to the User

Once we have the entire page in 's', we're ready to send it to the browser. Since we do this more than once, I've written a function for it. Let's look at the sendBody function:

This function does two things: it sets up the headers of our HTTP response, and it asks Node to send the actual content.

This line tells Node what the HTTP status code will be, and what the content type of the response will be. We want status code 200 ("OK"), and the content type text/html (as opposed to text/plain, image/gif, et cetera):

res.writeHead(200, {'Content-Type': 'text/html'});

Then we ask Node to send the body of the page:

res.end(s);

Notice I said "ask Node to send the body of the page." I did not say "send the body of the page right this instant." Node will take the HTML off our hands and then start delivering it in the background until the browser accepts it all, then politely say goodbye to the browser, without further help from us. But Node does all this "in between" future callbacks to our code. While we're delivering responses for other URLs for other users, Node might still be sending our page to the first user.

Does that matter? It does if our code tries to do something slow! That's why when we do potentially slow things, like requesting data from a database, we'll always do it with a callback function that gets called when the operation is done... allowing Node to keep muscling along while it waits for the database to get back to us. That's why you can expect to see lots of callback functions. Fortunately, in a functional language like JavaScript, using callbacks quickly becomes friendly and intuitive.

Delivering a Post

Let's take a quick peek at the post() function. This is similar to index():

Our post() function takes a url as an argument, and builds a page much like index() does, after first fetching the appropriate post from the posts object. And we reuse the sendBody() function to ask Node to deliver it to the browser.

Breaking Bad (News)

All well and good. But what happens when the browser asks for a post that doesn't exist? Enter the notFound() function:

Sending a "404 not found" error isn't much different from sending a successful response. We can still call res.end() to deliver the text of the response. We just need to send status code 404 (Not Found) instead of 200 (OK).

Listening For Connections

We're finished- almost! We've created the server and provided a callback to handle requests from web browsers. But we haven't actually told Node to start accepting those connections yet.

Here's the call to start accepting connections on port 3000:

server.listen(3000);

And we'll send a message to the console (the terminal window, on our Mac) to make it obvious that the server is ready:

console.log("Listening for connections");

Starting Up Our Server

We have a complete web application! All we have to do now is start it up.

To launch your blog, first open a Terminal window. Then use the cd command to switch to the directory where you've put it:

cd node-apps/blog-1

Now use the ls command to list the files here. You should see your server.js file:

Macintosh-5:blog-1 boutell$ ls
server.js

You're ready to start your web app! Just type:

node server.js

You should see:

Listening for connections

Leave the app running in the Terminal window for now and visit it with your web browser. Just visit http://localhost:3000/ to see your blog's index page. You can click through to individual posts, or visit the URL of a nonexistent post to see the 404 error message.

When you're through playing with the app, you can shut it down by going to the terminal window and pressing Control-C to return to the terminal prompt. After that, you can get back to the node-apps folder by typing:

cd ..

And then:

ls

To list your app and any other folders you've created for node apps.

Things Just Get Easier (And Cooler) From Here

We've built our first node app! But we did it the hard way, a little bit. I wanted to be sure to give you a sense of how Node really works. Next I'll introduce modules that do some of the hard work for you, making it a lot easier to respond to different URLs and send back nicely templated webpages with more complete markup. Then we'll start storing our blog posts in a real database so we can create, edit and delete them... and so on from there, until we achieve total world domination together. Isn't that heartwarming?

Node is up and running already. So I could skip along and start showing you things node can do. But I have this terrible, old-school habit of writing apps that store stuff. And storing stuff requires a place to put it. And I like to be able to store that stuff close by, especially when I'm developing a new app and testing lots of changes. I don't want to wait for some far-away cloud database server to respond sluggishly to an app that's running on the Mac in front of me.

So before we dig into Node, let's finish assembling our new toys by installing the MongoDB database.

Why MongoDB?

First: MongoDB speaks JavaScript. Note the title of this blog. 'Nuff said.

Second: MongoDB is much faster than old-school SQL databases.

Third: MongoDB has a more robust set of features than most "NoSQL" databases. When you consider how flexible and friendly MongoDB is, it's really more of a "NewSQL" database. Folks who have experience with MySQL and other old-school databases will be more comfortable with MongoDB than with most other alternatives.

I would sum up MongoDB's personality thus: "we like delivering cool features. We just don't like things that are clunky, tedious, inconsistent and slow."

Installing MongoDB

For starters, head on over to the MongoDB site and click Downloads. You'll be given a choice of many versions. Too many! But don't worry, it's easy to figure out which one is right for your needs. Just look in the first row. This is the latest stable, production version of MongoDB.

Second, you'll see versions for two kinds of Macs: those running a 32-bit kernel, and those running a 64-bit kernel. And this is a problem, because many of us have no idea whether our Macs are running a 32-bit or a 64-bit kernel.

Fortunately there's a quick way to find out:

1. Click the Apple menu.

2. Pick "About This Mac."

3. Click "More Info."

4. Scroll down and pick "Software."

5. On the right, look for "64-bit Kernel and Extensions." If you see it, congratulations, you have a 64-bit kernel. Download the 64-bit version of MongoDB. If not, you have a 32-bit kernel. Download the 32-bit version.

A warning for ambitious folks: on a 32-bit system, a MongoDB database can't store more than 2GB of data. That's still plenty for developing and testing your app (I have a 32-bit kernel on my own Mac as of this writing). As long as you're not storing originals of photos directly in MongoDB or running a Facebook clone in production, this is unlikely to be a dealbreaker for you.

Configuring MongoDB

Now that you've downloaded MongoDB, you'll need to unpack the archive and move the software to an appropriate place where you can easily find it with terminal commands. No, it's not quite as convenient as it could be. But it's not hard, either. Just follow these steps:

1. Open the archive. It'll be extracted to a new folder inside your Downloads folder. The folder will be called something like "mongodb-osx-i386-2.0.5".

2. Move that folder to your home folder (the house icon in Finder).

3. For convenience, rename that folder to just "mongodb".

4. Create a new, empty folder inside "mongodb" called "data".

5. Open up your .profile file again (remember this from the previous post?) and add one more folder to your PATH environment variable. With the new folder added at the front, it'll look like this:

PATH=~/mongodb/bin:/usr/local/bin:/other/stuff:$PATH

(That tilde at the beginning is terminal-ese for "my home directory.")

6. Start up a new Terminal window. (An old window won't see the new PATH setting.)

7. Type:

mongod --dbpath=mongodb/data &

And press Enter. (I'm going to stop reminding you to press Enter after terminal commands by now. I'm sure you've caught on that it is always appropriate.)

MongoDB is now running in the background. It'll keep running until you restart your computer. If you want, you can add an alias to your .profile that allows you to restart it easily at any time:

alias startmongo="mongod --dbpath=mongodb/data &"

Now you can just type startmongo at the terminal prompt to start MongoDB at any time.

You shouldn't need to stop mongodb. But if you really want to, you can do so with this command:

killall mongod

Speaking Fluent MongoDB

So MongoDB is up and running. Big whoop. What can we do with it?

For starters, we can talk to it! Let's create a database and put some information in it. MongoDB comes with a convenient command line tool for use at the Terminal prompt. It makes it very easy to get started and just play around with Mongo queries.

Open a second Terminal window, stretch it to the width of the screen and type:

mongo myfirsttest

You should see:

MongoDB shell version: 2.0.5
connecting to: myfirsttest
>

We've connected to MongoDB and created a new database called myfirsttest, just like that! MongoDB just creates a database on the fly the first time you ask for it.

Now let's insert some data - how about a blog post? Try pasting this command at the > prompt:

We just inserted a new blog post "document" into a "collection" called posts. Notice we didn't have to create the posts collection. Or define a schema for a table. Nope, we just wrote some JavaScript! It's very simple JavaScript, too: an object with title, body and 'created' properties. The first two are just plain old text, and the third is a Date object. MongoDB knows what to do with all of them.

This is exactly what we inserted- well, almost! The _id property is automatically created for us by MongoDB. It will always be unique, so it's a convenient way to fetch this particular post again. And the created property was stored in a standard international format. We'll see how easy it is to manipulate and format these dates with JavaScript in a future post. (Actually, you can see it in action in the upper right corner of this post.)

When we don't want the post anymore, we can remove it:

db.posts.remove({ _id: new ObjectId("4fb318583cda994bf5f23726") });

Since we have just one post, we could have written:

db.posts.remove();

That command deletes all posts from the collection.

We could also have written:

db.posts.remove({ 'title': 'My first post' });

Removing documents by id, though, is a better choice if you want to be sure you are not removing any other documents with the same title.

Ladies and Gentlenodes, Start Your Engines!

That's it- we've installed both node.js and MongoDB, and we're ready to do some real work with Node! Beginning in the next post I'll start showing off real Node-powered websites and examining the code that powers the blog you're reading. There's nothing quite like autobiographical programming!

]]>http://justjs.com/posts/getting-started-part-ii-installing-mongodbgetting-started-part-ii-installing-mongodb
I've sung the praises of node.js, but I haven't yet explained how to actually bring this goodness into your life. Fair enough. Time to install node.

Famous last words maybe, but I'm going to devote just two short posts to the semi-interesting topic of installing things. It's all levitating outer space kitten games after that, I promise. So just bull through and you'll be glad you did.

I'm going to make a few assumptions:

1. You want to be able to try out node sites on your own computer. Yes, you could do everything in the cloud... and wait for Internet lag every time you want to test something. But would this completely defeat the purpose of using an environment as fast and fun to work with as node? Yes! It totally would!

3. You don't have Homebrew or MacPorts or any other heavy duty geek oriented command line environment that already includes Node or offers it as a package. Or if you do, you're skipping this post. We'll use the official, easy installers. (Protip: if you do install Node and MongoDB via MacPorts, be prepared to wait a really long time.)

4. You have a decent text editor, one that can edit system files if necessary. If not, go get TextWrangler, a very solid free offering from the BBEdit people.

So much for exposition. After this it's just a chase scene.

Installing Node (on a Mac)

2. A pkg file will download. When the download is complete, open it and run the installer. The installer will install two things: node itself, and the npm package manager. npm lets you painlessly install optional modules for node. Which is good, because almost everything is an optional module.

"Hang on, I need to adjust my what?" When you run commands at the terminal prompt, your Mac needs to know where to find those commands. By default, your Mac looks in a short list of official places. Installers like the Node installer try to be nice and not muddy those places with their own contributions, so they pop things in folders like /usr/local/bin. To find things there, you adjust your PATH environment variable to include that folder.

Open TextWrangler or your editor of choice and browse to your personal home directory (the little house icon). Now check the "show hidden files" box. This allows you to see files like .profile, your personal profile for newly launched terminal windows.

Also pull down the "Enable" menu and pick "Everything."

Now, if you see .profile, open it. If you don't see it, just create a new file; you'll save it as .profile.

Look for a line like this:

export PATH=/some/stuff:$PATH

If you don't see an "export PATH" line or you're making a brand-new .profile, you'll add your own:

export PATH=/usr/local/bin:$PATH

If you do see an existing line, just add /usr/local/bin: at the front:

export PATH=/usr/local/bin:/some/stuff:/some/other/stuff:$PATH

Now save your .profile (make sure you include that leading dot if you are creating it for the first time, and do NOT include any file extension like .txt).

Next, open a new Terminal window and type:

node

You should see:

>

If you still get a "command not found" error, check your work.

If you get the > prompt, great! Try this:

console.log("Hello World")

And press ENTER.

You should get back... oh all right, I think you can guess.

To exit, press control-D (isn't the terminal intuitive?) and then close your terminal window.

What's Next

You've got node on your computer! And so far we haven't done anything really cool. But that's coming very soon. In my next post we'll install MongoDB so we can actually store stuff. Then we're ready to move on to those levitating outer space kitten games (well... awesome web applications actually).

]]>http://justjs.com/posts/getting-started-part-i-installing-nodegetting-started-part-i-installing-nodeWelcome to the first installment of justjs.com, a highly autobiographical blog about full-stack web development in node.js, MongoDB and JavaScript!

I've known lots of web development languages. Trust me... lots. So why code in JavaScript? It begins with one word:

Frustration

This blog is motivated by my own frustration with the constant "context switching" that comes with the use of separate backend and frontend programming languages. And not only that, but also a profusion of configuration file formats and templating languages jostling for space in my skull.

Taken individually, they all add value. Collectively, they make programming a frustrating exercise in remembering the rules of the environment you're in at the moment.

Meanwhile, all of these tools slow everything down! In the name of making it friendlier, speeding it up, or both. In the name of making things friendlier, many web frameworks and programming languages sacrifice performance. And in the name of speeding things up again, many frameworks try too hard to cache everything magically behind the scenes - creating tools that are fast in production but painfully slow in the "dev" environment.

Motivation

These are gripes. Gripes don't normally motivate people to tackle entirely new paradigms.

But for the past several years I've watched the web browser vendors compete tooth and nail to make JavaScript faster and faster and faster. I've watched as the JavaScript"just in time" compilers exceeded the performance of just about all other loosely-typed, fun-to-use programming languages. And pushed beyond that to a level of performance that's tough to match in anything but C, C++ or Java.

When you consider how fast the language is today (very!) relative to the sacrifices you have to make for that speed (none!)... and when you consider that the best implementation is open source... you quickly realize that node.js was a revolution waiting to happen.

Excitation

Why? Because when I started coding in node.js, I had to patch the tools I use to measure how fast a webpage is. I had to patch them to display more decimal places in order to measure any time at all.

And because I don't spend all my time switching contexts in my head. With node.js on the server, MongoDB (with its JavaScript-based query language) for storage and Underscore's templates, I get to code in one powerful, functional and fast programming language. And in exchange for that tremendous convenience, I sacrifice...

Nothing!

Starting to see why I'm excited?

What You Can Expect From This Blog

In each installment I'll be breaking down another part of the code for you, taking you through the evolution and continued future development of justjs, demonstrating how to tackle web development tasks that are nontrivial but familiar with the power and velocity of Node.