Rapid7 Blog

Unleash the power of node.js for Shell Scripting (Part 2)

POST STATS:

SHARE

Ready for our first proper node.js Script!

In a previous post, we learned about some tools that helped us create a script in node.js. It is now time to put this into practice by implementing a script that connects to a few online newspapers, searches in the news for specific keywords and returns those articles.

Our new script will need to accept the following parameters:

A file with the list of newspapers (one URL per line)

A file with a list of keywords (a keyword per line)

First, let’s create the following files: news_watcher.js and package.json. Make sure you remember to add the execution rights to your file. We will use three external modules and make sure they are added to our package.json (see Part 1 for details).

As you can see, we will use the Cheerio, Request and Commander modules. You already know about Commander (see Part 1 if you don’t). We’ll use Request to easily access content from URLs (based on a callback function). Finally, Cheerio is a great library that creates a DOM from a string and allows you to use some JQuery functionalities from then on. It can be very useful to manipulate HTML and web scraping.

Conclusion

If you are need a scripting language to run from your command line and you feel strong using Node.js, I think I have given you a really interesting option to create, run and even share your scripts. Practically speaking, I use this a lot to create quick tests and benchmarking scripts as I know I can leverage javascript capabilities fast and bring complex algorithms to my shell.

The other great aspect is to be able to use all those external modules: npm oficial page.