Oct 26, 2013

How We Made KettleCorn: NodeJS, CKEditor and Bing’s Translation API

The Office of Digital and Design Innovation (ODDI) development team always looks for ways to empower journalists to tell their stories with ease and elegance. The 1.0 release of the PopcornJS standalone JavaScript library at MozFest 2011 immediately intrigued us because it enabled anyone on our team who was comfortable writing JavaScript to annotate the organization’s audio and video productions with content from a wide variety of internet sources.

As developers, it was very easy for us to understand the power and flexibility of PopcornJS, so we wanted to get involved. We got our feet wet contributing by building a PopcornJS Storify Plugin. We had high hopes to use PopcornJS early and often. However, it was difficult for us to sell PopcornJS outside of our team because of the required programming skills.

We closely followed the development of Popcorn Maker, and we even set up our own instance of one of its alpha releases. When version 1.0 of Popcorn Maker shipped at MozFest 2012 we were thrilled because we hoped it would solve the ‘coding required’ problem. But as KettleCorn product owner Brian Williamson recently explained, we weren’t able to reach the level of adoption we were hoping for a number of reasons, so we began developing a version of Popcorn Maker tailored to the needs of journalists: KettleCorn.

Brian recently outlined KettleCorn’s rich featureset and I’ll use this opportunity to share some of the interesting technical experiences we had when building some of those features.

Rich Text Editing

We chose the open source CKEDITOR as our WYSIWYG editor. We wanted it to be optional for any given text field, so we added an “editor” option to the plugin manifest. There were a couple of interesting considerations to make this work:

Popcorn Maker sanitizes text content because it doesn’t allow HTML tags. We had to disable that in cases where the field had a CKEditor in use for input.

Popcorn Maker allows on-stage editing for its text fields, but doing so was causing us to lose the formatting that we were doing in the CKEditor. Due to the elegant architecture of Popcorn Maker, this was pretty simple and could be done on a per-plugin bases. We simply commented out calls to EditorHelper.contentEditable()

Translation

We looked at a number of machine translation API’s and settled on Bing Translator because Microsoft provides a free plan that allows for up to 2 million characters per month of translation.

We then set about writing the code to allow us to consume their service. First, we retrieved the list of available languages via a one-time call to the GetLanguagesForTranslate endpoint. At that point, all we needed to do was be able to use their Translate endpoint on a regular basis within the KettleCorn authoring environment. We tried the following approaches:

Microsoft’s AJAX examples were quickly ruled out because they require ASP.NET.

Microsoft’s PHP Client is excellent and we quickly made it work, but running PHP and NodeJS on port 80 on the same server could prove to be difficult.

We found our final solution when we stumbled upon Kenan Shifflett’s MsTranslator NodeJS library on GitHub. We installed this in KettleCorn’s node_modules folder, added a route for /translate, and accepted POST parameters for the phrase and target language.

GoogleMaps+ Plugin

If you click “publish to the web” on a Google Spreadsheet, you are presented with a variety of options for what format you want to retrieve a public link to. We were hoping that we would easily be able to consume JSONP (to avoid any CORS issues), but that option isn’t offered via the public interface. After digging through a number of blog posts and threads about the topic, we found our solution. Every Google Spreadsheet has a key that you can parse from its URL, and you can use that key to obtain JSONP as follows:

We didn’t want to require authors to format their links, so we allow them to enter a normal spreadsheet link, and then we parse the value of ‘key’ from the query string and plug it in. The only downside to this data is that it doesn’t seem to reflect the values from the first column, which is why we have an entire red “do not touch” column in our spreadsheet template.

Lower Thirds Plugin

Our Lower Third plugin allows the user to display an entity logo. In the context of the authoring environment, we wanted the default logo display to be based upon the user’s organization. However, someone who is viewing the published project on a website will not be logged into KettleCorn. We ended up checking for the existence of the window.Butter variable and setting the value of the logo based on the user’s profile when it exists.

End Card Promo Plugin

Getting the end card “90% done” happened pretty quickly, but we wanted the thumbnails and text to be responsive. We initially attempted to do this using just CSS, but getting the layout and images to scale the way we wanted ended up being difficult, so we settled on a table with three equally sized cells.

User Profiles and the Project Showcase

We added a table for user profiles to enable us to customize the presentation and save preferences on a per-user basis. If we were using PHP and MySQL, querying the database for all projects authored by a user would be as simple as writing an inner join and iterating through the results set. However, Popcorn Maker uses Sequelize as an ORM for data persistence, which greatly simplifies data access from NodeJS. Unfortunately, it wasn’t immediately obvious how to join our two tables together using Sequelize. It’s clearly something that can be done based on the documentation, but we ran out of time researching it, so we had to settle for a limited functionality in the showcase for our release, and we will revisit this functionality later.

There’s More!

We would be more than happy to share more technical information and experiences with any interested developers. Feel free to tweet us @bbgKettleCorn or leave a comment below!