Jekyll, YAML Front Matter, and the Liquid templating system

November 25, 2011

During the sprint to get Popcorn.js 1.0 out the door I was tasked with getting our documentation up to date with the current API and making sure everything was in a working state. It involved some wordpress nonsense, lots of time contemplating my wording ( which is also a work in progress ), and countless days creating small demos showcasing each part of the API. In the end we had a functional set of docs that everyone could use. They weren’t that pretty, didn’t have a table of contents, and were a pain to navigate through, but they did what they set out to do. We wanted to make the docs even better then our first stab at them, and thought it would be awesome if we could have them on github. Having the docs on github would mean making future alterations simple, meaning anyone with git could send in a pull request in order to make changes. This also meant that we could leverage the power of various templating libraries and order to make our docs really shine. This is what I have set as my goal for 1.1 of Popcorn.js, make the docs awesome.

To begin I talked a bit with Jon Buckley about what he had recently done with the Processing.js website and documentation. They recently migrated there website over to github pages and Jon explained that it was actually quite a pain free process. He showed me what he created and it looked pretty stellar. The best part about this was how easy it would be to make future changes. I think one of the main reasons that people hate documenting stuff has to do with the tedious nature in doing so. You have to get access to the web-server, which usually involves getting permission from someone, and then you can begin writing the docs. Using git instead someone could add a completely new function to the current API, write unit tests for it, and document it all in one pull request. Jon also showed me some of the templating stuff that he was utilizing, namely Jekyll, YAML Front Matter, and the liquid templting system. You can do some pretty amazing things with the 3 of these. Jekyll is described as the following:

Jekyll is a simple, blog aware, static site generator. It takes a template directory (representing the raw form of a website), runs it through Textile or Markdown and Liquid converters, and spits out a complete, static website suitable for serving with Apache or your favorite web server. This is also the engine behind GitHub Pages, which you can use to host your project’s page or blog right here from GitHub.

YAML Front Matter allows you to specify some predefined variables in your page, as well as access various pages across your site ( if you set up your directory structure correctly ). Jekyll interprets files with YAML data in them as special files and all data gets processed as the page is read. Using YAML with the liquid templating allows you to create some pretty amazing things, such as essentially define your table of contents based on your directory structure, create default templates to be used site wide, and use unique Liquid tags and filters to define which content goes where on your page. So what does all of this mean? Well it means that having our docs live in git, and using a libraries like we are, we are able to allow users to push there doc changes to git, have them be read in by our templating library at compile time, and then generate the content on the pages accordingly. This means once we have the pages set up, its simply a matter of updating our git repo to have our doc changes appear.

So 2 days ago I finally began the overhaul of the docs. I began by creating my directory structure according to how I wanted the docs to be laid out. I created a folder for each major part of our API, Static methods, instance methods, instance properties, media methods, plugins, players, parsers, effects, and modules. Each of these had there own folder which each had a sub-directory called _posts. Jekyll walks your directory tree and looks for folders called _posts and knows that this is where various posts will leave within the site. What this means in the context of our docs is that each “post” will be a doc for different part of the API. The next thing I needed to do was change the naming structure for each of our markdown files. Jekyll expects the files to be named in the following fashion: YYYY-MM-DD-fileName. I took this as an excuse to write a bash script ( as I don’t get to write enough of them ) that would do what I needed. After this I created an index.html page that had some YAML Front Matter defined at the top ( to let Jekyll know that this is a special file that needs to be processed ) as well as some liquid code to read in various categories ( defined by our directory structure ). This is when my day long battle with liquid and YAML came in. Im not gonna lie, I went down a bit of a rabbbit hole here, but refused to except what I was told by the internet, that I couldn’t do what I wanted to.

So what did I want to do that was supposedly not possible? What was this psycho idea that I had that YAML and liquid just could not handle? Sort. An. Array. Yes, you heard me, I wanted to sort an array and this took me the better part of a day to get working. So to begin, to access the categories of your site YAML provides you with a global liquid variable called site. On this variable we can access the array ( or what I thought was an array ) of categories. Sounds typical, so I attempt to loop over those categories and print them out. Well you see, if you wanna do that, you also get all of the categories posts as well ( makes complete sense right? ). Hmmm, well, I can work with that, I’ll just take the first word that they give me, as it was the name of the category, and print that out. Awesome, that worked. So it seems that the output isn’t sorted, well I’ll just call the sort filter on the array. Oh crap, I just took the first word off of that array of categories, so that means all those words that I just stored using a capture, are not longer an array, but just one big string seperated by spaces. So turns out I can’t use the sort filter. This is when the googling began. All searches came up with “you can’t do this”. Take a look at this stackoverflow question, where the person who posted wanted the same thing as I did, and the accepted answer was that it is not possible. I refused to accept reality. This must be possible. How can I not sort my categories? I spent the following hours hacking with liquid, battling it every step of the way, but in the end, I made liquid, YAML, and Jekyll my B***h. It did what I wanted, whether it liked it or not. My final code looked like the following ( tho to get there I hacked with probably 50 or so lines of code to make sure everything was working ):

In the end I probably spent more time then I should have on such a small issue, but I proved that it was possible. The next steps for the docs from this point are creating some small YAML files in each of the categories folders describing what each one is about. Then i will create an index.html file in each of those directories that will a table of contents for each category, linking to each item. Im also planning on creating a template that that each file will use, so I only have to define the YAML and liquid filters for each file. Reinforcing my earlier point, and part of the reason I was so adiment on getting the sorting working, was that if this is set up properly everything will just fall into place. When someone updates the docs, everything will just work. You add a new category? Just works. Add methods to the API? Still works. Deer driving a car? Nope. Chuck Testa. But thats a whole other blog on its own. Still quite a bit of work to do in the coming weeks, but it is going to be awesome when it is done, so expect an updated blog post on my progress in the coming weeks before Popcorn.js 1.1!