Since I last posted, we’ve been making development progress on a few different fronts. Much of this work is still in feature branches, and has yet to be merged into the main development branch. Therefore, I’m going to skip the full list of revisions in this post, and instead offer a high level description of our recent efforts.

Markdown and GitBook Support

Fundamentally, Manifold has been conceived as a tool for university presses. When it comes to thinking about how it will be used, UMNP has guided our thinking about what problems presses are trying to solve, and how Manifold can help. That said, I think everyone on the project has also been interested in the ways in which Manifold can be a useful tool for the broader academic community. At the very least, we want to make it possible for individuals to find ways to use Manifold. One impediment to this, in my mind, has been the fact that to date Manifold only accepts finished EPUB documents as an input format, and independent scholars and self-publishers may find it difficult to come up with finished EPUBs.

EPUBs are essentially packages containing HTML, CSS, and Javascript. In a sense, an EPUB is a small, self-contained website, and EPUB readers are offline, limited browsers. HTML, therefore, is already an input format that Manifold understands and can work with. What this means is that any document type that can easily be converted to HTML can likely be consumed by Manifold and made available to readers.

We refer to the process of slurping a book up into Manifold as “Ingestion.” Manifold can be extended by adding ingestion strategies to it. An ingestion strategy tells Manifold how to map a source document—eg, an EPUB—to a format it can understand. In short, we’ve anticipated that we’ll want to ingest other kinds of documents and have left places where other developers can create their own strategies. With the aim of testing out this approach and also furthering our goal of increasing Manifold’s user base, we spent a couple days writing a Markdown strategy that supports the GitBook format. This means that users will be able to feed Manifold a set of nested Markdown documents, which Manifold will then be able to make available online, just as it does an EPUB. Manifold’s documentation, for example, is authored following the GitBook format, and it can now be read within Manifold itself. Three cheers for eating our own dog food!

Backend Development

Since I last posted, we’ve also been hard at work at building out the backend interfaces. Most of the basic backend HTML and CSS has been completed, and we’re now in the process of developing reusable patterns and components in React to wire up the backend forms to the API. This has been challenging, as there is a lot of functionality required in Manifold’s admin interface, and figuring out how to model the requisite state while keeping in our Flux architecture has required some thinking. We’re close on this, with some initial proof of concept work, which I’ll be discussing more in subsequent posts.

That’s it for today! We’ll be posting more updates (hopefully with screenshots and some design comps) later this week.

Thanks for joining me for another installation of “This Week in Manifold.”

First off, please accept my apologies for missing last week’s update. When last Friday rolled around, we had a number of updates ready to go out to our staging site. However, when it was time to deploy, we hit a few unexpected problems.

DevOps, Hosting Manifold, and Deployment

This release adds a new metadata field to projects. Because the exact metadata requirements are difficult to pin down and may change from press to press, we wanted a relatively flexible approach to storing project, text, and resource metadata. Back in the day, we might have used an EAV model for this, but there are many well-documented drawbacks to that model. Because Manifold uses PostgreSQL as its database backend, we have access to a variety of field types for storing arbitrary amounts of data. In the past, we’ve used HSTORE for storing key/value pairs, but in this case we wanted something more flexible that could store different data types (HSTORE stores strings, but we want to store integers, boolean values, strings, arrays, etc). Recent versions of PostgreSQL include the JSONB field type, which is exactly what we want in this case.

Alas, that meant that the October 28th release required some infrastructure updates. Sure, we could have shelled into the staging server and done a quick upgrade, but that’s not how we roll on this project. It’s really important to us that it be possible to easily provision hosts for Manifold, which means automating as much as possible. As Manifold grows, it relies on more and more services to run, which makes setting it up increasingly complex. With this release, Manifold needs all of the following configured on the server to run correctly:

Ruby 2.3 (for the API)

Node 4.5 or higher (for the client)

Postgres 9.4 or higher (for storage)

Redis (for queuing background jobs)

ImageMagick or GraphicsMagick (for resizing images)

Nginx or Apache (for serving the API and the Client)

Moreover, in addition to having all these requirements in place, the server also requires that a number of background processes are always running. As of right now, there are four custom services (excluding the webserver and the database server) that need to be running on the host: 1) manifold_api (Rails app served by Puma), 2) manifold_client (Node app), 3) manifold_scheduler (Ruby daemon), and 4) manifold_workers (Sidekiq). Keeping four services running on a server (or on multiple servers) is not without its complexities. If we run these on Ubuntu Trusty (v14), for example, we need Upstart scripts that manage the services. If we run Manifold on Ubuntu Xenial (v16), then we’re going to need Systemd scripts. If it runs on older versions of Centos or Redhat, then we’ll probably need some init.d scripts.

Running a complex app with separate parts is tricky. Internally at Cast Iron, we use puppet and foreman to manage these services. Needing to support Postgres 9.4 got us thinking about deploying Manifold to Ubuntu Xenial, which in turn led us to revisit our manifests. Once we went to Xenial, we had to rewrite our startup scripts for systemd, which in turn revealed problems with how we were storing secrets in the environment. In short, instead of spending the last week working on project resources and feature development, we spent it paying down some technical debt on the infrastructure side, and improving Manifold’s devOps and stability.

That’s done now, and we’re deploying Manifold to Xenial and fully managing its environment and all dependencies via Puppet. Revisiting this infrastructure has also given us an opportunity to reflect further on Manifold’s installation story and continue discussions about how we can make Manifold easy for other presses to adopt and install.

New Features

We did manage to slip a few important features into this week’s release. The full list is below, but there’s one in particular I want to highlight.

This release includes a scheduler that runs constantly in the background while Manifold is running. One of the things this scheduler does is trigger a background job for fetching tweets related to projects in Manifold. That job currently runs hourly.

Editors can use twitter search terms to describe what kind of tweets Manifold should pull in for a given project. This gives editors quite a bit of flexibility. For example, Manifold might watch the author’s twitter account for all tweets with a specific hash tag. Or, it might watch twitter in general for all tweets that include a given hashtag. Our tweet fetching service accepts multiple search terms, so it’s possible for a project to follow multiple accounts and hashtags.

These tweets show up on the project activity feed. Readers can browse through the historical activity for a project and see the events—texts being added, tweets about the project, new resources uploaded—that took place as the project became a published book (see screenshot, below). Our hope is that showing this activity helps make public the iterative aspects of authorship.

Well, that’s enough for today. I’ll be back next week with another update. Enjoy the weekend!

Refactoring

Display project “makers” (authors, editors, etc) on one line

Improve header/icon lockup on project detail component

Refactor ingestion resources into a new model that is distinct from project resources

Clean up API binstubs

Features and Improvements

Add JSONB metadata field to projects; adjust project import mechanism to include metadata

Expose project metadata in API

Render project metadata in the client

Incorporate Sidekiq for background job processing

Add the concept of model ownership in various places. Manifold will now track who creates a project.

Welcome to a new regular feature on the Building Manifold blog: “This Week in Manifold.” As we move toward a beta version of Manifold (scheduled for March, 2017), I’ll be releasing a Manifold build to our staging server every Friday afternoon. Each release will be accompanied by a blog post explaining what the development team accomplished in the sprint leading up to the release. This post comes a few days late. When the Internet broke last week, our ability to release builds from Github was impacted, which delayed things a bit. I expect we’ll be back on schedule this week, with our next build going out on Friday, October 28.

Nota Bene: Our staging instance of Manifold is currently on a private server. However, we’ll be opening it up to the public in a few weeks, as soon as we’re able to load an initial set of texts with open access rights onto the platform. Once that happens, it will be possible for readers of this blog to check out our weekly builds for themselves.

Our last development sprint focused on tightening up existing code, mainly around the project detail view. Notable changes included the following:

In addition to these changes, the Manifold team also touched base last week during our recurring (bi-monthly) meeting. The group discussed text and project metadata requirements, the overall project roadmap, the agenda for our in-person Portland meeting in mid-November, among other things. Joining that call for the first time was JoJo Karlin, a Doctoral student in English at the CUNY Graduate Center who is working with Matt Gold on this project. You’ll be hearing more from JoJo in the coming weeks as she works to share Manifold updates and progress via this blog.

This week’s development sprint will be focused on a few main goals. Naomi will be finishing up markup and UX for the resource collection slideshow. Zach will be focused on the initial modeling of project resources and project resource collections in the API, and wiring that data up to the client. He’ll also be improving the project importer—delivered in last week’s release—to pull in sample resources for our demonstration projects and texts.

All work and no play makes for dull developers, and so I will leave you with a screenshot of Manifold’s new error screen (only visible in development mode), inspired by the first computer Zach ever owned, the illustrious Commodore 64.