My Digital Workflow

A friend is beginning a book project and asked me about my basic research-to-writing workflow. This is something I’ve spent a lot of time refining over the years, especially as the technology is always changing, and I think I’ve got a system more-or-less in place as I wind down my PhD thesis.

I thought I’d share what I’m doing now. Hopefully this is of use to other authors and postgraduate students.

Organizing Reading Material

“Read it later,” like Pocket and Instapaper, services are crucial. I switched from Instapaper to Pocket last year because of Pocket’s multimedia support, but I miss some of the extra features of Instapaper (like controlling margins) and am convinced Pocket forgets to sync some of my read items. I’ve now switched *back* to Instapaper for one big reason: they’re the first to incorporate highlighting and syncing. It’s not free, but it’s a small price to pay for the convenience if you’re in this line of work.

Both support tagging and a great way to read anything available online that isn’t a book or PDF. They’re especially useful if you use Twitter and Facebook and frequently come across things you’re interested in, but just don’t have time to read just yet. Add appropriate tags when you send pages to Pocket so you know what is work related (and how, specifically) and what’s not.

Admittedly, I haven’t been good at this and now have an unorganized list of roughly 500 articles to “read later.” If a specific article is important, you can star/favorite it and it goes to its own folder. You can set up IFTTT to then move that somewhere else. For instance, I use IFTTT to send a link from all of my starred articles from Pocket to Tumblr, where I then use Tumblr’s RSS to offer Comparativist visitors an updated list of recommended reading. You could send those articles to an Evernote folder just as easily.

As much as possible, I stick with reading PDFs that have optical character recognition (OCR) or Amazon Kindle books. The reason being that those are the two formats that make it easy to highlight and annotate books and articles and extract them easily later. You would be surprised how many books can be found online by searching for the “[book title] PDF” in Google. Even though it doesn’t look as good, I prefer reading a book or report as a PDF rather than non-Amazon reflowable text. Adobe Acrobat Pro can convert most non-OCR PDFs to PDF.

My primary reading material is in PDF format, which I’ve been organizing by subject in a Google Drive folder since beginning my PhD . For instance, I have an Education folder, a Nonformal Education subfolder, and a further subfolder containing all the PDFs from a recent CERC book on nonformal education. More recently, I’ve been organizing by a “People” folder as I’m focusing more on the work of (and responses to) specific people rather than sweeping up literature on a specific theme Generic Viagra http://hotcanadianpharmacy.com/drugs/Viagra+Generic/ pills. I’m more inclined to want to pull-up “Polyani, Karl” than “Economic History.” Your mileage will vary – the key is to organize your folders and sub-folders based on how you mentally organize the issues.

I looked at using iCloud about a year ago, but at the time there was no way to have files mirrored on the hard drive in specific directories. Right now, I’m using a mixture of Dropbox, Box, and Google Drive. Google Drive is for bulk storage, important work goes into Dropbox, and my PDF library is on Box.

EDIT: I’ve quit using Box because there’s some file types it refused to sync. Specifically, it wouldn’t work with Scrivener. Dropbox now has competitive pricing, Google is still amongst the cheapest and most functional, and I hear decent things about SkyDrive.

Another key feature of cloud services like Dropbox and Google Drive is that its very easy to share entire documents online, without having to attach them as a file in an email. I like being able to send a single link to a book that I recommend to somebody. When I was writing a book last summer, I was able to have a “nightly build” that updated at the end of each work day that my colleagues and publishers could look over. All I had to do was save my work on my desktop and Dropbox did the rest.

Scanning / Digitalizing the Analog

I’m to the point that I almost won’t read anything that isn’t electronic now. First, I don’t like writing up in the margins of books or highlighting. I’ve always felt like I’m desecrating my book when I do that. Second, I *always* turn my library books in late and rack up fines (I live almost an hour away). Third, I like to have my books with me everywhere to share or read.

So I’ve been using iPad and and Android scanner apps to scan books at the library (and even some I’ve bought). I tried a few iOS apps and settled on ScannerPro. I primarily use my iPad because taking hundreds of photos and converting them into PDFs is a big battery drain on a phone. My advice is to take the photos first, then process them in ScannerPro. It will cut down the time needed in the library by more than half. Otherwise, you’ll be asked to choose a crop after photographing each page. While standing, hunched over a book. Adobe Acrobat Pro can convert most non-OCR PDFs to PDF. I won’t tell you how here, but it’s fairly easy to crack a full copy of Adobe Creative Suite.

Another place to digitalize the analog is in basic note taking. Evernote really shines here and is one of the places I use my phone as much as my iPad in my workflow. First, it keeps every lecture, meeting, class, seminar, and interview note I’ve taken in the past two or three years. I can pull up my notes from almost every seminar and conference I’ve been to, as well as almost every meeting I’ve had with my thesis adviser. My only regret is not using it more.

Second, it’s an idea scratchpad – the kind of ideas you get when taking a walk, or right before you go to sleep. I have a little folder called “DissIdeas” where I’ve been dropping these for the past month. Evernote has a lot of neat tricks up its sleeve. For instance, if you open a note at the same time is scheduled on your mobile (iOS or Android) calendar, that becomes the title of the note. Your “Meeting With Mark” is defaulted as a “Meeting With Mark” note if you create a new note during that timeframe. I’d also note Evernote is where I’m storing business cards now. Their OCR technology is quite good and can search the text of the photo of the cards.

Reading Hardware I Have Known

Today I read the PDFs (and books) on my iPad Mini. I’ve owned an iPad 2, Nexus 7 (2012 model), and two versions of the Kindle (Kindle 3 and Kindle DX). Reading PDFs on the Kindle was an awful experience and it didn’t interface with cloud services like Dropbox, even the models with worldwide 3G. Kindles are great for reading Amazon-purchased books but little else.

I think tablets are a better platform, despite legitimate worries about screens and eye strain. The normal-size iPad, though, is just took bulky and heavy for comfortable reading, and the low resolution of my iPad 2 bothered me after a couple days with the Nexus 7. The Retina iPad 3 and 4 are worse, in some regards, because they’re almost the same weight as the original iPad. The problem is that all those extra pixels require twice the battery size to keep the same amount of battery life.

Both the 2012 Nexus 7 and iPad Mini hit the “sweet spot” for me in terms of size and weight for reading. By all accounts, the 2013 Nexus 7 is even better than the original model I used, and I think it’s a great option for anyone on a budget. That said, it’s worth noting that when I broke my Nexus 7, I sold iPad 2 and bought an iPad Mini over the Nexus 7 or Retina-screen iPad. The screen is sharp “enough,” it’s extremely light-weight, and iPad apps really are generally better. I think the iPad Air might hit a sweep spot of size, weight and screen resolution.

The key to reading without undue eye strain on LCD and LED screens is keep it dim. GoodReader has an option to make the whites darker when reading PDFs, and all of the “read it later” apps (Pocket, Instapaper, etc) have a dark-on-white theme for night reading that I use as a default. I would recommend looking at Samsung tablets if eye strain is a serious issue for you. Samsung uses AMOLED screens, a technology that doesn’t use a backlight and can produce pure black (e.g., the black pixels are simply turned off).

As reading relates to phones, I don’t use it my Galaxy Nexus for reading PDFs even though it has a comparatively large screen. I only use it for reading reflowed text (like Kindle books and Pocket). That said, half the pleasure of using the Kindle app platform is that I can put down a book that I’ve been reading on my iPad at home and pick up where I left off on my phone on a bus the next day.

Creating and Highlights and Annotations

For years, I had used GoodReader to read my PDFs in on my iPad. I’m using a mix of Readdle and Mendeley now. Readdle does everything GoodReader does but I think it’s faster – and it’s free. Mendeley is problematic (especially with getting highlights out), but oh-so-close to exactly what I’m looking for. I was a fan of Montano Reader for Android on my Nexus 7, and would use that if they offered it on iOS. GoodReader and others have some advanced syncing options, including using networking protocols connecting your desktop/laptop over the WiFi network, I don’t recommend it. I’ve had two or three situations where my entire library stopped syncing because of a problem with a single file (often a file name). I’ve had to then delete all three gigabytes of PDFs and start over.

If you’re going to use syncing, I recommend syncing individual, smaller folders rather than the entire library. Better yet, open the Dropbox app and download the PDF there, then “Open in” GoodReader. When you’re finished with the document, export back to Dropbox by “Open[ing] in” again, overwriting the original file. Another option is to have two seperate folders, one called “To Read” and the other “Annotated.” Sync the “To Read” files regularly, then push them out to the “Annotated” folder and sort them later.

Amazon-purchased Kindle books automatically upload highlights and annotations to the cloud and can be accessed at kindle.amazon.com, “My Highlights.” A much more difficult operation is getting highlights off a book/document that you emailed to Kindle as a personal document. Though Amazon will sync your highlights and book progress across devices (go to the Kindle website, “Manage My Kindle,” then “Send to…” the appropriate devices), it provides no options to extract these highlights or notes without a physical Kindle device (where they’re saved in a text file in the root directory).

My way around this has been to manually flip through the book when I’m done, and use either the original document or Google Books to search for two or three words in the sentence I had highlighted, and then copy the text into Evernote. In practice, this has taken me about ten or twenty minutes to extract all of the highlights from a single book.

Instaper is now the best way to highlight, and annotate web pages. Diigo does something similar, but Instapaper’s new premium functions were exactly what I was looking for. Use IFTTT to automatically sync all clippings to Evernote, or post to Tumblr, etc.

Organizing Highlights and Notes

Once I’ve finished a reading a PDF, I open it with Skim on my MacBook. Skim has an option to save all the highlights and annotations, which you can then export as a text file (there should be similar software for Windows, and I know all Linux PDF readers should have these abilities). I take these, and my Kindle highlights and notes, and paste the text in aEvernote notes. Evernote lets you organize by folders (Notebooks) and tags. My own organizational style relies more on folder/sub-folders than tagging, and Evernote’s search function lets me look for specific words anyway.

I have a Notebook called “Quotable Quotes” and subdirectories inside that I’m still thinking through and organizing. A primary sub-notebook right now is “Book Notes.” You can see the organizational structure with the two screenshots below. You’ll note that some of the sub-folders aren’t very full. This is a system I’ve only gone full-swing into since this summer. Far too many of the quotes I’ve used in my thesis were pulled ad-hoc from the document, without being saved anywhere else. A lot of this has to do with me using Kindle readers in my first eighteen months of my PhD, when I hit the literature hard, but couldn’t wirelessly sync PDF highlights and annotations.

Writing Software

I am sold on Scrivener. It’s a brilliant word processor and it truly feels like it was designed by a writer for writers. What the developers of Scrivener “get” that no other word processing software developer “get” is that (a) writers often need to be looking at other documents while writing and (b) that writers approach their work in bits and outlines rather than writing start-to-finish as a whole.

Two general points about starting Scrivener: first, its a little outside the comfort zone you’ve known all your life concerning what a word processor is supposed to look like. Force yourself to use it for a small project, try to use as many of its features as possible. Second, watch some introductory videos on YouTube so you can see what those features are and how to access them.

Scrivener fits into my general workflow described above because I can easily take relevant texts – highlights, notes, and annotations – from Evernote and copy and paste them into Scrivener’s “Research” section. I can then work with a split screen, looking at my quotes and highlights in one place and writing in the other. PDFs also work

Similarly, I copy in my Evernote “scratchpad” notes to their relevant sections in Scrivener’s “Document Notes,” seen in the inspector. I use this as a scratchpad in its own right while writing, where previously I would dump these thoughts (and quotes) into a Word document section and give it a different font. It’s also easy to put relevant quotes into the same box.

Of my two major concerns about Scrivener, one is that I haven’t quite figured out how I’m going to do references for my thesis. I’ve heard people have it working with Papers, but I wasn’t able to figure it out. I’ve been manually inserted them by copying and pasting APA styles from Google Scholar and Mendeley, but that’s clearly not a good strategy when there’s two or three hundred references that need to be sorted in a specific way. I expect to build my final bibliography by bringing the final document into Word. The second concern is that there’s no good way to access Scrivener writings on the iPad. Yet.

As per other word processors, I find myself using Google Docs whenever I need to collaborate or show somebody a draft document. Increasingly, I like the interface and UI of Apple Pages (a lot more than Docs). Basically, Docs has more capabilities and Pages looks a lot better. Pages is probably the better of the two if you’re already invested in Apple’s ecosystem.

In so much as traditional word processors go, I think LibreOffice (and OpenOffice, from which it forked) is just as powerful as as Word and I recommend it for older computers or for people who would otherwise pirate Office. The only place where it fails, a little, is when opening compile Microsoft documents. That, however, is an issue that Microsoft intentionally created. Personally, I tend to use Word more than LibreOffice.

Bibliography Managers

I’ve given EndNote a shot, played with Zotero, but stuck with Mendeley. It’s free, it more-or-less “just works,” and it has Cite While You Write plugins for Word and LibreOffice Writer. It’s pretty good at scanning a PDF (or an entire PDF library) and finding the rest of the metadata pretty quickly. Often adding or fixing text in a field it made a mistake on (like author’s name or the full title) is sufficient to get the full metadata with a second search. What I like most is the web importer, which I don’t think EndNote has.

Mendeley has a lot of potential that I’m actually surprised that they haven’t reached. They’re trying to be a cloud platform for your PDFs as much as reference manager. Like Dropbox, for whatever reason they only let you view – rather than highlight and annotate – your PDFs (I actually like Dropbox’s PDF viewer more than GoodReader).

In that regard, Mendeley is only one step from replacing my Goodreader + Dropbox combo: the ability to not just view, but also highlight and annotate with their mobile apps and have those sync back through their cloud servers. An integrated cloud storage + PDF reader + social platform + reference manager is actually something I would pay for, but there’s been almost no progress in that direction.

12 Comments

money said:

First of all, I would like to thank the author/ you for this really informative article. Just one question: What does or is the web-importer in Mendeley? (I´m using Mendeley, too, but either I’m not aware of the web-importer or I’m not using it.

Thank you so much for this incredibly detailed workflow! I just started a doctoral program and have been experimenting with different options. In undergrad and grad school, I literally printed every article. But iPads did not exist when I was in undergrad and were very new when I was finishing grad school.

I want to keep it as simple as possible. I’d much appreciate your feedback.

I use Readdle PDF Expert synced with Dropbox to annotate/highlight journal articles on my iPad. I have three folders – Unread, Read, and Notes Taken. When I finish highlighting, I move the PDF from Unread to Read. I then use the Readdle feature to export all annotations via email. I then copy and past the body of the email into a Word doc, which gets filed into a folder in Dropbox. The Notes Taken folder was supposed to house PDFs once I wrote up a one page synopsis. However, in practice, I found that I just have not had time to do this for all of the articles.

When I write, I use Word and have a split screen with all of the PDFs open. I refer back to my PDFs as I write. I have yet to actually refer back to the exported annotations. I do use an outline that changes as I write.

My thought was to add the read PDFs and notes to Mendeley once I finish with annotations. I do not like how Mendeley cannot successfully sync with Dropbox, so I haven’t done this yet. For my literature review, I created each reference in Word without any generators. I prefer to do this, though it is time consuming, as I have found that many generators have errors.

I’m not sure how I would use Evernote. Scrivener seems too out of my comfort zone.

Again, I would appreciate any feedback on what I might be missing or how I could better utilize these tools. Thank you!

I have been trying to manage my digital workflow for such a long time and have been missing some minor details.

Your statement, “First, I don’t like writing up in the margins of books or highlighting. I’ve always felt like I’m desecrating my book when I do that.” really connected with me on a very personal level. About 5 years ago I sold approx. $5,000 worth or design and architecture books. They were physical objects occupying space and I always felt terrible making any marks in them.

I could go on and on about all of the subtle nuances and wonderful details you provided here in this amazing post.

Nicely written and good to see that we use some of the same tools. Instapapper/Paper, yeah whats’ with hiding the highlights feature! You didn’t mention RSS feeds. With the demise of Google Reader, consider TinyTinyRSS. You’ll typically only have access to it from your main laptop, but the offline article and image sync is a great way to keep up with hundreds of blogs while riding the train. I use a 8 year old program ActionOutline for managing my scribbles without polluting my Scrivener notes.
Time to give Mendeley a try, because my integration of pdfs sucks. Oh, but one feature I added to my computer was to do a daily scrape of all pdfs on my laptop into one “active” folder on a usb stick. Great for grab & got!

I need a little help. 🙂 I use Scrivener for my writing (books, blog, etc., but I can’t figure out how to send a Scrivener document to Apple Pages using the compiler. I know I can use Word but I don’t want to pay a monthly subscription. I also realize I can “cut & paste” into Pages but that is hardly and elegant approach. What is the best way to get content from Scrivener into Apple Pages? Thanks!

I recently took my thesis out of Scrivener and it was a nightmare. I’m on the fence about recommending Scrivener. Literature and Latte don’t seem to be developing the app anymore (the iPad app is long, long overdue). Depending on the complexity of the document, your best option is either to export as a Word file or RTF. Apple Pages knows how to handle the Word file, so you don’t need any MS software acting as a middleman. Open it as a Pages file and let Pages covert/save in its own proprietary format. Expect problems. It won’t be a perfect translation.