Title: Academic Workflows
Post by sampsasamila on Feb 17th, 2008, 10:17am
I'm very interested in hearing from other academics how you use Tinderbox together with other software, such as bibliography managers, writing software, and possibly some databases.1) What do you do in each?2) How do you take things from one program to another?

I'm currently trying to build a workflow to organize my note taking and also general activity, in the style of "Getting Things Done." My goal is to practically implement in contemporary software what C. Wright Mills did in his paper files:

"... what really happened is ... that the idea and the plan came out of my files, for all projects with me begin and end with them, and books are simply organised releases from the continuous work that goes into them"

Back when I was a physical chemist, we used to speculate on a lab computer so good that it would have a green button marked "Run Experiment", a yellow button marked "Collate Data", and a blue button marked "Write paper".

I think, though, it might be really useful to discuss patterns and components that have proved useful in a variety of projects. For example, I have find the need for an Event Container, a pattern characterized by

A user attribute for dates, representing the initial date for an event

Note names that provide convenient labels for the event

All events are created inside a container, which keeps them sorted in chronological order.

A prototype Event is usually convenient; the event container's OnAdd action typically assigns the prototype for convenience.

Convenient sentinel notes mark the beginning and end of the period of interest; these also help identify notes with missing or mistyped dates

The note body describes the event and links to references or additional resources; the URL attribute might also be used for this purpose

Outside agents collect and arrange subsets of events, both functional (events in which Roosevelt was involved) and operational (events for which we need better references). Agents may also assign badges and other distinctive visual features to highlight events that require attention or research (e.g. events that have not yet been mentioned in chapter 3).

Title: Re: Academic Workflows
Post by sampsasamila on Feb 20th, 2008, 4:40am
The last part did make it seem very broad. So going back to the original question of integrating Tinderbox with other software:

Does any one have good ideas on how to integrate BibDesk, Skim, and possibly Devonthink with Tinderbox?

BibDesk and Skim together are great for managing a bibliographic database and taking notes on specific papers. Devonthink on the other hand seems great for storing collections of news articles and smaller clippings from the web and elsewhere. I've been thinking of using Tinderbox for general note taking, in particular for notes that touch on many papers that would be hard to manage in BibDesk & Skim.

So the practical issues are:1) Is there a way to get BibDesk annotations and Skim notes into Tinderbox?2) Is there a way to update the notes in Tinderbox if the notes in BibDesk or Skim are changed?3) What's the best way to link Tinderbox notes to papers in BibDesk and clippings in Devonthink?

Thanks in advance.

Title: Re: Academic Workflows
Post by Jim Delaney on Feb 21st, 2008, 10:57pm
I've been struggling with a very similar workflow. Tinderbox, Skim and Bibdesk got me through my comprehensive exams last October. Reading and note-taking was done in Skim, which aggregates notes well via Bibdesk. Most of the notes were then copied manually to Tinderbox, with attributes for author, title, etc. and some extended attributes for keywords. It would be nice (and possible, I think, if I had the time) to automate some of this, as it requires some duplication of effort, though some practice made the manual copying pretty quick and painless.

You can link notes in Tinderbox to Papers in BibDesk by linking to the actual file. I doubt that you could presently do the same in Devonthink, though they are planning to move to a file-based storage system that will allow the same thing. At the same time, given how fast search is getting, I am not certain that the links are necessary. It just takes seconds to hunt down a file in BibDesk.

Any other ideas as to how to better integrate this workflow would be appreciated.

Jim

sampsasamila wrote:

The last part did make it seem very broad. So going back to the original question of integrating Tinderbox with other software:

Does any one have good ideas on how to integrate BibDesk, Skim, and possibly Devonthink with Tinderbox?

BibDesk and Skim together are great for managing a bibliographic database and taking notes on specific papers. Devonthink on the other hand seems great for storing collections of news articles and smaller clippings from the web and elsewhere. I've been thinking of using Tinderbox for general note taking, in particular for notes that touch on many papers that would be hard to manage in BibDesk & Skim.

So the practical issues are:1) Is there a way to get BibDesk annotations and Skim notes into Tinderbox?2) Is there a way to update the notes in Tinderbox if the notes in BibDesk or Skim are changed?3) What's the best way to link Tinderbox notes to papers in BibDesk and clippings in Devonthink?

The last part did make it seem very broad. So going back to the original question of integrating Tinderbox with other software:

Does any one have good ideas on how to integrate BibDesk, Skim, and possibly Devonthink with Tinderbox? ........................

Before considering my highly idiosyncratic process for adaptation, please be aware I rely quite a bit on several GUI and UNIX-layer pieces of software which support scripting and regular expressions (BBEdit and textutil, for example). I also occasionally make (intermediate level) use of PERL. I am not a programmer.

I have a full workflow that involves EndNote, Tinderbox, Skim, the occasional PERL script, HTML export, and a few other pieces I cannot elaborate here and now. Much of the glue that makes this all work is BBEdit and a few text factories that I have assigned keyboard shortcuts (using KeyboardMaestro).

It all came out of a post on the Tinderbox Wiki available at ImportFromEndNote. (As a tip, rather than introducing underscores into the attribute names and correcting them in BBEdit, escape them with backtick (`) characters. This way, you won't need to have BBedit replace anything.)

Getting complete bibliographic references into Tinderbox with hooks into any file you might like, including PDFs:

0. Export your items from EndNote.1. Open your main document (TARGET) in Tinderbox.2. Create a very simple secondary document (SOURCE) in Tinderbox. Close it.3. Open the EndNote export in BBEdit.4. Open SOURCE in BBEdit.5. Insert the EndNote export into SOURCE. Save SOURCE. Close SOURCE.6. Open SOURCE in Tinderbox.7. Copy-paste/drag the notes from SOURCE to TARGET.

This takes a bit of setup, namely creating user attributes in a Tinderbox document to receive the EndNote attributes. (Keep in mind that in this case, more is more.) Though I had a working version up in a few hours, the system took me a a couple of months to nail down and three projects to consider relatively complete. A lot of the kinks got worked out as I became more experienced with Tinderbox. For example, it took me a number of months to figure how to have EndNote export links to PDFs that worked within Tinderbox. (Automation is everything.)

I've been meaning to make a video-based tutorial for all this but haven't had the time or the (tenure activity) incentive. The ImportFromEndNote link, however, should be enough to get you started.

Once you have bibliographic-standard items in Tinderbox, you'll be able to make use of your Skim notes. In my case, I export anchored Skim notes (with 3-digit page numbers as titles) to text, open the resulting text file in BBEdit, and transform the text file using text factories. I can then bring the text file into TARGET and attach the notes to the corresponding item.

As I write, my citations have page numbers, sources, and links back to the original PDF material. When I'm ready to produce output (suitable for a journal) I use HTML export and transform the resulting material using John Gruber's MarkDown and SmartyPants and Mac OS X's built-in textutil (two steps with a simple PERL script).

The big part is building the basic Tinderbox/EndNote structures.

Short term, I'd love to share tips with others (especially humanists with some programming knowledge). Long-term, I'd love to see someone or some company write a piece of software that combines all of these functions in a stand-alone application or for Tinderbox to develop an API/plug-in architecture/scriptability that would allow many of these functions and interconnections to be formalized/automated (as opposed to requiring me to intervene in several parts of the process with BBEdit and PERL).

Title: Re: Academic Workflows
Post by David Nelson on Mar 6th, 2008, 3:52pm
I was reading this thread just to get ideas for my own academic work when I saw you all using Skim. I was not familiar with that program and went to research it. Wow! I have been looking for something like this to deal with all the PDF files I am accumulating for my next book. Thanks for the tip.

Title: Re: Academic Workflows
Post by Peter Pawelek on May 22nd, 2008, 7:31pm
Yes, Skim is so good that I prefer it to Adobe Acrobat for marking up PDF files. Much more lightweight and just as effective.

Title: Re: Academic Workflows
Post by Peter Pawelek on May 23rd, 2008, 9:31am
It should be noted that Skim is also a good partner for Tinderbox. If you have a Skim-annotated PDF you can export the notes into raw text format. Skim will export them as a single file, but each note is separated by a line saying something like "* Anchored Note, page 2". It's trivial to then chop this up into individual text files corresponding to each note by using a tool of your choice (Perl, Python, shell script, etc.). These files can then be imported (or dragged) into Tinderbox as notes.

Title: Re: Academic Workflows
Post by Mark Anderson on May 23rd, 2008, 10:11am
You could probably do this with Explode inside TB. I don't have Skim to have but if you want to email** me a specimen Skim note export text file I could have tinker.

Yes, you're absolutely right. I just tried Explode on an imported Skim annotations file and it works great. You need to use '*' as a delimiter. Actually, can we use Booleans in the delimiter field? For example ('* Highlight' OR '* Anchored') would be useful if the notes themselves contain asterisks.

I can't seem to find your email but you should be able to get mine from my contact info. If you email me I can send you a representative Skim annotation file if you want to play around with this.

Title: Re: Academic Workflows
Post by Mark Anderson on May 24th, 2008, 6:15am
Peter, thanks for the specimen output. You want to use "* " (asterisk+space) as your delimiter. Look at this forum thread for some ideas as to how you can further refine your content post-explode. For instance you might remove the page name from the note title whilst sticking the note into a container for the appropriate page. The title part of the note can also be deleted from the note's body text. You might use different colouring for 'Highlight' notes vs. 'Anchored' notes, etc.

For others following the thread, a sanitised version of Peter's source data is like this:

When you drag in the source note (or paste in it's text) ignore the fact that TB may indent some lines starting with * as it thinks they are markers for quick lists (for HTML output). Don't worry, it won't effect the way the text gets exploded.

Yes that's perfect, thanks. I'm especially interested in stripping out titles occurring on the same line as the '* '. I'll look into it on the other forum thread that you mentioned.

Cheers!

Peter

Title: Re: Academic Workflows
Post by Jim Delaney on Jul 22nd, 2008, 1:21pm
I spent a couple of hours yesterday evening fiddling a little with my research notes workflow, and thought of another simple step to ease integration between Tinderbox, Skim and BibDesk. BibDesk will allow drag and drop export of citations in tab separated values through a template (a simple version can be found on the BibDesk wiki). With just a little setup, you can drag any of your references from BibDesk to an open Tinderbox window, which will create a note with all of the attributes mapped out for you thanks to the spreadsheet import feature. This works brilliantly with BibDesk, and can probably be set up with any reference manager that allows for user-definable export templates.

Jim D.

Title: Re: Academic Workflows
Post by sampsasamila on Jul 22nd, 2008, 3:26pm
Beautiful. I was thinking of making an OPML template for BibDesk, but no need now.

Title: Re: Academic Workflows
Post by DerikBadman on Jul 23rd, 2008, 1:09pm
This kind of works for me. Some of the new attributes are coming up as boolean instead of string (Journal is either true or false!). This should be useful with a little tweaking, as I just started using BibDesk (love the ability to organize pdfs along with citations.

Title: Re: Academic Workflows
Post by Mark Bernstein on Jul 23rd, 2008, 1:15pm
If Tinderbox needs to create new attributes for dragged data, it has to guess at the type by looking at the data.

But there's no need to guess; if you know you're going to need a string attribute called Journal, make the attribute yourself and make it a string. Tinderbox will use it when importing.