How I Archive My Tweets

The same chaos that bereft the demise of the Library of Alexandria would also occur should my tweets be lost to the world!

Last week, I had a brief Twitter conversation with my friend Mike about Timehop. Timehop is an online service will send you a daily email digest of your social activity from one year ago. It works with Twitter, Facebook, and Instagram, to name a few services.

The idea is great, but when it comes to Twitter it falls short. Due to Twitter limitations, Timehop can only access your 3200 most-recent Tweets. If you tweet more than 3200 times a year — like moi — your daily Timehop email digest is going to be absent any tweets. Timehop’s workaround is sound: when you signup for their service, they begin caching your tweets. So in a year, I should start seeing my old activity. It’s disappointing having to wait for full functionality, but it’s not Timehop’s fault.

This got me to thinking how else would people view their old Twitter activity. Luckily for me, I’ve been archiving my tweets for several years. I first signed up for Twitter on September 27, 2007, and earlier this week I topped 34k public tweets. Here’s a rundown on the different methods I’ve used to record my history.

Why would anyone want to archive their tweets?

In my case, the short answer is because I have an inflated senses of self-worth. 🙂

The long answer is because of an newspaper article I read during college. It detailed the dilemma expected as the internet was becoming more adopted in all aspects of our life. Up until this point of time, historians were always able to refer to existing records of human activity: letters, ledgers, newspapers, books, etc. But as we progressed towards becoming a fully-digital civilization, such physical records were becoming more scarce. The shift to electronic records would be problematic to historians for several reasons, the primary of which is the ease at which something digital could be lost on a server.

The article recommended two deliberate steps be taken to archive such material:

Put into place a regular and publicized backup strategy; and

Backup the data in such a way that the data could be extracted without running into a limitation of the program used to create said backup.

In other words, backup often and save such data in the simplest form.

At the time, I was a history minor in college, and I have several professors who lamented the upcoming knowledge apocalypse that digital records might accelerate. Citing the examples of how mankind was robbed of knowledge as the result of catastrophe (example: the Library at Alexandria), they encouraged us to adopt personal archive strategies. That way, should we ever want to reflect back on our life, the data would be there. And if we ever did anything significant enough in our lives that others would want to do the same, the data would be there.

What I took out of it: keep a copy of everything, no matter how mundane. Although I don’t expect to be President of the United States, I have found having history at my fingertips to be invaluable as a blogger and personal history nut. It’s allowed me to go back in time and flesh out the stories I tell. And this habit of backing everything up has extended to Twitter as well.

Twitter Archive Method #1: Twitter Tools Plugin for WordPress

My original Twitter archive solution was the excellent WordPress plugin Twitter Tools by Alex King/Crowd Favorite. I originally installed it to cross-post this blog to Twitter, but discovered the nice side-effect that my tweets were also being cached within custom tables.

While it does require you to be running WordPress (not a problem in my line of work — hire me!), there are several advantages to this archive method:

Tweets are stored in a relational database, specifically within a custom table under my MySQL DB for my WordPress install. From here, other applications on my website could consume them.

Each row in the table had a separate field for timestamp and unique Tweet ID.

Because they are stored in a table, it’s easy to search, sort, export, and report against the data.

They are included in my regular backup solutions, including my daily BackupBuddy backups of my blog database.

The disadvantages are minor, IMO:

The tweets are stored in custom tables vs. WordPress core tables. This is heresy to some WordPress developers (but not me).

They take up room in my database. Back in the day, this was a major worry. Although I now have unlimited space, having this volume of tweets around sometimes bogs down activities such as database exports/imports.

I don’t actually do anything programmatically with the data, so there’s not much use to having the data in the same database as WordPress.

To do something with the data requires me to write reports or code.

Overall, Twitter Tools was a great option. I started using Twitter Tools to archive my tweets in September 2008. During the two years I employed this solution, 17985 total tweets have been saved.

Method #2: FeedMyInbox + Evernote

Update 2012/01/13: FeedMyInbox recently announced they would shut down. As alternatives, they recommend Blogtrottr, Zapier or IFTTT.

I have the never-ending itch to constantly iterate my websites every few months. And running a WordPress plugin I rarely use like Twitter Tools was bogging me down. I wanted to disable the plugin, but I also wanted to keep archiving my tweets. Ergo, an alternative solution was needed.

Some of my friends at Livid Lobster recommended FeedMyInbox. It’s an online service that consumes any RSS feed and sends you the contents as emails. They have free and paid flavors, but the free option works great for me — for up to 5 feeds, I get one daily email digest of the feed’s contents.

So to archive Twitter, I configured FeedMyInbox to consume my Twitter RSS feed and send it as a daily HTML-formatted email. There is the option to format emails as text-only, but I like the HTML format: it’s prettier on the eyes, and I also get to see everyone’s Twitter avatars.

Originally, I sent the emails to GMail, where I collected nearly everything else since 2004. Lately, I’ve been forwarding them to Evernote, which seems like a more-natural location for archive data like this.

I see a few disadvantages of using Evernote vs. Twitter Tools, but not enough to sway me from this new method:

My tweets are no longer stored in a relational database. This makes it harder to determine exactly how many tweets are in this new archive.

I can no longer do anything programatically with my tweets. Because they are no longer stored in a table, I can’t leverage individual fields like timestamp. Since I never actually did anything with my Twitter Tools data, this is a minor disadvantage, at least for me.

I am at the mercy of third-party services. Twitter Tools was a complete solution that ran on my personal website, so I have increased the number of gears in motion by going with FeedMyInbox and Evernote.

I’ve been using FeedMyInbox + Evernote since April 2012. It’s hard to count just how many tweets have been archived, but since I average nearly 11 tweets/day I’d say Evernote is holding 4100+ tweets.

“Wait…your numbers don’t add up, McGarity!”

You’re too quick for me, ninja reader. As I mentioned above, I have over 34k public tweets. But if you add up the numbers above, there’s a 12k+ discrepancy in the count. Where did those tweets go?

Remember how I said I like to iterate my websites? I suspect that over time, I might have changed things up on my blog and forgotten to turn Twitter Tools back on. The last tweet stored by Twitter Tools was in August 2010, while the first one I stored in Evernote occurred during April 2011.

August 2010 was around the time I changed my personal brand from Spamboy.com to McGarity.me. This included a major blog restructuring and a brand-new, from-scratch install of WordPress. I suspect that’s where the gap occurred. Nothing interesting could have possibly occurred during that time, right? Right…?

So those tweets are lost to me forever, right?

Nope. Thank God for taxpayer dollars…

Method #3: Library of Congress

And that leads to the last method, which I wish I could take credit for.

Back in April 2010, the Library of Congressannounced an agreement to archive all public tweets since 2006 (blog post, FAQ). Right now, the tweets are not publicly consumable, but maybe that will change one day.

Now future generations of American researchers will be privy to my thoughts on Usher trying to have sex with me through the television and A-Rod’s propensity to strike out while looking during a deciding baseball game. Someday when I am President of the United States, my tweets will be valuable psychological insight into the mind of your future leader. You’re welcome.

Conclusion

Thanks for listening to me ramble about my attempts at historical posterity. If you are archiving Twitter, what methods do you use? Let me know if you have something better — I’m always on the lookout to improve my solution!

Author: Matthew

Husband to Jenn, father to Zachary and Penelope, blogger, artist, WordPress consultant, OpenCamp organizer, and running enthusiast. Brother, can you spare an extra hour in the day?
View all posts by Matthew