Twitter Followers and List Membership Tracking

I have created my own historical reporting of my Twitter account followers and list memberships. I have hosted the reporting on this open webpage.

Most people know what followers are but the list memberships aren’t as well known so here are definitions of them.

Twitter Followers – Twitter followers are other Twitter accounts who follow my account. A Twitter Follower may follow my account for a few days and then unfollow me at any time afterwards.

Twitter List Memberships – Twitter accounts can also create their own Lists and then add other Twitter accounts to the lists. For example a Twitter account may create a List called “Tweet about Big Data” to track Twitter accounts they believe “Tweet about Big Data”. Twitter accounts may add my account to their List and remove it at any time afterwards.

The Twitter data retrieval, writing to database, and data querying are all done on a web server.

In order to record changes in counts of these you need to have daily historical data. The Twitter API doesn’t provide historical data so you need create it yourself by retrieving and saving daily data somewhere.

Three Python scripts using Twitter API, Python Tweepy and AWS SDK are scheduled to run daily using cron jobs.

Two scripts retrieve followers and list memberships and insert the data into a PostgreSQL database. This daily retrieval builds the daily history.

Another script queries the database table to create reporting datasets of new, active and dropped followers and list memberships that are exported as csv files to a AWS S3 folder which also has files for a AWS S3 hosted static website.

The AWS S3 hosted static website uses Chart.js and D3.js Javascript charting libraries to read and visualize the data. This post does not describe how to read the csv files but I have written another post that describes this AWS S3 csv file as D3 report data source

A screenshot of one of the visualizations showing active followers by day, with counts of new follows and unfollows by day is shown below.

A second screenshot of another visualization showing active list memberships by day, with counts of new listings and un-listing by day is shown below.

The Python code to retrieve the Twitter data, transform it and create csv files and upload them to AWS is below.

I have learned most of what I need to use the Twitter API through searching other people’s blogs and StackOverflow. I have had to modify almost all of the code on my own though once I found other people’s code.

Conceptually it is pretty straightforward but the details as always make it a bit more challenging.

First you need Twitter Developer account which can be same user as any regular Twitter account. Then you need to create API keys that will uniquely identify you to the Twitter API. Depending on what you are doing you may also be required to setup authentication.

I recommend using Tweepy which is Python module that is made to interact with Twitter API. You then have to just learn how to use Tweepy rather than interact with Twitter API directly which makes it easier.