Algorithmia Blog - Deploying AI at scale

April 16, 2018

How To Get All Different Types of Twitter Data

Twitter isn’t just my favorite way to waste time when my boss isn’t looking––it’s also a powerful data source for understanding what your customers are saying about you. Getting access to Twitter data for analysis is easy with the Algorithmia platform, and this post will walk you through how to do just that.

Twitter’s API and Authentication Process

In order to get up and running with programmatically accessing Twitter data, you’ll need to create what Twitter calls an “application” under your account. To start, head to https://apps.twitter.com/ and click the Create New App button.

Note: we recommend creating a new, separate Twitter account to utilize for this project just in case your API keys get compromised.

The setup process asks you for some basic information about your application. For Website, you can put https://google.com or any other valid URL.

Once you’ve successfully created your Twitter app, head over to the Keys and Access Tokens tab. You’ll want to grab 4 values:

Consumer Key (API Key)

Consumer Secret (API Secret)

Access Token

Access Token Secret

You may need to generate your Access Token and Access Token Secret from scratch using the button on the lower half of the screen. Keep these credentials safe! You won’t be able to use the Twitter API without them.

Working With the Algorithmia Client

Once you have your Twitter API credentials locked down, it’s easy to get up and running with Algorithmia in your language of choice. We support all major programming languages, but Python is my choice for data focused tasks (and anything else, frankly). Installing the algorithmia client in Python is as easy as

Each different GET request (we won’t be working with POST or PUT) has a unique rate limit (full list here). We’ll be using three different types of GET requests: tweets, followers, and friends. Here are the rate limits for each:

The Retrieve Tweets With Keyword algorithm returns complete Tweet objects with additional parameters like number of retweets and favorites. You might want to package the returned JSON into a Pandas dataframe or some other sort of more analyzable format.

To avoid that issue, Retrieve Twitter Followers takes two extra parameters: page and count. The page parameter specifies which page of followers you’d like to access, and count dictates a hard cap on the maximum number of followers to return. To work with rate limiting constraints, consider grabbing a few pages at a time and capping the maximum count at a few hundred.

You’ll get back a list of follower usernames that can easily be passed into other algorithms for further analysis.

Getting Friends

As many, many people in this Quora question mistakenly ignored, Twitter does have a concept of friends (hint: don’t answer Quora questions that you don’t know the answer to). Friends are people that you follow, while followers are people that you follow. You can get a user’s friends with Retrieve Twitter Friends. Since it’s rare for users to follow that many people, there is no count or page parameter.

Next Steps and Analysis

Getting any type of Twitter data––whether it’s the tweets themselves, followers, or friends––is almost always just the beginning. Twitter data is a wonderful medium for different types of analysis, and a great example is the sentiment analysis demo we created here. In addition to sentiment analysis, there are a number of other interesting types of algorithms that you can apply to your Twitter data: