Python, Data Science, Text Analytics

Mining Twitter Data with Python (Part 1: Collecting data)

Twitter is a popular social network where users can share short SMS-like messages called tweets. Users share thoughts, links and pictures on Twitter, journalists comment on live events, companies promote products and engage with customers. The list of different ways to use Twitter could be really long, and with 500 millions of tweets per day, there’s a lot of data to analyse and to play with.

This is the first in a series of articles dedicated to mining data on Twitter using Python. In this first part, we’ll see different options to collect data from Twitter. Once we have built a data set, in the next episodes we’ll discuss some interesting data applications.

More updates: fixed version number of Tweepy to avoid problem with Python 3; fixed discussion on _json to get the JSON representation of a tweet; added example of process_or_store().

Register Your App

In order to have access to Twitter data programmatically, we need to create an app that interacts with the Twitter API.

The first step is the registration of your app. In particular, you need to point your browser to http://apps.twitter.com, log-in to Twitter (if you’re not already logged in) and register a new application. You can now choose a name and a description for your app (for example “Mining Demo” or similar). You will receive a consumer key and a consumer secret: these are application settings that should always be kept private. From the configuration page of your app, you can also require an access token and an access token secret. Similarly to the consumer keys, these strings must also be kept private: they provide the application access to Twitter on behalf of your account. The default permissions are read-only, which is all we need in our case, but if you decide to change your permission to provide writing features in your app, you must negotiate a new access token.

Important Note: there are rate limits in the use of the Twitter API, as well as limitations in case you want to provide a downloadable data-set, see:

Accessing the Data

Twitter provides REST APIs you can use to interact with their service. There is also a bunch of Python-based clients out there that we can use without re-inventing the wheel. In particular, Tweepy in one of the most interesting and straightforward to use, so let’s install it:

pip install tweepy==3.3.0

Update: the release 3.4.0 of Tweepy has introduced a problem with Python 3, currently fixed on github but not yet available with pip, for this reason we’re using version 3.3.0 until a new release is available.

More Updates: the release 3.5.0 of Tweepy, already available via pip, seems to solve the problem with Python 3 mentioned above.

In order to authorise our app to access Twitter on our behalf, we need to use the OAuth interface:

The api variable is now our entry point for most of the operations we can perform with Twitter.

For example, we can read our own timeline (i.e. our Twitter homepage) with:

for status in tweepy.Cursor(api.home_timeline).items(10):
# Process a single status
print(status.text)

Tweepy provides the convenient Cursor interface to iterate through different types of objects. In the example above we’re using 10 to limit the number of tweets we’re reading, but we can of course access more. The status variable is an instance of the Status() class, a nice wrapper to access the data. The JSON response from the Twitter API is available in the attribute _json (with a leading underscore), which is not the raw JSON string, but a dictionary.

So the code above can be re-written to process/store the JSON:

for status in tweepy.Cursor(api.home_timeline).items(10):
# Process a single status
process_or_store(status._json)

What if we want to have a list of all our followers? There you go:

for friend in tweepy.Cursor(api.friends).items():
process_or_store(friend._json)

And how about a list of all our tweets? Simple:

for tweet in tweepy.Cursor(api.user_timeline).items():
process_or_store(tweet._json)

In this way we can easily collect tweets (and more) and store them in the original JSON format, fairly easy to convert into different data models depending on our storage (many NoSQL technologies provide some bulk import feature).

The function process_or_store() is a place-holder for your custom implementation. In the simplest form, you could just print out the JSON, one tweet per line:

def process_or_store(tweet):
print(json.dumps(tweet))

Streaming

In case we want to “keep the connection open”, and gather all the upcoming tweets about a particular event, the streaming API is what we need. We need to extend the StreamListener() to customise the way we process the incoming data. A working example that gathers all the new tweets with the #python hashtag:

Depending on the search term, we can gather tons of tweets within a few minutes. This is especially true for live events with a world-wide coverage (World Cups, Super Bowls, Academy Awards, you name it), so keep an eye on the JSON file to understand how fast it grows and consider how many tweets you might need for your tests. The above script will save each tweet on a new line, so you can use the command wc -l python.json from a Unix shell to know how many tweets you’ve gathered.

You can see a minimal working example of the Twitter Stream API in the following Gist:

Hi, I haven’t worked much on MySQL recently, but recent versions support a JSON data type so you just create a column of type JSON and dump the entire tweet in it. Another option is that you read the JSON and normalise the structure with only the fields that you need.
Cheers,
Marco

File “C:/Users/Hamza-HP/Desktop/untitled/Tweepy.py”, line 24, in
process_or_store(status._json)
File “C:/Users/Hamza-HP/Desktop/untitled/Tweepy.py”, line 18, in process_or_store
print(_json.dumps(tweet))
NameError: name ‘_json’ is not defined

Hey Marco, I’m relatively new to coding and I was trying out your script to see 10 of my twitter feeds but its giving me a UnicodeEncodeError. What else do I need to add to the script that you provided to make this work?

Hi Nav,
it depends on which line is trowing the error and what the exact error is. The examples are tested in Python 3.4+, so if you’re using Python 2 please keep in mind that the string data type is different (unicode in Python 3, non-unicode in Python 2). If that’s the case, please consider upgrading to Python 3. Also have a look at this one for more details: https://wiki.python.org/moin/UnicodeEncodeError
Cheers
Marco

Hi,
I tried to the Streaming example given above. But unfortunately it gives a syntax error indicating the ‘&’ symbol before ‘quote’, in “print("Error on_data: %s" % str(e))” line.
How can I get this error fixed ?

hi, i got same error.
i just put commend on exception like this :
#print("Error on_data: %s" % str(e))
and put this code on the next line or replace it both working wonderfull :
print (‘erorr’, str(e))

Hi Justin, in terms of hosting I’ve only used aws for this, not sure if it fits your requirements. I’m sure there are other options but I don’t have a specific recommendation at the moment I’m afraid.
Cheers
Marco

Hello, these days I have saved several tweets with your script.
Today I was watching them and I seem all written by professionals and not by individual.
They are tweets that talk about the weather, news, promotion activities etc …
it’s normal? or am I doing something wrong?
For me it would be more useful to analyze the common people tweet, in order to have a vision of what people really think.
Here I have published some are in Italian, but your last name seems Italian: Dhttps://docs.google.com/document/d/1gYZVnFSpnAYqAEQBWeLJHC6QdpA2koY4i3_mF85Rf7E/edit?usp=sharing

Hi
if you use the code as it is, it creates a “python.json” file in the same folder where you’re running the script. If you check out the examples from my book you see how the filename is created dynamically.

Hi
yes you can come up with the file name dynamically, using a different name every time, for example using the tweet id to ensure the names are unique. Maybe adding the timestamp as well so you can sort them “easily” (you’ll end up with too many files)

Hi
Actually I have to do sentiment analysis and for that purpose I need to collect some Twitter data so can you please tell how to we get consumer I’d and consumer secret mentioned in your tutorials first part.
How the application will be set up ?

HI Marco, I really enjoy reading your blogs and attending meetups you speak at. I am trying to collect a year’s worth twitter data from the current date based on selected keywords . I’ve used the twitter search API but it only seems to give me 12 days worth results (around 3000 tweets) for a keyword. Would I be able to get results from a longer period using tweepy (ideally I would like to specify the start and end date for my search) ? or would I need to subscribe to Twitter Firehouse ?

Dear Marco,
i hope you can help me out.
I sucessfully set up tweepy and now i wanna get the latest trends for a specific place (set by woeid) as a list and not more than 10 trends. I wish i could do that myself but i’m a complete douche.

Hi!
i got this error, can you help me out
Traceback (most recent call last):
File “C:\Users\YousufM\Desktop\abc.py”, line 17, in
process_or_store(status._json)
NameError: name ‘process_or_store’ is not defined

my code was this
for status in tweepy.Cursor(api.home_timeline).items(10):
# Process a single status
process_or_store(status._json)

Hi
you need to defined the function as described at the end of the paragraph (as an example, there’s a simple implementation that prints out the json), you’ll have to implement it depending on your needs (print it, store it on DB, etc.)
Cheers,
Marco

Hi Marco,
I am a student assisting a professor who wants to analyze twitter behavior of Dutch politicians (most of which have a public twitter profile) during the elections. To do this, I want to construct a database with the following information:

Per politician (about 300 in total), how many original tweets / retweets / replies they sent per day, for all days in the period 01/01/2017 – 03/31/2017. I have no experience with programming whatsoever (I do like math and working with Excel / statistic software).

Currently, I am doing this all manually (counting by scrolling through their timelines and putting all information in Excel). As you can imagine, this takes ages, and I suspect that it might be possible to do it so much quicker. Besides, learning (the basics of) programming has been on my wishlist for a long time.

However, I would like to know if it is likely that there is a ‘programming solution’ for this problem at all or that due to privacy/request limitations it will not be worth to devote all these hours to learning because it won’t work anyway. With all your experience: Do you think this is possible? Or might I just as well continue counting?

Hi Dani
there are several limitations imposed by the Twitter API, but there are definitely some workarounds. If you’re tracking a specific account, you can retrieve up to 3,200 of its most recent tweets using this method (https://dev.twitter.com/rest/reference/get/statuses/user_timeline). An example of implementation using Python is in my book (https://github.com/bonzanini/Book-SocialMediaMiningPython/blob/master/Chap02-03/twitter_get_user_timeline.py). On top of the limitation given by the total number of tweets that you can retrieve with this approach, there is also a rate limit (described in the Twitter API link above), so retrieving a lot of data will likely require some time just because you need to pause the requests (they don’t let you hammer the API). If a user tweets a lot, you’re unlikely to be able to capture a specific time window in the past because you can only retrieve the most recent 3,200 tweets.

On the other side, for upcoming tweets, you can keep the stream open and track the activity of specific accounts, using the Streaming API as described in this tutorial. The only difference is that you’d use the option “follow” to spell out the user names you want to include in the stream, rather than “track” that is used for keywords. This second approach will require an always-on server and possibly additional configuration to monitor the stream.

As a starting point you can use the user_timeline script from my book (link above) and try it with a few usernames.

Hello,
Thank you for the great job you have done. I tried the code and it works with a single term but when I tried using multiple terms for track term I got errors. How do I modify this code “python twitter_stream_download.py -q apple -d data” to account for the change of terms. Thank you

Hi Ola
you can use double quotes around your query, and all the query terms will be passed to the API, for example:
python twitter_stream_download.py -q “apple football” -d data
this will query the API for “apple AND football”
otherwise with:
python twitter_stream_download.py -q “apple, football” -d data
using the comma between keyword is equivalent to an OR query (in this case, apple OR football)
It’s always worth checking out the official docs in case the API changes in the future https://developer.twitter.com/en.html

This is Truly a masterpiece .Very much explained and clear in meaning . I followed this and working fine .
Is there any way we can get rid of “tweepy.error.TweepError: Twitter error response: status code = 429” ? I searched this in google but not much help I got . I put this in tweepy github page but the response redirecting me to twitter Rate Limit page . I have seen many online application available which are making much more request than what I am trying to do .
Any help in this regard is highly appreciated .

Dear Marco, thank you for this tutorial and your book, which is really great and addictive :)
I have collected quite a lot of tweets so far and tried different analyses.
Now would like to convert my json-Tweets into a csv-format but am struggling for some reasons. Could you give an example on how to do that based on your example here?
I am interested in the conversion of existing json-files with data collected from twitter, as well as in how to store twitter streaming data in csv as they directly come out of the stream.
I would appreciate your help very much!
Thank you

Hi Mister Bonzanini,
Thanx for your book which has given me a new approach to the social media. I’m especially interested in Twitter mining but I can’t run the scripts. I constantly receive the code error 400 while trying to run the the twitter_get_home_timeline.py script.

How do I proceed ;
– I go to the directory I’m want to work in;
– I call my virtual environnement by typing .\data\me\Scripts\activate
– I set each of the four environment variables :
(set TWITTER_CONSUMER_KEY=”Variable1″
set TWITTER_CONSUMER_SECRET=”Variable2″
set TWITTER_ACCESS_TOKEN=” Variable3″
set TWITTER_ACCESS_SECRET=” Variable4″
– When I type py twitter_get_home_timeline.py, which is found in my directory, I have the error code 400. The script generates a file called home_timeline.jsonl but it is empty.

What should I do to run ths script properly given that I have been reading tons of documentation for the last ten days to solve the problem but I haven’t met the appropriate answer. For your information, I am using Python 3.6.4 and Tweepy.3.6.0.

Your help is most welcome beacuse I am really keen about progressing with the practical aspects of Twitter data mining for my job. I am a journalist working for a French newspaper.