Copyright notice: I hereby release all of the writing content that is tagged by R, under the cc-by-sa copyrights (date: Jan 26 2012), as long as the copied content comes with proper attribution which also includes a link to the source of the article .

I was trying – for the last few weeks – to get the 2001 Indian census data. Alas the census website is under construction. But fortunately the Internet rewind button works! Thankfully the literacy data was online there. The raw data is available here.

I cleaned up the data so that it is easy to work with R. I removed the commas in the numbers. Also, under the urban status column I removed the dots and capitalized the status codes. One of the urban status became ‘NA’ and since R treats ‘NA’ as a missing data I changed it to NA1.

The cleaned up data is available here. Please download and rename it as india-census-2001.csv

Looks as if the data in the Kachchh region was not collected. Wonder why those two Rajkot districts also suffered the unfortunate fate. Maybe they are close to Kachchh region. Anyway let us look at the data which has non-zero population.

Let us plot the literacy rate of the city/town (x-axis) against the State (y-axis)

Looking at the plot, no surprise that Kerala has very high literacy rate in all the towns and the spread is also low. Tamil Nadu has a bigger spread in the literacy rates. The Northeastern states are doing very well in the educational aspect if we evaluate them by their literacy rates.

Let us check which city/town has the highest and the lowest literacy in India

Well, this is a surprise. The city with the highest literacy is in Jammu & Kashmir (Gulmarg) and the lowest is in Maharashtra (Tarapur). What is shocking is that the literacy rate in Tarapur is less than 1%. I hope that there was mistake in data collection, otherwise it is a damning indictment of a huge administrative failure in that district. This is unacceptable.

In the next few posts, I will concentrate on Tamil Nadu and Coimbatore. It should be pretty easy to modify the code in the coming posts to look at the states and districts of your interest.

A few days ago I was reading an article by Jean Drèze and his colleagues on how the first two years of National Rural Employment Guarantee Act (NREGA) has progressed (There was another article by Drèze on NREGA in 2007). The NREGA is empowering the rural people in a radical way:

[ …] NREGA programmes visualise a decisive break with the past. Ever since independence, rural development has largely been the monopoly of local contractors, who have emerged as major agents of exploitation of the rural poor, especially women. Almost every aspect of these programmes, including the schedule of rates that is used to measure and value work done, has been tailor-made for local contractors. These people invariably tend to be local power brokers. They implement programmes in a top-down manner, run roughshod over basic human rights, pay workers a pittance and use labour-displacing machinery.

NREGA is poised to change all that. It places a ban on contractors and their machines. It mandates payment of statutory minimum wages and provides various legal entitlements to workers. It visualises the involvement of local people in every decision — whether it be the selection of works and work-sites, the implementation of projects or their social audit.

After going through the articles I thought about reproducing the color (gray) coded maps. Of course the best tool to do this would be R. It took a few days to figure out how to do this. The rest of the post (hopefully clearly) will be on how to produce an Indian map gray coded with literacy rate of the state.

First we need an Indian map. Fortunately there is a free Indian map although it is little old. You can download the Indian map from here.

Extract the zip file in an directory and let us call this directory “dir”

I got the literacy data [PDF] from the Indian Budget website and put it as a csv file here. Please download it and rename it as literacy.csv and place it in the same directory (“dir”) as the maps.

Note: I removed a few states (Jharkhand, Chhattisgarh, Uttaranchal) from the original data since the Indian map we have is little dated and does not contain these states. Maybe later on we can figure out how to draw the boundaries for those states.

A week ago, Amma was telling the weather was getting hot in Coimbatore. I was telling her it is going to get worse in the next two months. She shot back saying that March is the hottest month while April and May are less hotter in Coimbatore. Growing up in India you are thought that your mother knows the best and she is right (almost) always. Well I could not resist the thought of putting the thesis to test and the internet comes to my help. So here goes the perl and R code to get the temperature data and explore it. The metric of choice would (arbitrarily) the average temperature. In R plots in this post the average is plotted by a big black dot. The month with the highest average temperature will be adjudged the hottest month. There are a lot of metrics to use but this is the simplest and the most intuitive.

(1) perl code to scrap temperature data from web. The wunderground website has data in csv format. I did not do checks like limiting days in June to 30. We will fix that in clean up script which will build a single csv file with data from years 2005 to 2008. You need the CPAN package LWP.You should be able to google and figure out how to install LWP package.

Save this file as [dir]/src/get_temperature_files.pl
To run% cd [dir]/src% perl get_temperature_files.pl
You will have the data in the directory [dir]/data
The data is for four years from 2005 to 2008.

Looks as if Amma thesis is probably rejected! Four years worth data shows that highest average temperature is in April. Let us see a split by years and see if there is a year in which March’s average temperature was the highest. Looks like we need to do little clean up of the data. There is a zero in October. Definitely not possible in Coimbatore!

The one surprising thing one was the fact that the lowest temperatures occur between 2:30 AM and 5:30 AM not around midnight. The highest temperatures are around 2:30 PM not noon. Well the zero is definitely an error since it shows up at 11:30 AM.

Update 1 (March 16 2009):

You can download [~700 KB, rename it as cbe.csv] the big csv file which contains the data for the weather in Coimbatore for years 2005 — 2008. Now you can skip to the Step (3) and use R to analyze the data.