Field workers being trained on the Collect app, which was used to digitally collect data from every household.

Jiwati taluka (or sub-district) in Maharashtra’s Chandrapur district is often called the “punishment block” in government echelons. Almost on the Andhra border, it takes two hours to reach the thickly forested area from district headquarters. Unlike in the surrounding talukas, where Marathi is spoken, the Telugu and local variant of the language resident tribes speak here poses a barrier to an outsider. And not to mention that phone networks reach only select “hello points”—elevated areas in villages that have limited connectivity (only 2G here, no fancy high-speed data connections).

In this remote, inhospitable setting, a mammoth task is under way—a survey to gather data in villages on every single individual. The objective: setting up a real-time data system that can help the authorities and communities plan at the local level according to their specific needs.

***

It's January and field managers Gopal Singh and Sandeep Sukhdeve, along with 300 volunteers, are pulling out all the stops in their quest to data-map Chandrapur. Gopal, who is camping in Chandrapur for the task, has his day cut out—he leaves his temporary digs in the district headquarters at 7am in the cold mornings to reach the taluka interiors two hours later, distribute tablet computers and get the data survey started in the 83 villages by 10am.

To make sure respondents understand the questions, he carries out periodic spot checks in a few gram panchayats. He also does back checks on collected data to correct mistakes made by the field staff, many of whom might have never used smartphones or tablets before.

After a long day of data collection, he gathers up the tablets and traverses through the secluded jungles, reaching Chandrapur by 11pm. Given the limited bandwidth in the field, Gopal painstakingly verifies and syncs data in the long January nights. That’s the rigmarole of most volunteers specially trained for the task: collect data during the day and travel to the nearest town at night to upload the data collected.

Jiwati is one of the three talukas of the district being surveyed for the Sansad Adarsh Gram Yojana, a rural development programme the central government launched in 2014. The programme mandates that MPs identify and develop model villages in their constituencies, not just by improving the physical infrastructure, but also by pushing for complete socioeconomic development.

When the Tata Trusts, which work with other trusts on community development, offered to support MPs and MLAs in this regard, Maharashtra's minister for forest, finance and planning, Sudhir Mungantiwar, asked them to develop a holistic development plans for his home district—Chandrapur.

Mungantiwar, the “Bhau” (older brother) of Vidarbha had been rooting for the region’s development over the last few years. But there was only so much that could be done when planning with paper-based surveys, registers and files. It was clear that things had to be done differently. “If we have to turn a village into a model village, I feel data lies at the core of this exercise,” he says in a video interview SocialCops filmed while the project was being carried out, “Without data... our transformation efforts would be incomplete and directionless.”

Solutions for a country as diverse and complex as India, where even parts of a district may be starkly different from each other, can be baffling, says Prabhat Pani, head of partnerships and technology at the Tata Trusts. “That is why geography- and citizen-focused microplanning is the way to go,” he adds, “to capture village-level nuances that centralized datasets lack.”

The Tata Trusts partnered with data intelligence company SocialCops to completely rethink district planning. Their brief was formidable: create a comprehensive data-based microplan for each of the 290 villages across three talukas of Chandrapur—Pombhurna, Mul and Jiwati.

Development like online search?

“Traditionally, planning for development has always happened top down,” says Prukalpa Sankar, co-founder of SocialCops. “So, we thought of collecting data for solutions that targeted development the same way the online world does—you search for a Goa holiday package and sure enough, (ads for Goa vacations) appear every time you open the browser.”

"With data mapping, is it possible to similarly target people and personalize the solutions for each of their needs, be it for education, health, sanitation, or livelihood?” she says.

The needs of every household, every individual, needed to be mapped to the resources, helping bridge the gap between welfare outlays and beneficiaries. That could happen only with unique targeted development plans of the most granular kind. The SocialCops data intelligence platform helped identify 80 development indicators across five broad categories—economic development, education, health, aanganwadi and welfare schemes outreach—to create village profiles and develop village development plans.

There was, however, a small problem. Microplanning at this level of granularity generally takes anywhere between six and nine months. Chandrapur’s had to be completed in 90 days flat.

It wasn’t going to be easy.

For one, Chandrapur is a study in contrasts. In addition to a handful of cement plants, the district hosts the state’s largest thermal power plant, the largest paper manufacturer and exporter (Ballarpur Industries) and also a manufacturing facility of the country’s largest steel producer, Steel Authority of India Ltd. The city has such an abundance of coal mines that some call it Black Gold City.

Thanks to an abundance of coal mines and coal-based power plants, Chandrapur is called the Black Gold city.

At the same time, it is home to over 2 million people who mostly live in dismal conditions, relegating Chandrapur to the list of India’s 250 most backward districts. The district as a whole mirrors what ails Jiwati—half the houses are kutcha, grid connectivity in areas is as low as 31% and computer literacy no higher than 5%. More than a fifth of the population comprises Scheduled Castes—the highest in the state—and about 10% of households use LPG for cooking. The district is in dire need of development initiatives.

Mul is a mixture of rural-urban pockets, Pombhurna had both rural and tribal population, and Jiwati is 90% tribal. Across most key parameters, the three blocks showed hugely varying development needs. A one-size-fits-all plan would have been utterly meaningless.

Microplanning can be tough even under the best conditions. Gathering granular information is cumbersome and time-consuming. By the time the data is up for analysis, most of it is obsolete. Tata Trusts senior programme manager Paresh J. Manohar estimates that almost 40% of the data dies by the time a nine-month project, like Chandrapur, concludes.

Data works as efficiently as it is gathered

Speed and agility was of the essence. To collect huge volumes of data in a short time, it was vital to get the questionnaire right. The questions had to be framed carefully, in logical sequence and in the local language. Through piloting in the field, data scientists created customized surveys for different villages and demographic conditions, all on an Android app—the perfect data collection tool. “We learned that the same dialect of Marathi can change from taluka to taluka. So we had to change questions to ensure that people understood what was being asked,” says a field volunteer.

Training was organized locally in Marathi to orient volunteers in different aspects of data collection.

None of this would be possible unless both the surveyors and those being surveyed understood the importance of the entire exercise. So, the people were first made aware of what microplanning was and how it would benefit them. Next, it was explained these plans would work only as well as the data they collect. About 300 volunteers—100 per taluka—were rigorously trained in mobile data collection for a week. Imagine educating people from scratch about a tablet – how to operate it, what would it mean to press a button, how to open an app and fill a question, how to ensure the response is registered.

And the volunteers delivered, surveying 160,000 people and collecting 6.9 million data points.

At the same time the data was being gathered, algorithms were used to verify and flag inaccurate responses in real time and re-collect that data, significantly improving the quality of the information.

The final data from the baseline survey was then structured and analysed to create village development plans, listing everything that needs to happen to transform each village into a model village. And at last, a dashboard was built based on the data so that government officials could access it at any time in an easy-to-understand snapshot.

Creating granular to-do plans

At the end of the three months, a composite database was in place for all the villages. In Jiwati’s Patan village, for example, government officials could find out with just a click that of the 250 households, 133 needed ration cards, 127 had no grid connectivity, an overwhelming 204 had no LPG connection, all save one had no access to processed tap water, 185 needed toilets, and so on.

The village development plans listed out in detail what should be on their agenda, even at an individual level and mapped each task to the relevant government department. For instance, only 12 youths in the 18-35 age group were categorized as "skilled" in Patan; 381 needed to be trained under the rural development ministry’s skilling initiative for rural youth, the Deen Dayal Upadhyaya Grameen Kaushalya Yojana.

Or consider social security parameters. The dashboard showed that among those in the 18-70 age group eligible for the Pradhan Mantri Suraksha Bima Yojana (a subsidized insurance scheme), only 90 out of a total of 631 people were covered. The village also drew a blank on health insurance—none of the 174 eligible people (classified as below the poverty line) were covered under the Rashtriya Swasthya Bima Yojana.

Almost a year later, the results are strikingly visible.

A sub-divisional officer in Mul is using the dashboard to find villages where she should conduct Aadhaar camps. She hovers over the red dots in the map to zero in on areas with the lowest use.

The block development officer in Mul wants to track how his block is faring on the Open Defecation Free parameter. The snapshot shows what percentage of households don’t have toilets; he clicks again to find out which villages in particular are lagging behind.

The electricity department can now find out exactly which households are still not connected to the grid. And being the only such database on LPG use in the country, the ministry of petroleum now knows which households to target for providing cooking gas connections.

That’s not all. The district collector, Ashutosh Salil, will now be able to rely on the data dashboard to cross-check in real time what officials claim needs to be done, and also to appraise villagers’ development priorities during his field visits. “Earlier if I had to budget Rs100 for 10 sub-districts, I would distribute them equally,” he says. “That can’t be correct allocation if a block has been covered before and does not need more resources.”

The villages have been sent their plans to be incorporated into their next annual development plans. One out of every three suggestions is a part of the Mul block’s 2016-17 development plan. Based on these plans, minister Mungantiwar has adopted 18 villages, all of which are to be converted into model villages.

Amberrazi, Loldoh, Malguda and 287 others in Chandrapur are no longer just dots on the map.

Aarti Gupta is a Delhi-based business writer and editor, and also a consultant with SocialCops.