Unexpected challenges, new opportunities, and an entirely new evaluation design: the evaluation work we’ve been doing this summer has taken numerous twists and turns as we adapt our initial design to the actuality of conditions here in Tbilisi.

We spent the spring semester designing an evaluation for the McLain Association for Children (MAC) as part of a project our 4-member team did for Dr. Heinrich’s program evaluation class at the LBJ School. We knew we were looking at the impact of MAC’s trainings for teachers of students with disabilities. We knew that our results would help MAC document the benefits of their work and make changes going forward. But our focus was very different.

We originally planned to evaluate the quality of Individualized Education Plans that teachers learn to write in MAC’s trainings. We wanted to evaluate the effect of these IEPs on children’s learning outcomes, but we knew this would be problematic due to a lack of a standardized measure of students’ levels. But we were going to try.

We found out very quickly once we landed in Tbilisi, that writing IEPs is only a small part of what MAC does. Interacting with actual people who had been the target group of the intervention helped us understand the goal of the intervention in a different light. As we talked with staff and interviewed training participants, we found that the real focus of the trainings is on the increase in knowledge of disabilities. Teacher after teacher told us they took the training in order to improve their ability to diagnose disabilities and to utilize the appropriate corresponding educational strategies to help each child develop and learn. Following our focus group discussions and interviews, we made changes in our initial design to shift the focus from IEP writing to knowledge transfer.

Returning from a site visit where we tested for knowledge of disabilities.

Another challenge that we ran into was the accessibility of institutions where we wanted to conduct the tests and focus group discussions. The Georgian educations system has been undergoing a comprehensive infrastructural reform since 2009. One result of these reforms has been a consolidation of some institutions and the closing down of inefficient institutions. So institutions we contacted were a little nervous. In addition, the position of school directors at public institutions is largely based on contacts and political affiliation. As the election season is approaching in Georgia, directors tend to be more on edge. They wanted to make sure that they do not allow any activity in their school that would in any way harm their reputation with the central government or that could provide the government with an excuse to close down the institution.

Some of the institutions flatly refused to meet with us. In some cases we were able to conduct tests solely because the MAC director used his personal contacts to assure the directors that we were not at all connected to the government. Once we gained access to the institutions there were even more challenges awaiting us.

The biggest challenge for us, in terms of data collection, was the collaborative culture of taking tests in Georgia. The education system in the country encourages collaboration even in individual assignments or at least the system does not enforce serious penalties for such activities. Despite the fact that we made it very clear to the respondents that the purpose of the tests was in no way to evaluate their performance, we had to use frequent reminders during the test that it is extremely important for us that they complete the tests on their own. It was quite the culture shock for us to watch teachers surreptitiously whispering answers to each other, or glancing over their colleagues’ shoulders, while completing the exams.

As we were to figure very soon, not knowing the native language was also a major hindrance in our information gathering exercise. Although we always had translators, it was very hard to connect with the focus group participants and we got a feeling that quite a lot could be lost in the translation. For our first two site visits, we only had access to a MAC psychologist as our translator. While this helped us immeasurably with gaining context as we visited sites, we worried that we were receiving only positive feedback as a result of her presence. MAC hired translators for our remaining visits, but we then lost some of the accuracy of the translation that we had gotten from her content expertise. (As an aside, we still received only positive feedback from participants, even when we specifically requested critiques.)

Also, if we ever had any hopes to learn the Georgian language, they were quickly dashed when we started practicing our new vocabulary in front of co-workers. The Georgian language contains many distinct sounds unlike any of the five languages that we collectively speak. Legend has it that ancient (and probably modern too) Georgian armies would ask people who were suspected of being spies to repeat certain Georgian words that contain these unique sounds. This was the ultimate test of Georgian nationality, failing which they would be punished for being spies from another country, particularly Russia. When we attempted to pass the test, our friends dissolved in uncontrollable giggles, erasing any hopes of reaching a level of respectable conversation before the end of the summer.

We enjoy a lunch of mushroom khinkali at a Tbilisi restaurant.

Fortunately, Georgian cuisine is much more approachable than the language. We quickly mastered the art of eating khinkali, dough dumplings most typically filled with pork, beef, cheese or mashed potatoes.

We also made sure to share our own cuisines with our co-workers. After a day of grappling with evaluation challenges together, it was important to find time to relax together as well.

Life has taken me back 12 years. It was in year 2000 when a few years after having graduated as an engineer and working as a software developer I landed in Hyderabad (India). Part of a start-up dreaming to make it big during the dot com boom. From the moment I was here, I fell in love with the city. The coconut palms, the southern breeze and great work. After a few months and having exhausted all avenues of keeping myself busy and entertained on weekends, I ended up in a hamlet right behind Golconda fort. I was looking to volunteer, and the community here – especially women – were looking for support. After a few conversations, we settled on a long-term action plan. I would give the women the money to buy raw material and the tools, and the women would do what they were best at – Hand embroidery, weaving and knitting. That’s how Fateh (meaning victory) was born. The size of the group grew and so did our confidence. Now I was convinced that projects helping the poor were possible, and you needed no charity for that. With the Fateh women on their own, I moved back to Delhi where my family was. It took me some time but finally everyone gave in to my plans of starting a non-profit. So in the fall of 2001, The Vigyan Vijay Foundation (V.V.F) came into being. The grassroots organization is chugging along, having touched more than 10,000 lives in its 11 years.

In 2006, I gave up my Program Director’s job in V.V.F, though I am still on the Board. It is always heartwarming to see how much good V.V.F is doing in its community, but there is still something missing. There are many V.V.F’s in India, all working tirelessly and incessantly to make the lives of those around them better. But what we are able to do is only a miniscule part of what the country really needs. How much difference can grassroots organizations like V.V.F make without changing the system? For a few years I went around working with other non-profits, talking to thinkers, doers and academicians. Slowly, the idea of going to school took root. I learned about the graduate programs in public policy in various schools in the United States, and one of them had me hooked. Its tag line said – What happens here changes the world. That’s how I came to LBJ.

In the first year of the MPAff program, I struggled – trying to always apply what I am learning in the context of my country and in the rest of the developing world. Without ever planning, my interests and my work drifted towards the field of health care and health care policy. It was fitting since health care has become the biggest challenge not just in developing countries but also in the most developed ones. When looking for my internship, I knew I had to be in an organization working in the developing world with a focus on health care.

ACCESS Health International is everything I was looking for. It is closely working with the government of India, state governments, the World Bank in India and other international development organizations like Results for Development. It is actively working on building a knowledge base that will bring to the foreground various innovations happening in the health sector in India and the rest of the world. At the same time, it is working to build a community of practice that will bring together different stakeholders in the sector to share their resources and expertise. Most importantly, it is building up as an interface to government at different levels, and becoming the government’s go-to for finding data on different aspects of health care access and delivery.

In my 10 weeks here, I am studying specific for-profit models in primary care that have a scope for adoption by the public sector. We will also be studying the scope of public-private partnerships (PPP) in health care that can be based on these models, as is or with some modification. In a team lunch, Sofi Bergkvist, executive director of AHI, shared her experience from a conference she had been to. There were participants from Brazil­ and South Africa. They were talking about PPP’s being implemented in their countries over the last few years and the structural framework they have built to implement and support these partnerships. India has had PPP’s for over a decade now, we are still lacking however a standard institutional and structural framework in the field. It’s about time we shift our focus from grassroots innovations and work to build a policy framework that will take the best of these innovations and reach the whole country. That’s the mission AHI is following, and I am lucky to be part of this crusade.

Hi from Panajachel (Pana), Guatemala! This beautiful lake town offers a small-town feel, but with constant activity from the artisan kiosks, street food vendors and tuk-tuks (motorcycle taxis).

My first 2 weeks have been eye-opening to say the least. I am working with a non-profit called Mayan Families that does great work in this town and all around Lake Atitlan. I have never known an organization that did as much and in so many capacities as this one does. Through generous donations from individuals and organizations/communities, primarily in the U.S., Mayan Families makes a difference in so many people’s lives. The organization enables impoverished children to go to school, provides clean water and fuel-efficient cooking stoves to families, runs food shelters for elderly indigenous people and orphans, helps in emergency housing situations, and has even organized vet clinics to help the stray cats and dogs here in Pana. Mayan Families also plays a vital role in making sure its clients receive adequate medical attention when necessary; whether it’s transportation to see a doctor in Guatemala City, or help covering the costs of a life-saving surgery.

So far I’ve been getting acquainted with the various projects and services that Mayan Families provides through site visits, and I have been working on organizing its Community Service program that the older scholarship recipients are expected to participate in. One of the projects I will soon be working on is making sure that the beneficiaries of donations get the things they truly need. I am excited to get started on this project and to get to know several of the families that Mayan Families is working to help. I know that I will probably encounter trying stories and circumstances, but hopefully I can make at least a small impact.

My name is Becca Moore. I am pursuing my Masters in Global Policy Studies at the Lyndon B. Johnson School of Public Affairs, focusing on international development. Now granted the opportunity to apply my academic knowledge to a practical setting, I have begun a rigorous internship with People Helping People Global (PHPG), a non-profit micro-lending organization based out of Vermont. PHPG offers micro-loans to individuals in Nicaragua who earn less than $2 a day, enabling disadvantaged entrepreneurs to start or expand their business. The majority of the loan recipients live in Granada, which is where I will be based for the duration of my internship in Nicaragua.

While in Granada, I am working directly with PHPG’s Nicaraguan employees, Gilberts and Marcela, and receiving guidance from the organization’s co-founders, Alex and Isabel, who are currently managing PHPG’s microfinance activities from the States. My role within the organization is to assist the local staff in the loan process, to identify ways in which to increase PHPG’s efficiency and effectiveness, and to act as a liaison between the donor community and the loan recipients.

This past semester I researched how health education programs have been combined with micro-finance programs. My goal is to apply what I learned through this research to help develop and implement a health education component to the loan process. I plan on working with the local employees, loan recipients, local health professionals, and fellow interns to explore this idea.

Through shadowing Gilberts and Marcela during the repayment meetings, I have become familiar with the community and some of the loan recipients. This week, we (the local staff and summer interns) began to interview prospective loan recipients for a new set of loans that will be given out in August. The initial interviews both assess the applicants’ respective incomes and business plans to determine whether they qualify for loans and gather information about their families and quality of life. Over the next few weeks, we will be interviewing a total of 75 new recipients.

We enjoyed many a Georgian feast while we were recovering from Jetlag in the village of Dzevry.

While we both have international policy interests, neither of us has regional expertise in the Caucuses. We chose this internship due to the importance of disability as a development issue in the region and the lack of prior research on this subject. This internship will give us an opportunity to learn, share our knowledge and contribute to the development of disability policy in the region.

We began work on this as part of our semester-long project in the Program Evaluation class with Dr. Carolyn Heinrich. Our four-person team designed an evaluation for one component of MAC’s work. We traveled to Georgia to refine the design, to expand the scope to other components, and to begin implementation.

During our first week here we first spent time recovering from jetlag and observing Georgian culture in a laid-back manner. We spent four days in a village 3 hours from Tbilisi, toured a facility that will soon begin working with MAC, and ate, and ate, and ate.

We are interviewing the director of one of the institutions MAC has trained. Nino Lomidze (right), a psychologist with MAC, translates the interview.

As we begin our formal workdays, we have outlined our goals for the summer. We will be producing an evaluation report by the time we leave. Our focus will be on process evaluation of the implementation of MAC’s trainings at the different institutions they work with. Of the 27 locations where they have conducted trainings, our goal is to conduct in-depth evaluation of nine. We will be conducting focus groups and key informant interviews in order to obtain the opinions of trainees as to what has been most useful for them. We will also be able to get a better picture of what elements of training they are implementing on a regular basis. In addition, we will administer a second round of the post-tests that MAC uses to test the participants’ knowledge at the end of training. We want to see how much knowledge they are retaining. Moreover, we plan to gather information from the rest of the institutions through survey questionnaires. We will also run some basic descriptive statistics of the results of the original post-tests for all 27 institutions.

Professionally, we are most looking forward to seeing how our theoretical evaluation design from class works when we implement it in the actual situation it was designed for. We are also very excited to have an active part in MAC’s work to bring best practices to Georgia in education and care for individuals with disabilities. This evaluation study is going to serve as an important resource for other institutions in the region with similar programs. The results of this evaluation study can be used for enriching the educational policy discourse in the region and developing evidence-based educational and health intervention programs for community members with disabilities.

Neither of us had much experience with education of children with disabilities in our own countries. We have learned much already about this important sector of education. It is also fascinating to learn about Georgian culture in the context of education for children with disabilities.

My first couple of weeks working at Development Gateway has been great. Our geocoding team has students from LBJ, BYU and William and Mary. We all sit together cozily in a big conference room. But as much as I enjoy my workdays, it is my evenings and weekends in DC that I really love. There is so much to do in this city. The past two Fridays we walked down to the National Gallery of Art for Jazz in the Garden. On our way there the first time we stopped to watch President Obama’s helicopter land on the White House front lawn. I am pretty sure we saw his feet on the other side of the helicopter. After our picnic and some free jazz we headed to Chinatown to eat a variety of delicious Asian foods, including $1 sushi and Peking duck.

Last week the World Bank held their Innovation Days event at their headquarters. The event focused on innovations and technologies that are being applied to international development. Their lobby was filled with booths touting innovative projects that use technology to improve the lives of millions of people in developing countries. We listened to panels discussing the latest innovations in East Asia and we saw a film screening on sustainable green energy development around the world. We also had a treat Sunday night when LBJ School Dean Robert Hutchings invited us to a reception hosted by the Ben Barnes Group at the Roosevelt House for the Health Privacy Summit which took place in D.C. this week. I feel like I have done so much and I have only been here two weeks. I am looking forward to seeing what the rest of the summer has in store.

My internship with Development Gateway (DG) in Washington D.C. is off to a great start. First of all, I love this city. As an international development professional and a news/politics junkie it feels great to be at the heart of the action. I am living in an apartment in Georgetown with three other LBJ students. Savin and Anustubh work with me at DG and Jessica Tibbets works at the State Department. Our apartment is one block off M Street and 100 meters from the Potomac River. Morning runs along the riverside trails and evening picnics in the riverside park have already begun. My first day at work was 31 May and that evening the Washington Area LBJ Alumni Chapter coordinated a happy hour that was a big hit and a great way to kick off my summer here.

Development Gateway is in the middle of the downtown D.C. scene. We are across the street from the IMF and the World Bank, so we go over to their cafeteria to eat lunch. If I ever see Robert Zoellick there I will pull up a seat next to him and then I will really have something to blog about. The work I am doing for DG is geocoding all of the donor aid that is being given to the country of Malawi, Africa. It is one of the first projects of its kind and when it is successful, other countries will hopefully want to do the same. Basically, the government of Malawi provided documentation related to every project that uses donor funding. There are 33 donors and hundreds of projects across all sectors – health, education, economic development, infrastructure, agriculture and governance. I read through the project documents, identify where the money is being spent (cities, villages, districts, etc.) and capture that information so that the project can be mapped using GIS (Geographic Information System).

A similar effort was made last year and resulted in the World Bank’s “Mapping for Results” platform - For this effort, the team mapped every project in the world that the World Bank funded. Our current work will produce similar output, but for every donor and every project in Malawi. Many groups benefit from the work we are doing. Donors are better able to assess the needs of Malawi and guide funds to areas that are underserved. The government of Malawi will be able to identify gaps in donor-funded services and respond accordingly. But most importantly, the people of Malawi will benefit by getting the right services where they are needed the most. I am very much looking forward to all of the work and play I will doing in D.C. this summer!