Evaluation events – Commonwealth Scholarship Commission in the UKhttp://cscuk.dfid.gov.uk
The Commonwealth Scholarship Commission in the UK (CSC) awards scholarships and fellowships for postgraduate study and professional development to Commonwealth citizensWed, 13 Dec 2017 18:09:14 +0000en-UShourly1https://wordpress.org/?v=4.9.1Tracing the outcomes of study abroad scholarshipshttp://cscuk.dfid.gov.uk/2015/10/tracing-the-outcomes-of-study-abroad-scholarships/
Mon, 12 Oct 2015 14:47:32 +0000http://cscuk.dfid.gov.uk/?p=8385The week of 15-18 September 2015 saw the 27th annual conference of the European Association for International Education (EAIE) held in Glasgow, UK. As part of our ongoing investment in new methods and strategies for understanding the outcomes of scholarships, Dr Matt Mawer from the CSC’s Evaluation team joined a panel of colleagues from EP-Nuffic and DAAD to discuss ‘Tracing the outcomes of study abroad scholarships’.

For his contribution to the panel, Matt highlighted three main types of challenges facing evaluators:

Tracing or tracking challenges

Measurement challenges

Analytic challenges

He also presented several possibilities for working across academic, professional, governmental, and non-governmental communities to overcome or circumnavigate some of the more difficult facets of understanding scholarship outcomes. Below are a few highlights from his contribution.

Tracking challenges

To get an appreciation of the timescale involved in tracking scholarship recipients, it is useful to reflect that if one were interested in the early career experiences of doctoral students then the course of an evaluation may run for 15 years.

This is a long time to maintain contact with alumni and engage them to participate in evaluation research. Additionally, evaluators want to track outcomes as they happen and inform policy in response, not to react to policy decisions ten years or more after their effects are felt.

As fellow panel member Dr Martin Bruder (DAAD) noted, longitudinal studies – rather than retrospective snapshots – are becoming increasingly important in scholarship evaluation because they allow tracking over time and not only ex-post-facto evaluation.

Measurement challenges

Scholarship programmes frequently use cross-national surveys, an instrument designed for collecting information from alumni in different countries. Often the context is very wide – sometimes globally – and frequently concerned with processes like employment trajectories that might work differently in each location.

One challenge faced by evaluators is that, while these tools suit their particular evaluation, they often produce largely incomparable data if they are not coordinated internationally to be used in the same way by different evaluators. In many cases, evaluations tend to measure at least some of the same variables: employment outcomes, contacts maintained with host countries, and research activity, among other topics.

With these convergences, it should be possible to develop several core areas in which outcomes can be tracked across many scholarship programmes – a ‘post-scholarship trajectory survey’ – and thus produce useful combined data for wider analysis.

Analytic challenges

Sometimes in analysis of scholarship outcomes there has not been close examination of how the academic and non-academic components of scholarship programmes might lead to particular outcomes.

What features of courses help to promote future civic involvement? What features of courses promote return home, and what features encourage staying abroad? What role do institutions play in preparation for returning home or in the reintegration of students when they do return?

Answering these kinds of questions will require close collaboration between scholarship funders, evaluators, and academic institutions to link up data on post-scholarship outcomes to data on study experience. Understanding the wider consequences of particular programme designs for the long-term trajectories of scholarship recipients is an exciting project for scholarship providers, evaluators, and host institutions to take up together.

The need for collaboration

Working in collaboration to ‘trace the outcomes of study abroad scholarships’ has great potential to improve our understanding.

There are topics we can investigate better together, because we have more data, more experts giving collective input, and more alternative scenarios for how scholarships may operate. And there are topics we can only investigate together: such as whether there are any synergy effects or interference effects between programmes operating in the same disciplinary and geographical space.

In addition to their other collaborative work, the CSC will also be contributing to the Association of Commonwealth Universities’ (ACU) new blog series, Measuring Success?, which invites guest contributions on the outcomes of international higher education scholarships and of evidence-informed policymaking.

]]>CSC evaluation at the Donor Harmonisation Grouphttp://cscuk.dfid.gov.uk/2014/07/csc-evaluation-at-the-donor-harmonisation-group/
Fri, 04 Jul 2014 15:02:21 +0000http://cscuk.dfid.gov.uk/?p=6561Each year, a network of organisations administrating higher education, research, and capacity building development cooperation programmes meets in Europe to share insight, collaborate, and, where possible, align objectives. The Donor Harmonisation Group this year met on 11-13 June in Helsinki, Finland, for their annual forum, hosted by the Centre for International Mobility (CIMO).

The CSC was represented by Dr Matt Mawer, from the Evaluation team, and Dr Jonathan Jenkins, Director of Operations. They both participated in a forum panel discussion on ‘Effects on capacity building from scholarship programmes: What do evaluations and tracer studies teach us?’

With colleagues from SIU (the Norwegian Centre for International Cooperation in Education), Nuffic (the Netherlands), and DAAD (the German Academic Exchange Service), Matt and Jonathan led a discussion on recent evaluation research, the current state of methodology in evaluation design, and challenges for the future.

Colleagues from elsewhere in Europe and Africa updated the forum on developments in the Pan-African University and Erasmus+ (the extension of Erasmus Mundus beyond the EU), and on the growing network of higher education connections between China and Africa. A significant theme of the event focused on facilitating dialogue between implementers of cooperation programmes and scholarships and their funders, including contributions from the UK Department for International Development (DFID).

]]>Alumni Fellows focus group discussionhttp://cscuk.dfid.gov.uk/2014/01/alumni-fellows-focus-group-discussion/
Wed, 08 Jan 2014 11:32:26 +0000http://cscuk.dfid.gov.uk/?p=5920The Commonwealth Scholarship Commission’s (CSC) Evaluation team held back-to-back focus group discussions with Alumni Fellows on award and a group of supervisors on 12 November 2013. The Evaluation team viewed this as a valuable opportunity to not only attain in-depth information regarding the Fellowship scheme itself, but also as a way of verifying specific outcomes of Fellows’ first PhD award and the perceived added value of studying in the UK.

Specific objectives of both focus groups were:

To collect qualitative evidence regarding impact of both the Commission’s PhD and Fellowship awards;

To determine whether the current three month structure of the Fellowship was reasonable;

To foster working relationships between the Evaluation team, the Fellows and supervisors, with the added aim of identifying potential case studies for future work.

To feed into information the Commission’s strategy and advising on how this Scheme might be carried forward.

The focus group with the Fellows was broken up into five areas: the first investigated their initial PhD Commonwealth Award; the second their Academic Fellowship; followed by their experiences during the Fellowship and their aspirations post Fellowship; lastly their views regarding CSC Evaluation. Many interesting themes arose, particularly supporting the notion that the experience of spending time in the UK does add value to both PhD studies and to professional development, and that tangible outcomes from the awards are produced.

The focus group with the supervisors was structured similarly to that of the Fellows and yielded comparable results to the Fellows’ session, but also provided an interesting perspective into supervisors’ experiences. Interestingly, supervisors remarked on the benefit to the UK that Academic Fellowships, and indeed scholarships, impart. It was noted that Fellowships serve to enhance cultural diversity in the workplace and promote international collaboration, which are invaluable assets to the UK.

The Commonwealth Scholarship Commission’s (CSC) Evaluation team held a focus group discussion on 20 November 2012 with a group of Professional Fellows who, at the time, were under award on the CSC’s scheme. The purpose of the focus group was to collect robust qualitative data that answered key questions regarding pre, during and post award experiences, which ultimately fed into a review of the Professional Fellows Scheme, a report that was later written evaluating the scheme to date. The event served as a platform for Fellows to voice their opinions regarding the programme, and was therefore both an important information-gathering session and a valuable networking opportunity.

The focus group was attended by 15 Fellows from Sub-Saharan Africa and South Asian countries, with the majority of Fellows undertaking a placement within an NGO, charity or community organisation, whilst a few had taken placements at education institutions and within private sector companies.

Overall, the focus group revealed very positive results in regard to the Professional Fellowship Scheme. Fellows showed keen enthusiasm toward the programme, and all of the participants asserted that it provided valuable skills and experience both professionally and within their personal lives. Subsequently, the CSC Evaluation team has implemented a new method of data collection that surveys Professional Fellows separately from other alumni due to the unique structure of the programme.

Host institutions focus group discussion

The CSC’s Evaluation team held another focus group discussion with host institutions on 6 December 2012. It was attended by representatives from 14 different organisations working within education institutions, charity organisations and the private sector. The main aim of the hosts’ focus group was to gain an insight into first hand experiences and motivations regarding their participation in the Professional Fellowships Scheme. It was also an opportunity for the Evaluation team to build longer term working relationships with host organisations and a useful networking event for hosts themselves. The Environment Agency, a host organisation that has been part of the scheme since its inception, made a presentation and provided an insightful account of their evaluation programme.

The focus group was a success both in the sense that it provided the Evaluation team with an invaluable opportunity to learn about the experiences and ideas of host organisations, but also because it gave host institutions the chance to learn and share with one another for the first time.

These focus groups represent an on-going type of evaluation that the CSC hopes to engage more with in the future.

]]>Measuring the impact of international scholarshipshttp://cscuk.dfid.gov.uk/2012/05/measuring-the-impact-of-international-scholarships/
Fri, 11 May 2012 15:40:37 +0000http://cscuk.dfid.gov.uk/?p=3307The CSC hosted a workshop on 16-18 March 2012 at Cumberland Lodge for around 45-50 representatives from scholarship providers, funders and NGOs, as well as current Commonwealth Scholars. The aim of the event was to share knowledge and experience of good practice in evaluating international scholarships and to identify the best methods for measuring impact.

A similar event was held at Cumberland Lodge in May 2007 which resulted in the first three phases of the CSC Evaluation Programme. As the CSC moves into a new stage of activity, the aim of the 2012 event was to strengthen its relationships with similar programmes and funding bodies, and also to generate mutually useful discussions on the best way to tackle impact evaluation in scholarship programmes.

discuss issues arising from the challenges of evaluating scholarship programmes and similar capacity building programmes, and identify potential benchmark indicators for future comparisons

learn more about the various evaluation methodologies that can be used by such programmes and identify those which are the most appropriate

explore the possibility of establishing a forum of like-minded professionals that allows for future collaboration between different organisations, so that best practice and experiences can continue to be shared

Many participants also attended a complementary event, Measuring Impact of Higher Education for Development – organised by the London International Development Centre (LIDC), in partnership with the Association of Commonwealth Universities (ACU), and held on 19-20 March 2012 at Birkbeck College, University of London. The conference looked at the broader picture of higher education and development impact.

]]>Launch of the ‘Assessing impact in the health sector’ reporthttp://cscuk.dfid.gov.uk/2010/05/launch-of-the-assessing-impact-in-the-health-sector-report/
Fri, 28 May 2010 11:00:28 +0000http://cscuk.dfid.gov.uk/?p=2983The CSC evaluation report Assessing impact in the health sector was launched on 27 May 2010, at the London School of Hygiene and Tropical Medicine.

Guests included current Commonwealth Scholars and Fellows, members of the CSC’s panel of Academic Advisers, supervisors from UK universities, and representatives from Professional Fellowship host organisations in the field. Dame Sally Davies, Director General for Research and Development at the Department of Health, also attended.

]]>Alumni evaluation surveyhttp://cscuk.dfid.gov.uk/2008/11/alumni-evaluation-survey/
Sat, 15 Nov 2008 12:00:13 +0000http://cscuk.dfid.gov.uk/?p=1078The evaluation survey was sent to 5,673 alumni for whom we had contact details in early 2008.

We asked about:

location, field, type and level of employment

the level of skill and experience gained attributed to the award

the extent to which alumni are able to use these skills

the level of contact maintained with organisations in the United Kingdom

the extent to which alumni have been involved with projects, policymaking and wider socioeconomic impact

The main results of this survey were also published in a report that was launched and distributed at the 2008 Welcome Day for new Commonwealth Scholars and Fellows.

]]>Evaluating the impact of international scholarshipshttp://cscuk.dfid.gov.uk/2007/05/evaluating-impact-international-scholarships/
Thu, 10 May 2007 11:00:16 +0000http://cscuk.dfid.gov.uk/?p=1076The CSC held a workshop to launch its Evaluation Programme from 4-6 May 2007 at Cumberland Lodge, Windsor, UK

Entitled ‘Evaluating the impact of international scholarships’, the workshop was attended by representatives of the World Bank Institute, the German Academic Exchange Service (DAAD), the Netherlands Organisation for International Cooperation in Higher Education (NUFFIC).

Other attendees included representatives from Commonwealth Scholarships national nominating agencies in Canada, Ghana, Malawi, the Maldives, Malta, Mauritius, New Zealand, the UK and Zambia, as well as current Commonwealth Scholars and Fellows and other interested parties.

With such a variety of delegates, the event offered a forum for discussion and an opportunity to compare approaches to evaluation. Methodologies for evaluating capacity building for international development are well established, but it seems that reliable data for the impact of scholarships, in particular, are not so well documented.

Those involved in discussions were keen to establish a better view of the impact of scholarships and, where possible, to collaborate and share the data generated. Guidelines were drawn up to determine what information to collect, to design a framework for acquiring this information, and to develop best practice in evaluating the impact of scholarship funding in capacity building.

Dr John Kirkland, Executive Secretary, Commonwealth Scholarship Commission in the UK: The Way Forward: Proposals for Future Collaboration

Commonwealth Scholars and Fellows’ views

Melissa Felician (Commonwealth Professional Fellow, Institute of Policy Research in Engineering, Science and Technology):

Having experience in evaluation, including a recent ‘Evaluating Research’ module at my host institution, methodological issues were not new to me, but it was good to see attempts to integrate all relevant aspects. Points relating to quantifying benefits – which I initially saw as a major challenge – were well presented and provided food for thought.

The session on the recipient’s viewpoint was the most interesting part; I came away from that session acknowledging that the impact of the scholarships on the recipient (country and individual) goes beyond the measurable outcomes of the scholarship programme.

While much still needs to be done in building capacity at country level to improve the selection process, it was clear that the CSC is changing many lives by the opportunities that it has created over the years!

Imran Majid (Commonwealth Scholar, MPhil in Engineering, University of Surrey):

Highlights of the event for me included a cost-benefit analysis of the scholarship business and a review of academic and development approaches to scholarship evaluation. The World Bank Institute’s tracer study programme and the German Academic Exchange Service (DAAD)’s success in coordinating alumni activities inspired brainstorming sessions.

My contribution included ideas for prioritising regional cultures in impact assessment, while on the other hand applying the onion-shell concept (from a presentation at the UN Space Generation Forum, 1999) in scholarship evaluation; the individuals (inner core) through skill and experience affect institutions (middle layer) directly and through actions benefit the society (outer shell) indirectly.