Community of Practice Evaluation Strategy

As discussed in the case study post on context, this initiative is expected to contribute to reducing equipment down-time and decreasing risk associated with concentration of deep process knowledge in a small number of long-tenured Engineers. A number of Key Performance Indicators (KPIs) were identified up-front to evaluate the value created by this community. This was informed in part by the conceptual framework of value creation from online interactions from Etienne Wenger, Beverly Traynor and Maarten De Laat. See Jane Bozarth’s article for an introduction to the value creation cycles in the framework.

KPIs and Measures

Measurement Points & Methods

Three main measurement points have been identified:

Prior to community launch – to establish a baseline

Immediately following the Work Connect & Learn program (guided social learning program to support the development of skills and behaviours to participate in the community)

Six months after completion of the Work Connect & Learn (WCL) program

Monitoring of some of the measures will occur on a monthly basis during this six month period to help inform community management.

Data and feedback will be gathered using the following methods:

Surveys of all maintenance and engineering team members at the three main measurement points

Monitoring of activity on online community spaces in SharePoint

Interviews / discussions with Maintenance & Engineering Managers and Capability Managers (who are providing on the ground supporting to community members to participate)

Focus groups with community members immediately following and six months after the WCL program

Capture of value creation stories on an ongoing basis in the six months after the WCL program

Additional Thoughts on Value Creation Cycles

The careful reader may have noticed that indicators of Reframing Value have not been explicitly included in the KPIs. While we have not explicitly set objectives for value at this level, it will be surprising if this does not follow success in the other value creation cycles. Reframing Value will be identified through value creation stories.

The value creation framework has recently been expanded to include strategic value and enabling value. We’ve not yet considered whether / how to include these in our evaluation strategy.

always interested and intrigued on how people approach evaluation as it’s so often done poorly, if at all. Some good ideas here, I like the focus on value creation as opposed to just simple statistical measures of usage. My fave measure: “serendipitous stories of joint projects and problem solving”! I am a fan of qualitative data and think narrative approaches to evaluation provides such a rich source of data – over and above the stats. And taken together the range of your measures should provide a really nice holistic view of the outcomes.