Working Group 4: Demonstrating Impact

Mission Statement

Using data for a change: This group's mission is to identify, explain and disseminate information on metrics that hold the potential to document, foster, accelerate, and communicate systemic change. Documenting, by creating empirical evidence for the effectiveness of various approaches for changes in undergraduate STEM education. Fostering, by enabling stakeholders to reflect upon and assess current conditions. Accelerating, by using existing measures more broadly and effectively and by identifying important elements of change that are not being adequately assessed. Communicating, by establishing a common language and set of expectations for studying change. Identification involves both cataloging what is available and suggesting what might be usefully developed, including the potential for a set of common change metrics. Explanation includes a discussion of the implications of each metric, illustrated by cases that describe how metrics have been used successfully to foster change. Dissemination comprises making this work readily available in an accessible format via a community-curated website.

Knowledge Claims and Established Ideas

Key Questions

Who are the stakeholders who need to be informed of the metrics that can and should used to establish baselines and monitor progress?

How do we inform these stakeholders of what can and should be measured?

What are the impediments to the use of existing metrics and what can be done to reduce them?

What metrics do the stakeholders believe they need and currently engage, and how does this vary by institution and student type?

What metrics do we believe they need, and how does this vary by institution and student type?

What do we know about how these metrics are now being measured, and which instruments will need to be built or improved?

How can we help stakeholders effectively engage with these metrics?

How do we infuse this entire process with the importance of inclusivity?

How do we assess our own effectiveness in providing these resources?

Potential Outcomes or Products

Responding. This group will provide consensus feedback on relevant reports and recommendations of national organizations or agencies as the opportunities arise.

Cataloging. This group will create a list of possible metrics and identify an essential subset that will enable comparative analyses. The longer list will serve to metrics that may have significance to only a subset of institutions. As part of the job of creating the full list, this group will identify sources of educational transformation metrics. In addition, this group will identify gaps in what is now measured or measurable.

Engaging. This group will engage other people or groups as part of the ASCN conversations, especially people or organizations currently working with measurement and evaluation.

Communicating. Once this group has products to share, it will make a concerted effort to communicate their existence to the stakeholders, drawing on existing research to identify the questions and concerns of change agents so that it can tailor its outreach to what is interesting and relevant to them. In particular, this group will strategize how to reach adjunct/contingent faculty and course assistants, stakeholders who are often difficult to reach via traditional channels.

Action Plan for Moving Forward

2017/18

We are engaging the Working Group on identifying concrete actions needed to move this work forward. As an example, two such actions that we are now contemplating are:

The identification of and sharing of information among organizations that are currently engaged in this activity. One example is the Research Advisory Group of the Conference Board of the Mathematical Sciences (see http://www.cbmsweb.org/research-advisory-group/)

Soliciting information on the pressing concerns of mid-level academic administrators to better understanding where information on measurement tools and data can have the greatest impact. As a first step, we will be asking the 34 chairs in the Mathematics Advisory Group for Transforming Post-Secondary Education in Mathematics to each identify two or three of the most pressing issues they are facing and for which information on measurement tools and data might be helpful.

2016/17

Kacy Redd is coordinating the response of this working group to the BoSE draft of Developing Indicators for Undergraduate STEM Education

Over the coming year, the group will collect lists of indicators for STEM education, including the AAU's STEM Teaching Indicators, and determine consensus on which of these covers should be in a master list that covers educational activity at organizational levels from individual to institutional to network and which are usable and attractive to a broad range of users.

Over the coming year, this working group will make contact with the following efforts to understand how they are using measurements of STEM indicators:

Understanding Interventions

AAU STEM Initiative

University Innovation Alliance

Tools for Evidence-based Action

Over the next two years, identify the stakeholders with whom this working group will need to communicate and begin to shape our message