On May 6-7, 2008 UPCLOSE ran a two-day CAISE Inquiry Group on the topic of how to conceptualize and evaluate successful professional online learning communities, especially those serving the multidisciplinary field of informal science education (ISE).
A second goal was to exercise the new NSF/ISE guidelines for &quot;Evaluating Impacts of Informal Science Projects&quot; www.informalscience.org/research/show/3643 by putting the Framework into practice with example research designs, instruments and metrics to assess the impact of recently funded professional online communities like InformalScience.org and InSci.org and ExhibitFiles.org This gigapan documents the discussion artifacts from BREAKOUT SESSION TWO which explored the appropriateness of the new Evaluation Guidelines for professional online community projects.

See snapshots for more details.

We are experimenting with sharing the breakout group discussion in this format. Please let us know your thoughts.

Gigapan Comments
(1)

Participant Notes: Focus (of ISE NSF-funded
proposals) shifts from User Needs --> Project
Impacts How to build a proposal or a specific
research project with measurable impacts? -
Portfolio approach (diversification idea: within a
proposal, assess multiple things, multiple
hypotheses, multiple impact areas/categories, so
that at least 1 of them comes out supported/as
expected. Then report back the successes and
failures, the combined reporting of both successes
and failures is a worthwhile significant
"innovative" contribution because
we'll know which things worked, which didn’t
all within a single project). This isn't a new
approach - the diversification idea has been
around in business for a while as a
risk-management approach. (Assessing multiple
impact categories within a single project spreads
the risk across those impact categories and
ensures that a worthwhile contribution is made
from results of both expected and unexpected
findings. In contrast, if only unexpected findings
emerge, which is more likely to happen with
un-diversified single-focus projects, the
researcher might not be able to publish those
results because negative results by themselves
tend to "get buried" and are
under-valued in the field, unfortunately). New
Challenges: - How to define what each impact
category is/should be specifically? The burden is
on the ISE community to figure that out and arrive
at some kind of consensus that helps build
cumulative research around a shared definition of,
say, what "attitude change" as an impact
category should mean. Online Communities as a
cost-effective tool for professional development.
- Really interesting idea (I'll bring that
into my project at some point) but how to go about
planning that? Eg., need to figure out what are
some of the professional development activities
that an online community can cost-effectively
support. Who are the activities for (profiles of
users for whom professional development can be
promoted through the online community: PIs vs
Chloe (student end-user who doesn’t intend to
become an ISE PI/professional but is peripherally
interested in ISE topics of environmental science
and media studies). One thing not brought up in
the Chloe-related discussions was that her
activities are a form of "legitimate
peripheral participation" that communities of
practice literature has discussed as one possible
way of participating and of identifying
users/stakeholders (people not central to ISE or
to the main focus of a community but who are
nevertheless tangentially interested in some
aspect of the community and are therefore a
worthwhile asset for the rest of the community and
whose needs the online community can target in a
different way than those of the main
'core' members). Distinction of online
communities participation: free membership,
anytime/anyhow/anyway participation vs. directed,
focused participation like msp net. Under which
conditions should a more directed form of
participation be encouraged? Units of Analysis:
Discrepancy between NSF "impacts" RFP
(individual-level) and goals/intentions of how to
move the ISE field forward (community-level
impacts). If proposals are required to frame their
impacts at the individual level, wouldn't the
ISE field lose sight of the broader
ideas/thinking/theories in the field, which really
need development? - (my comment): this discrepancy
isn’t really created by NSF's focus on
"project impacts" as presented in
Randi's slides today. Impacts can be defined
and assessed at any level of analysis the
researcher/ ISE community of researchers wishes.
Randi's slides didn't explicitly state,
"Proposals should measure individual
behaviors, attitudes, and engagement". Those
can be group-level or community-level behaviors,
attitudes, and engagement. The discrepancy seems
to be in what the ISE field has traditionally
evaluated (individuals' learning and
knowledge) vs. other possible impacts (group-level
ones). - Collective Efficay, Social Capital Two
types of innovation: practice-innovation:
knowledge advancement and implementation of things
such as curriculum design (educators), and
research innovation. Innovation will vary by the
user group...hard to get a community-wide
definition that applies to everybody. Somebody in
my group mentioned whether more or broader
proposals is a good outcome/impact measure or an
intermediary one. Could be both: more research
proposals can be a measure of the ISE
community's innovative output...but what good
is innovation for unless its impacts have been
defined and measured in those proposals? How can
online communities support different types of
innovation? by supporting different types of
innovator, so we need to identify what a good
innovator does (profiles of activities: advances
and disseminates new knowledge, dissemination very
important, develops relationships with other
colleagues, promotes learning and is able to learn
from the work of others) Those activities will be
more specific for each user type (eg good PI
innovator is able to learn and promote learning
from the research of others rather than
replicate).