An Evaluation of “The Learning Circles
Project” By Shawn Conwayshawnco@ralphthornton.org

Background

Many months before the Learning Circle Project was approved and funded
by the National Literacy Secretariat (N.L.S.), a group of literacy
practitioners began meeting in response to the Movement for Canadian
Literacy’s National Action Plan for Literacy. The group
was concerned about the proposal for a national adult education system
and what they felt was the potential for a focus on academic learning
in academic environments. The group called for “inclusive
lifelong learning” to be one of the plan’s goals and considered
piloting inclusive community learning centres. But after several
meetings the group realized they did not have enough information about
the variety of community learning situations that currently exist. They
decided that a more valuable project would be one that explores and
sheds light on inclusive community learning in its various forms. It
was in this context of open-ended exploration that the “Lifelong
Learning Working Group,” as the group called itself, was formed
and the Learning Circle project was conceived.

The Working Group consisted of highly skilled and seasoned literacy
practitioners who had observed the trends and fashions of literacy
policy for well over two decades. They defined themselves as
a learning circle, which meant, among other things, that they were
committed to an emergent, open-ended process of observation, analysis
and reflection. This exploratory group process mirrored what
they believed to be the kind of process that occurs in many other learning
circles and is, in fact, a defining characteristic of inclusive learning
circles. It was in this context that I was chosen as the Outside
Evaluator for the Learning Circles project.

Evaluation Process

Designing the Evaluation Framework

The initial project proposal submitted to the N.L.S. in late 2003
stated that the objectives of the project were:

To describe models of informal learning that have managed to include
participants who have difficulty using written language, or who might
not want to use written language to support their learning;

To explore some of these models in depth;

To identify best practices from the models;

To identify benefits to participants;

To identify benefits to communities;

To identify ways in which knowledge about informal, inclusive learning
can be shared;

To explore possible links between this kind of learning and adult
literacy programs;

To propose ways in which inclusive lifelong learning can be strengthened
across Canada;

To propose new approaches to lifelong learning.

As I prepared for my first meeting with the Working Group in the fall
of 2004, I considered these objectives and I wondered what the group
might define as the indicators of success and outcomes for each of
the objectives. For many years, in human services work and education,
nationally and internationally, the accepted wisdom has been that a
project is legitimate and meaningful to the degree that it has well-defined
and pre-defined indicators, benchmarks and outcome statements. For
example, in a project of this kind, we might create a list of indicators
that equate success with participants mastering job application forms
or reporting that they have more confidence in using written language
for their day-to-day tasks.

Soon into my first meeting with the Working Group it became quite
clear that the Group members had no intention of creating lists of
pre-defined indicators of success and tailoring their research in order
to look for certain details and experiences and not others. In
fact, the Group was ambivalent about the whole concept of “evaluation” laden
as it is, especially in the literacy field, with histories of testing,
benchmarks, employment preparedness, etc. much of which has been counterproductive
and sometimes destructive. At the same time the Group was very
committed to deep engagement with and analysis of learning circles,
how they work and how they are effective for their members and communities.

While the Group welcomed the idea of developing a process to keep
them focussed on their larger goals, they were opposed to a process
that restricted or predetermined opportunities for learning circles
and their participants to present or speak for themselves. As
one Group member put it, they wanted to “build indicators without
anticipating the end.” They wanted to keep their broad
destination in mind but not “filter” people’s experience
through predetermined criteria.

The Group also wanted to avoid coming up with a new formula for lifelong
learning that could become a new fetish for policy makers. That
is, the Group did not want their work to support a new policy directive
whereby learning circles are defined in a particular way and are promoted
as a solution to “literacy problems.”

If, as the original proposal stated, they wanted to “provide
a view of the possibilities for inclusive lifelong learning” that
learning circle models represent, then they would need an evaluation
process peculiar to their open-ended, exploratory research process. The
evaluation process would need to allow for, rather than circumscribe,
the organic, dynamic process of the Working Group as well as the open-ended
emergent lessons and recommendations that follow from the engagement
with the learning circles.

So, instead of a set of indicators and narrow outcomes we designed
an analytical framework that focussed on the three levels or areas
of outcomes in the original proposal, namely, the resource book, the
research process, and policy recommendations. The Group agreed
to address the following questions on a continuing basis:

Are we doing / have we done what we said we would do?

What would we have done differently?

What are we learning?

So what? That is, what has changed / will change / should
change because of what we have learned?

Now what? (What is left to do?)

What are the core factors that contribute to the learning of the
Working Group?

Although these guiding questions were uncomplicated, they were in
keeping with the Group’s adherence to rigorous and honest reflection
at each step in the research. Thus, while the evaluation framework
seemed deceptively simple, it provided a useful guide to a group as
strong and committed as the Working Group.

Interim Evaluation Meeting

The next stage in the evaluation process occurred after six months
of further research. During this period the four researchers
had been working with their respective learning circles and had been
meeting or communicating via the telephone or email regarding their
work.

In May of 2005, the whole Working Group and the Outside Evaluator
met to discuss progress to date. In a note from the Project Facilitator
prior to the meeting, it was stated that the meeting would be an opportunity
to “look at how far the narratives and analysis have moved us
toward a resource book and recommendations. Do we have what we
need to develop the resource book and recommendations? What additional
pieces do we need? At this point, what kinds of recommendations
do we see coming out of this project?”

By this time considerable work had been done by the researchers. The
Working Group members were very satisfied with the sixteen draft narratives
and the six draft analyses that had been written by the researchers. From
my vantage point as Outside Evaluator, I saw that the researchers had
already begun to articulate important lessons and suggestions regarding
the overarching themes of how learning circles work and what makes
them successful for their members and communities. There were
already specific and rich drafts of writing with titles such as “How
Do Things change Because of Learning Circles?”, “Inside
the Learning Circle: What Makes It Work?”, and “Literacy
and Inclusion."

It was also clear that the evaluation framework was providing the
Working Group, and the researchers more particularly, with what they
needed to stay focussed and to assess progress towards the project’s
goals.

My role at this meeting was to help facilitate the Group’s reflection
on how far along the work was, what was being learned and what else
needed to be done. In general, the Group members responded well
to the draft narratives and analyses and considered other questions
and issues that the research had spurred. For example, an interesting
process occurred whereby each level of learning circles provided opportunities
for reflection. In its work with the sixteen learning circles,
the researchers relied on each other and became a kind of learning
circle themselves. In turn, the larger Working Group was a learning
circle that reflected on and provided feedback about what the researchers
were learning. This multi-layered process of reflection was named “analysis-by-discussion” by
the Group and it encouraged a broad and exciting fabric of reflection,
a sort of proof that learning circles are good vehicles for lifelong
learning.

By the time of this meeting the researchers had also begun work on
a website and had started to summarize what they had learned about
learning circles. The lively and provocative “Beginners
Guide to Learning Circles” eventually became the “resource
book” envisioned in the original proposal.

The Widening the Circle Symposium

The Group’s recognition of the significance of continual reflection
and discussion led the Group to question the completeness of its observations. They
agreed that if they were to continue to emphasize an open-ended reflection
process, the project, and all those it touched, could benefit from
a symposium involving some of the larger group of learning circle participants. In
September, 2005 the “Widening the Circle” symposium was
held and seemingly represented a shift in direction. The Group
felt they needed to check their draft conclusions, to reflect and to
explore experiences and concepts with a broader circle of people and
specifically current learning circle participants. (See report
for more details.) In this sense, the symposium was less a shift
in direction and more a creative event directly in line with the principles
and questions guiding the Working Group.

To the Working Group’s credit, the symposium was a highly successful
addition to the whole project. Discussion at the symposium enriched
the draft ideas and conclusions of the researchers and Working Group
and provided further material to build into the overall analysis and,
in turn, the Guide, the final report and recommendations. As
an outside evaluator, I found the decision to hold a symposium an important
testimony not only to the integrity of the Working Group’s process
but also to the validity of the emergent, open-ended process of learning
circles themselves. The symposium also showed that the evaluation
framework was working well: the Group was continuing to ask open-ended
questions and to gather diverse and abundant material about learning
circles.

Feedback from Organizations

The last stage in the project’s evaluation was the solicitation
of feedback from literacy practitioners across the country. The
Working Group asked interested people and groups the following questions:

What are the implications of a learning circles approach to community
learning for adult literacy work?

What are the implications of this approach for community development
work?

What are the implications for a community that you are involved
with?

Do you have any thoughts about what kind of follow-up would be
useful?

Plus any other thoughts that occur to you.

Here again the questions are broad and open-ended and indicate the
researchers’ thoughtfulness in seeking to hear about others’ experiences
and knowledge without circumscribing people’s responses.

Project Outcomes

My final meeting with the Working Group in November, 2006 was dedicated
to considering the feedback from the organizations and what was achieved
overall in the project. This was also the time for a final reflection
on the evaluative framework as a departure from the standard form of
evaluation.

At this meeting the Working Group debated about some of the feedback
particularly whether the intent of the project had been made clear
and whether readers could see that the audience for the writing was
wide open. Some felt the project writings needed to be a little
more explicit regarding these points while others felt that it was
important for the project outcomes to throw up more questions and to
spur further reflection. The various documents are not intended
to “represent” all learning circles or to offer a strict
formula for them. Instead, the outcomes (the Guide, the report,
the narratives and analyses, and the recommendations) are meant to
represent a thoughtful exploration.

Perhaps more significant was the discussion by the Group about the
absence of Francophone voices in the project. While the Group
did deliver what it promised, namely the French translation of the
final report, the Group members recognized that it would have been
useful for Francophone voices to have been involved at various stages
of the project. It was noted that the Working Group had made
efforts to involve some Francophone groups but that insufficient resources
had been available to do more especially with respect to translation
and feedback.

Regarding the key desired outcomes of the project – how learning
circles work, of what value are they, and what can we learn from them
as vehicles for lifelong learning – the Learning Circles Project
has clearly delivered what it promised. In particular, the project
has provided the following:

A plain-language Beginner’s Guide to Learning Circles that
can be used as a conceptual introduction to learning circles. As
some symposium participants stated, the Guide also recognizes and
provides legitimacy for learning circles as an important model for
learning;

A rich collection of narratives that shows the range and variety
of learning circles in Aboriginal, rural, and urban settings;

A collection of highly insightful and unique analyses cum reflections
regarding the learning circles model. Taken together, the Guide,
the narratives and analytical writings make up an ample view of how
learning circles can work as models of informal learning, and some
of their benefits to participants and communities;

An example, in the form of the symposium, of how an open-ended
exploratory process can lead to a unique forum that engages learners
in a sort of meta-analysis (analysis-by-discussion) of a learning
process in which they are intimately involved;

An evaluation framework that resists pre-determined ends but that
serves as a rigorous guide in helping a group reach a project’s
destination;

A set of recommendations that tentatively suggest how learning
circles can contribute to the public good and how they might be supported;

A strong case for learning circles as effective and inclusive opportunities
for lifelong learning and for contributions to building social capital.

The Learning Circles Project has provided an abundance of ideas and
material and exceeded in some ways what it set out to do (symposium,
website, etc.) Although its conclusions and recommendations are
especially relevant to the literacy field they are arguably as relevant
to many other areas of civil society. In particular, the
concepts of open-ended reflection, emergent knowledge and continual
exploration of what is meaningful for the individual and the community
would be welcome additions in areas as diverse as environmental justice,
community health and community development.

In the field of health, for instance, “learning” rather
than behaviour modification could be tied more closely to determinants
of health and could itself become a determinant of health. So
too in the fields of community development and community economic development “learning” and
learning circles could take an important place beside the standard
language of “development,” “growth,” and “progress,” terms
which flow from the dominant values of the marketplace rather than
core community values.

Though beyond the scope of this project, how learning circles models
could be used elsewhere would constitute very valuable social research. In
the meantime, the many lessons and conclusions of the Learning Circles
Project will, it is to be hoped, encourage further work and action
towards placing lifelong learning, in all its varieties, closer to
the centre of how we live.