What does it take to create a program that matters to youth? We consider the issues from the theoretical to the practical through our work in Project Coach. PC is a Smith College program that teaches teenagers to be youth sport coaches. As a coach, our teenagers must inspire, communicate, problem solve, resolve conflict, plan strategically, and deploy a range of emotional intelligences.
PC uses sports as a means to engage, connect, and empower adolescents.

Tuesday, June 18, 2013

One of the greatest challenges facing youth development programs is for
them to demonstrate that the activities they offer are really having a
meaningful impact on the kids that they serve. From a strict experimental
design perspective, an assessment strategy for determining program impact would
seem fairly straightforward. All an evaluator would need to do is to randomly
select a group of participants and controls, engage one group in an array of
program activities, while having the other engage in its “normal” activities, and
after a fixed time period compare the groups on selected youth development
measures such as emotional regulation, attentional control, perseverance, empathy,
intrinsic motivation, and various character traits. If the data showed randomly
selected program participants to have an advantage over their randomly selected
non-participating peers, controlling as much as possible for other factors,
then evidence would exist that the program and its activities are having their
intended effect.

Unfortunately, the “roll up your sleeves” reality of youth
development programs rarely affords the opportunity of random assignment of
kids to participation/non-participation groups, and, even if this were possible,
it is really impossible to know whether pre-post changes in developmental
measures are solely a function of the activities that kids in a program engaged
in and the controls did not. The question of whether some other factors
affected the two groups differently can always be posed. Programs typically run
over the course of several months, and to be certain that comparative groups
had identical histories, except for exposure or non-exposure to program
activities is a difficult assumption to make, but one that would be necessary
for a randomized control group design to be tenable. While randomization and
control of group histories are two requirements for doing high-level
experimental research, we think that most programs wishing to do meaningful
applied research need to find alternative strategies for assessing their
effectiveness.

During the past decade, Project Coach, like many other youth
development programs, has been doing pre-post assessments on our participants.
One of the assessment tools that we use is the Developmental Asset Profile
(DAP), developed by the Search Institute. It is designed to measure 20 internal
assets (e.g., responsibility, caring, integrity, interpersonal competence,
self-esteem) and 20 external assets (e.g., adult relationships/role models,
community values, service to others, engagement in creative activities). These
assets can then be combined in various ways to show what the Search Institute
labels as five contexts views (e.g., Community, School, Family, Social, and
Personal) or asset views with four being internal (e.g. Positive Identity,
Social Competency, Positive Values, Commitment to Learning), and four being
external (e.g., Constructive Use of Time, Boundaries/Expectations, Empowerment,
Support). Without getting bogged down in the specific meaning of these measures
and how they can be aggregated, data shows that the more assets kids possess
the better that they do in school, and the less likely they are found to engage
in health compromising behaviors such as alcohol and drug use. Clearly, an
objective of programs should be to help youth develop assets that build
capabilities that make them strong, successful, and resilient people.

The figure below shows pre-post data for one of our adolescent
coaches. Just looking at the context bars (i.e., the top 5), one can see that the
youth scored higher on 4 of 5 contexts in June than she did in September. As
well, from an asset perspective, the adolescent was higher on 6 of 8 of the
factors (lower 8 bars) in June. From the casual observers perspective this
might seem like fairly positive support for Project Coach in that our program
hypothesizes that participants acquire many critical assets as they acquire
leadership skills as coaches of young children.

However, the experimentalist could easily shoot holes in such an
interpretation as alternative explanations for the pre-post differences could
be proposed. Suppose all youth, whether or not they participated in Project
Coach, might have had such positive change as a result of simple maturation. Or,
perhaps, youth became sensitized to simply taking the DAP, and were better at
responding to questions in June than in September, as they became more savvy
about presenting themselves more favorably. Many other explanations for
observed differences could also be proposed. Without randomization and strict
control over the lives of youth between September and June, any and all
factors, including Project Coach activities, could be responsible for observed changes.

Given that such data can be misleading and that strict experimental
design and control is difficult to operationalize in real world settings, what
are programs to do to show the value of their activities? While no perfect
solution exists, one strategy might be to collect data showing what kids are
doing in their daily day-to-day lives between September and June. If it can be
shown that when they are involved with program activities they are engaged in
more activities that promote the sorts of outcomes measured by the DAP than
when they are not engaged in the program, then one could start to make the case
that pre-post changes are real and naturally result from program experiences.
Other twists to this might be to also contrast kids in a program with
acquaintances who are not involved in the program and determine whether their
day-to-day experiences are qualitatively different with regard to building developmental
assets.

Project Coach has actually been experimenting with such a technique
known as the Experience Sampling Method (ESM). The idea here is to probe youth
at random times and to ask them a variety of questions related to what they are
doing, who they are with, and how they are feeling. Using the ESM, one of our
staff, Katlin Okamoto, sent text messages to Project Coach youth between 3 and
6 p.m., the after school hours. These texts were sent on the three days a week
kids were working as coaches, and on the other two days when they would be
“off” and engaged in non-structured, non-Project Coach activities. When they
received a text from Katlin, they filled out a brief questionnaire that was
based on Lerner’s model of the 5 Cs - Character,
Competence, Confidence, Connection, and Caring, which overlaps with many of the
assets found on the DAP. Her results are shown in the following figure. As one
can see, the findings are fairly dramatic with 4 out of the 5 categories (* p
< .05) showing kids more deeply engaged in developmental types of activities
when they are working in Project Coach than when they are just on their own.

Katlin also recruited the friends of Project Coach adolescents who were not in the program, and also probed them to see how they compared with the Project Coach kids. The results of this analysis are shown in the following graph. Again, we can see the impact of being engaged in program activities, which are represented by the blue bars. As well, one can even observe differences between Project Coach kids when they are not involved with Project Coach and their friends. While only tentative, differences are consistent with an hypothesis that suggests that Project Coach kids are involved in more developmental activities even in their own free time when contrasted with non-Project Coach peers. This might be a transfer effect that, although weak, shows adolescents doing a better job of structuring their free time than peers who are not exposed to a program such as Project Coach.

While, we do not claim that such data are as powerful as those that
might be acquired by using a randomized-control group design, they do help to
build a case that the pre-post DAP chart is not just an artifact attributable
to factors unrelated to Project Coach activities. With ESM data the case can be made that if
kids are differentially engaged in developmental activities on a day to day
basis in program activities vs. non-program activities, then over the course of
ten months there should be changes on scales, such as the DAP, that probe the
development of such assets. The in-Project Coach -- not-in-Project Coach contrasts
also point in this direction.

While perfect empirical support for youth development programs is a
worthy and highly sort after goal, we believe that it is unattainable. On the
other hand, “less perfect” data, as portrayed in this blog entry, is critical
to acquire, as it can help “connect the dots” between non-experimental designs
that utilize pre-post evaluations. As we argue, if kids are engaged in
developmental activities on a regular basis, it is logical that they will
acquired the sorts of developmental assets contained in the DAP and 5 C’s.

Monday, June 17, 2013

It's sad and exciting to be on my way out. I had a great experience in Springfield and Northampton. I recently accepted a job to teach English and Economics at Avon Old Farms in Connecticut. The key skills I've honed as a red shirt are relationship-building, patience, and persistence. Each of those traits will serve me well when I am running my own classroom. It was also eye-opening to be a part of a small non-profit organization and to see the complex inner workings that go into our programming. I'm fascinated by business and youth development, and Project Coach toes that line every day.

I wish we had more time with our blue shirts, but I'm sure there is a talented crop of red shirts to pick up where we left off. I'll do my best to check in with Kyle, Madison, Joanuel, and Gabe to make sure they're still working toward their goals. My advice to people involved in Project Coach going forward is use your unique talents to add value to the program. Project Coach is growing and dynamic, it is possible to significantly impact curriculum, fundraising, and networking efforts if you work hard. If you go through the motions you won't get as much out of the experience. All the best!

Monday, June 3, 2013

On Friday, May 31st leadership staff from Project Coach traveled to see the Boys' Club of NYC operating their Project Coach program. As part of an existing partnership with BCNY, Project Coach runs a training camp for new coaches over the summer and then travels to NYC to visit the program, see the coaches in action, provide feedback, and meet with the directors. Veteran coaches from Project Coach who are now in college and have leadership roles in our organization travel with us to see Project Coach in another setting and help coaches from BCNY improve. On this particular visit, we went to the Gerry Clubhouse in Harlem and observed two floor hockey games.

Our visit started with a great welcoming from Manny Maldonado, physical director of the clubhouse, and three BCNY Project Coach teens. The teen coaches gave us a wonderful tour of the facilities, explained their experiences coaching and how it impacted them, and discussed their great experiences with the BCNY. Along the way we also met a veteran coach who was about to graduate high school and head to college in North Carolina. He spoke eloquently about choosing his college and major, being excited to go far away, and his love for coaching and the Boys' Club.

Let the game begin!

Following our tour and meet and greet, we went to watch the two floor hockey games. Immediately upon entering the gym, the energy from coaches and players was apparent. Coaches began with a huddle to prepare their team and set the tone. The coaches did a great job focusing on teamwork, passing, and hard work. Once the game started, players were fully engaged, running around, and enjoying this excellent physical activity. Goals were scored, passes made, and red faces flourished quickly. Coaches yelled out positive feedback and encouragement to their players. At the end of each period, teams huddled up and coaches gave both group and individual feedback. At each break, coaches made sure all players drank water as the gym was particularly warm. At the end of the game, teams shook hands, debriefed the game, and went into the locker rooms.

Coaches run a time out!

During our visit, PC staff reflected on the Project Coach model, how easy it is to replicate in new places, and how exciting it is for younger children to be coached by teens. Many of the coaches, players and staff we spoke to explained how having PC at BCNY gave the teens more of a reason to stay connected to the club and the younger children really looked up to their coaches. We watched teen coaches utilizing behavior management skills, coaching voice, and elements of positive youth coaching to create an exciting and fun atmosphere for children. The opportunity that Project Coach provides both teens and young players is truly incredible and we could not be happier to spread this opportunity to new locations.