Main menu

Post navigation

I find it interesting that the two major evaluation conferences I attend are usually scheduled in spring and fall (respectively), the seasons of change1. At its heart, the field is about change: both in terms of understanding and cataloguing what has transpired as the result of an intervention or program, but also supporting new developments by providing feedback and ideas for further improvements and refinement. There’s one more reason why change is central to evaluation, though – our work is not immune to it either. We have to be aware that our contexts and those we partner with also face change regularly, requiring us evaluators to identify and understand what’s happening around us and respond in kind.

The two sessions I’m presenting at the upcoming Canadian Evaluation Society (CES) conference in Vancouver both touch on this idea of being aware of and responsive to our contexts2. Kirstie Gibson, a former placement student with Strong Roots Consulting, and I will be presenting on research that we conducted with local non-profits that examined their perspectives on evaluation. My other session, with Craig Moore of South Shore Evaluation in Nova Scotia, is a thematic breakfast roundtable that invites other consultants from smaller cities and regions in Canada (loosely defined) to chat about the benefits and challenges related to being “the only one in town” when it comes to evaluation consulting. Two pretty different topics, but I think there’s a common thread around the importance of communication and collaboration to successfully navigate situations that are much more complex than the safe examples we practice with in workshops or classroom settings.

With CES approaching in less than two weeks, I’m looking forward to the opportunity to connect with colleagues from across Canada and even other continents (I’m attending a workshop that includes presenters from New Zealand!). More importantly, these events always provide an opportunity to expand my thinking, identify new approaches and methods that I can incorporate into my practice, and most importantly, remind me that when change happens, there’s a community there to help me make sense of it. For my friends and colleagues who will be in attendance, I’ll see you soon in Vancouver! Everyone else, keep an eye on this blog and the Strong Roots Twitter account the week of April 30 as I share insights and ideas from the conference.

Yes, I understand that there are only so many months in the year to hold a conference, that traveling during the winter sucks and many people would rather be on vacation in summer than sitting in a convention centre somewhere – it’s a metaphor, just let me roll with it! ↩

I’d like to pretend that this was intentional – perhaps subconsciously? ↩

It’s most definitely fall in Saskatoon – nights dip below freezing, the trees have shed most of their leaves, and it’s only a matter of time before the snow hits1. Another sign of fall? Upcoming due dates to submit presentations for conferences, including the Canadian Evaluation Society’s 2017 conference in Vancouver.

Earlier this year, I wrote about the conference and its theme of Facing Forward: Innovation, Action, and Reflection. Today, I’d like to focus on Innovation – but instead of looking at innovation in evaluation, let’s talk innovation in conferences. My experiences at evaluation conferences (both in Canada and south of the border) have been overwhelmingly positive: at the same time, I’m aware that these events can be criticized for not being the most effective use of time and resources. Indeed, there are numerous ways today that we can learn from others without having to leave the comfort of home – including blogs such as this one! As a result, it’s a legitimate question to ask whether it’s worth spending hundreds of dollars (if not thousands, when you account for travel and accommodations) to make the trip.

One of the key benefits I have taken away from conferences is the opportunity to connect with others walking a similar journey. These events give me the chance not just to learn from others, but to share my own experiences and have a real dialogue where ideas can arise from the space between. True, we can have this two-way communication through words or video, but there’s something about being in the same room together or perhaps pitching ideas over a meal (or pint) that contributes to the generation of innovative ideas.

Although most of these connections are informal, the conference organizers can set things up to encourage such dialogue. For CES 2017, the team behind the conference have provided some interesting presentation types that I’m excited to both see in person and also participate in as a presenter. One type in particular, called “Consultation and Collaboration”, is a highly interactive format that asks attendees to work in groups in response to a problem, issue, or topic brought forward by the session organizers. Giselle Patrick, Carolyn Camman, and I have submitted a proposal under this format on the topic of evaluation capacity building (ECB), with a focus on how ECB can be conceptualized and put into practice for different types of organizations. We look forward to bringing some unique engagement activities to our session and seeing what comes out of the conversation!

There are several other interesting presentation types this year, including Storytelling, Thematic Breakfast Roundtables (I’m thinking to host one on consulting in small and mid-sized cities), and Lightning Round Tables that sound like a mix of speed networking and information sharing. And of course, there’s my perennial favourite, Ignite presentations, where your slide deck automatically advances through 20 slides, 15 seconds a slide, for a total of 5 minutes. I’ve found it a fun challenge to craft a coherent presentation to fit within these limitations, and likewise have enjoyed seeing what others do with this format!

One activity that I’d love to see at CES conferences is based on an idea from our American colleagues, specifically the Community Psychology Topic Interest Group (CP TIG) of the American Evaluation Association (AEA)2. At the annual AEA conference, this TIG hosts an event called “Walk the Talk”, where conference attendees can sign up for a tour of a local community agency or group to learn more about their work, see how they use evaluation, and provide ideas and suggestions for the agency to move forward in accomplishing their mission. Vancouver, like communities across this country, is home to numerous agencies of all stripes and sizes doing amazing work, and I think it would be worthwhile to go beyond the conference venue to learn more and contribute back to these organizations.

So suffice to say, I’m looking forward to this conference, both as a participant and contributor – the hard part is deciding what ideas I want to present on! Fortunately, I still have another week and a half to finalize and submit my proposal(s). Over to you now: What are you planning to present on, if you’re going? What are some innovative presentation formats that you’ve seen at other conferences? Share in the comments below or on Twitter, or drop me a line!

That is, until the snow hits again: Saskatoon saw an extended preview of winter at the beginning of October. ↩

Tomorrow, Apple will lift the veil on its newest product, most likely a new iPhone1. If past behaviour is any indicator, there will be much discussion and many questions asked in the immediate aftermath: is this a good product, should people buy it, is Apple being forward-thinking or foolish in removing the headphone jack (if the rumours are to believed), and so on. One topic that likely won’t come up, though, is what the company will do with the profits it realizes from the new phone and its other products.

What we don’t ask of Apple or other for-profit companies stands in notable contrast to the questions and restrictions imposed on non-profit organizations when they receive funding. I won’t belabour the general point here, as others have written amazing pieces dismantling these harmful ideas: as one example, Vu Le provided a hilarious-because-it’s-absurdly-true example of trying to run a bakery if we brought to bear the overhead and sustainability myths that non-profits routinely face. More specific to my focus on evaluation and research, Ann Rosenfield (hat tip to Joe Travers) pointed out that donors should actually want to pay for a charity to collect and analyze data about their programs, spend time understanding local trends, and develop and test new approaches to fulfilling their mission. Although these dollars may not go directly towards the front-line program or service, they ensure that the organization continues to operate effectively and is prepared for what the future will bring.

When buying any kind of gadget from a for-profit company, it’s understood that the amount we spend will be used for more than just the product itself. In fact, I think we would be surprised and maybe even angry if companies like Apple didn’t spend at least some money on improving their products and developing new ideas. In the non-profit world, our R&D work (which includes evaluation for me) is often seen as an afterthought, not just by donors and funders but by those working in the field. Sure, we may never see our work featured and feted like a new technological marvel: instead, though, we can develop new ways of creating positive change that save lives, dismantle systems of oppression, and generally improve quality of life for individuals and communities around the world. To reach that potential, though, our sector as a whole needs to invest the time and resources necessary to improve what we’re already doing, understand what’s coming over the horizon, and develop our own “next best thing”.

And possible a new Apple Watch, depending on which prognosticators you want to believe. ↩

From previous blog posts (see here, here, here, and here), you can tell that I’m a fan of expanding my skills and learning in evaluation through conferences. Attending these events, hosted by the Canadian Evaluation Society and the American Evaluation Association, has also connected me into a diverse community of practitioners, academics, consultants, students, grantmakers, government officials, and more who are brought together by a common interest in the field of evaluation. Given this blend of learning, sharing, and connecting, I try to make it out to these conferences whenever circumstances allow.

The next Canadian Evaluation Society conference will be taking place in Vancouver from April 30 to May 3, and I’m excited for a couple of reasons – not least because Vancouver is a beautiful city to visit, relatively close1 to my home base of Saskatoon, and home to several evaluation colleagues and friends. What really piqued my interest is an initiative for the conference that’s being led by one of those colleagues and friends, Sarah Farina of Broadleaf Consulting. Evaluators are being encouraged to blog on the conference theme (Facing Forward: Innovation, Action and Reflection) in the months leading up to the conference. It’s a neat way to start the conversation well in advance of the meeting itself, and hopefully have it continue afterwards!

Evaluation’s Present and Future

Over the next 8 months, I’ll be delving more deeply into each component of the conference theme: today’s post will take more of an overarching view.

Innovation, Action, Reflection. My first thought on seeing these three words is that I wish that the organizing group hadn’t chosen this theme, as I would be tempted to use it as a tagline for my business! Unpacking the ideas a bit further, I think it reflects where evaluation is today and our current trajectory. Our field has had the reputation (with some historical basis) of acting as arbiters of a program’s fate by rendering simplistic thumbs-up/thumbs-down judgements, and thus raining on the parades of those who are trying to make a difference in the world. While evaluation continues to play a role in promoting accountability, to say that’s our only focus does the field a great disservice.

Based on my experience, the core element of evaluation is using applied social research skills to help those seeking to create change learn more about the work they do and how they can improve their efforts. Evaluators help organizations understand their successes (and failures), provide feedback and suggestions for change, and work alongside developing projects or initiatives to provide real-time data and insights. Yes, there will always be times where we have to pass judgement and say that this program did or did not work as intended to achieve its desired outcomes: that being said, we have both the opportunity and the responsibility to ask how those assessments can support action, encourage reflection, and ultimately promote innovation to tackle the complex issues we see affecting our communities and the world.

As noted above, my plan is to blog about each of those three aspects (Innovation, Action, Reflection) over the next several months leading up to the conference. For more information about the conference, check out http://c2017.evaluationcanada.ca.

Professional Development on the Prairies

This summer will mark four years for me in Saskatoon, and while there are many things I love about this city1, there are a few small challenges. One specifically related to my work is the relative isolation: the nearest city of comparable size is the provincial capital of Regina, which is a two hour drive away: for larger centres, it’s five hours to Edmonton, seven to Calgary, and eight to Winnipeg. There have been many times where I’ve learned about interesting professional development opportunities in those places or further afield: unfortunately, the time and cost associated with travel can be huge barriers to attending. Although technology has helped fill the gap somewhat, such as through webinars, those methods also have their downsides. Interaction with other participants and the presenters tends to be minimal or non-existent, and it’s all too easy to skip a webinar or attempt to multi-task during it (hint: that strategy doesn’t work, at least not for me!).

Yesterday, I learned about an interesting professional development model that could provide the best of both worlds. Shelley Borys and Natalie Kishchuk presented a webinar hosted by the Canadian Evaluation Society on the concept of learning circles. In their use of this model, a small group of participants (5-7 typically) would meet regularly through one-hour teleconferences: through a cycle of six such sessions, they would discuss different evaluation topics and methods, chosen based on the interest of the group members. Each session would have one of the members act as leader who would be responsible for facilitating the discussion and also selecting the materials and resources for everyone to review in advance. Although the group used teleconference as the means for conversation, online tools such as Dropbox and a shared spreadsheet helped disseminate resources and select discussion topics, respectively.

Evaluate Thyself

Beyond finding this idea interesting, there’s another reason why I’m sharing this example. As you’d expect from a group of evaluators, they evaluated how well this method worked as an educational tool through a brief survey of participants. The outcomes were positive, though with an unexpected result when they delved a bit deeper into the findings. As one would reasonably assume, participants brought different levels of knowledge on different topics: one person may have a high learning need for, say, developmental evaluation, but would only need a refresher on empowerment approaches. However, some participants would initially rate their knowledge on a certain topic as low, but after participating in the discussion would realize they knew more on the subject than they thought. In these cases, dubbed “Comforted” learners by the evaluators, participants would gain confidence in the knowledge that they already possessed (usually from their own personal experience), and also gain support and validation in their skills from the other group members.

These and other findings, such as the importance of balancing similarity and diversity among the group members, highlights the importance of looking beyond surface-level question such as “Did the program work?” or “Were participants satisfied?”. The knowledge gained from examining the reasons for success and underlying processes could help identify who would most benefit from the program: for example, the finding around Comforted learners could indicate that such circles could support early- to mid-career evaluators who have developed a skill base but may be uncertain in their own knowledge. Alternatively, if a group doesn’t work out, the evaluation findings could help determine why: perhaps the composition was too uniform and thus lacked diversity of knowledge and opinion, or the trust among group members wasn’t sufficient to allow people to admit uncertainty or lack of knowledge. Rather than throw out the learning circles model with the bath water, changes could be made to improve the odds of success for the future.

Let’s Start!

Based on what was shared in the webinar, I’m really excited about the prospect of being involved in a similar group. I already belong to a small online group of Canadian evaluators, and I think this approach could be a promising one to introduce some regular discussion. A learning circle approach held in-person could potentially be fruitful here in Saskatoon, though perhaps on a broader topic such as community-based research. If you’re interested in working with me to start up a learning circle in either (or both) directions, or would like to share your experiences with this type of mutual learning, feel free to share in the comment section below, drop me a line through the Contact page or find me on Twitter.

Especially now that we’re fully into spring and patio season is just around the corner! ↩

Student practicums or placements can be a great opportunity for non-profit organizations to provide a student with real-world experience while benefiting from their knowledge and skills. Most placements are required for the completion of a graduate or professional degree; however, some upper-year undergraduate courses include this form of experimental learning as well. Depending on the nature of the placement and the course or degree requirement, the student will be required to complete a set number of hours for the organization, although they may also need to complete specific project or reporting requirements in order to receive credit.

Here in Saskatoon, we’re fortunate to have multiple opportunities to engage with students through programs at the University of Saskatchewan and Sask Polytech, as well as students living here who are enrolled in programs at the University of Regina (notably social work students) and other institutions. Working with local non-profits, it’s pretty rare to encounter one that does not have previous experience with a placement student: in particular, I don’t know if I have worked with an organization that doesn’t have a report penned by a Masters of Public Health student somewhere in their file drawers!

Today I wanted to quickly draw attention to two placement programs that may fly under the radar, plus a learning opportunity that I'll be offering through Strong Roots starting tomorrow!

Applied Social Psychology

As the name suggests, the Applied Social Psychology focuses on the use of social psychology research methods to address social issues1. Part of their coursework includes learning about evaluation approaches and techniques. In fact, the program regularly participates in the Canadian Evaluation Society’s Student Evaluation Case Competition, with the university’s teams winning in 2004, 2011, and 2015, and being selected as finalists in 2002 and 2010. Clearly, they know something about evaluation!

Graduate students in both the Masters and PhD programs are required to complete two placements, one over the summer and one during the school year, with external organizations. I’ve been fortunate to have Kirstian Gibson, who’s in her second year of the Masters, working with me since the fall. In the past several months, she has taken lead on a pro bono evaluation project with a local non-profit, provided support on some ongoing client project, and also co-initiated a research project on how non-profit staff and leaders in Saskatoon view evaluation. It’s been great working with a student who has a strong grasp of evaluation and social research methods (she was part of the winning case competition team in 2015), and I look forward to working with another student from the same program this coming summer.

For more information about the placement, the best person to contact is Dr. Karen Lawson.

Governance Leadership Development Practicum

The for-profit and non-profit worlds don't always see eye-to-eye, but in the case of the Governance Leadership Development Practicum at U of S' Edwards School of Business, it's a great match. This fourth year undergraduate course combines an intense seminar focusing on non-profit governance issues with a practicum experience that places students with local non-profit boards over the course of two semester. I've been involved with this program for the last two years now as a non-profit board mentor, and it's been a great experience with both students. The course provides them with a wealth of knowledge about board governance issues, and the students’ business backgrounds have provided new ideas and insights for the board moving forward.

For more information about this opportunity (applications for the fall should be opening later this spring), contact Dr. Chelsea Willness.

Numbers and Stories office hours

On the topic of learning and sharing skills, I’m happy to announce that Strong Roots Consulting is partnering with The Stand Community Organizing Centre here in Saskatoon to host a regular ”Numbers and Stories" office hour! Any non-profit is welcome to visit me at The Stand (615 Main Street, just east of Broadway: same building as Turning the Tide bookstore) on Wednesdays from 1-3pm for free advice and assistance on topics related to evaluation and community-based research. Some of the help I can provide includes giving feedback on a survey, deciphering what a grant application is asking for in the evaluation section, or brainstorming ideas on how to best engage with your organization’s members.

The first office hour happens tomorrow afternoon – hope to see you there!

These three opportunities are just scratching the surface for different initiatives that support non-profits. Share your experiences and recommendations for placement programs or other educational opportunities in the comments field below, or continue the conversation on Twitter!

Right now, I have a bunch of ideas floating in my writing list from this conference. New tools, new insights, new ideas for programs or approaches that could be useful back home in Saskatoon. I think I could write up one of those topics each week between now and the end of the year and perhaps get through them all, though I still have a half-day left to add to that list.

I’m not going into any of those takeaways right now, because one thing keeps coming up in my mind: the importance of connections at this conference. Over the past few days, the laws of probability have been strained around how often I’d run into colleagues and friends, including one person who (we managed to figure out) I had met at a conference 8 years ago. I’ve had conversations with people from around the world and even ran into a few non-Canadians who know where Saskatoon is. I’ve had the opportunity to learn from and be inspired by other independent consultants, and hopefully provide them with some ideas in return. And last but not least, I’ve had the opportunity to laugh and share some inside evaluator humour, including through the magic of Chris Lysy’s cartooning.

Given that this is my third such conference, I should no longer be surprised that Eval 2015 is an amazing, overwhelming, encouraging, and ultimately empowering event! Both of my workshops were great learning opportunities leaving me with lots of ideas to bring back home to Saskatoon, I’ve run into several friends and colleagues already, and I managed to push past the butterflies in my stomach to successfully deliver my Ignite presentation. Seriously, give that format (or its cousin, Pecha Kucha) a try sometime – a set number of slides that change automatically, it’s a rush to present!

The theme of this year’s conference is Exemplary Evaluations in a Multicultural World, and if the number of flags on stage at the opening ceremony is any indication, it’s definitely a global audience. I was proud, though perhaps not too surprised, to see that Canadians make up the largest non-American contingent at 139. And yet we don’t have our own badge ribbon? Next conference I should bring some maple leaf stickers.

I was heartened to see the opening ceremony beginning with a traditional welcome from a local Indigenous group. There’s a growing recognition in Canada and apparently the US as well of at least acknowledging the history of the land and people where we are meeting, and I hope this becomes the norm for events generally.

If there’s one take away from my experiences here thus far, it’s that we have the opportunity and the call to up our game as evaluators. Our field is gaining international recognition and legitimacy (2015 is the UN-recognized International Year of Evaluation), our skill sets are increasingly viewed as important and even necessary in creating sustainable social change, and at the same time we’re recognizing that we’re working in increasingly complicated and even complex situations. Our toolkit is growing to incorporate new ways of presenting data, engaging stakeholders, and demonstrating impact. We’re called to be exemplary.

Yes, it can be overwhelming and even a bit anxiety-provoking – who am I to take on these tasks and responsibilities? At the same time, it’s exhilarating to be here at this place and time, and to have this opportunity to learn and grow. I look forward to the rest of the conference, and more importantly, what I can and will do in the weeks and months to come to create a better world.

Greetings from Chicago! I arrived in town last night to attend the American Evaluation Association’s annual conference, the third such time I’ve attended this massive gathering of evaluators. Taking a quick look through the program, I’m still excited and overwhelmed by the sheer number of interesting sessions and workshops: even after three years, it’s difficult to choose what to take in without burning out!
This year, I’m also presenting in two sessions, one an Ignite talk on my professional path to evaluation, and the other one a more traditional presentation on development evaluation. I’m looking forward to the next few days, but between those talks, two workshops that I’m participating in, networking with colleagues old and new, and of course some time to explore this beautiful city, that time is going to pass by fast.

If you’re at the conference, feel free to drop me a line – I’m always up for a conversation! The best way to reach me the next few days is by Twitter or by email. Note that I am not reachable at the phone number on my contact page between now and the 15th: if you need to reach me by phone, contact me by email and I can send you my temporary US number.

As with past conferences, I will be tweeting regularly and also blogging on a daily basis if possible.

Good news for non-profits: there’s a ton of information and data out there to help you understand your community and the people who live in it. Bad news for non-profits: there’s a ton of information and data out there. It can be hard to find what you’re looking for across multiple sites and agencies, especially if you aren’t certain who hosts which indicators. Even if you know where to look, it often takes a lot of time to gather that information and convert it into a useful format.

Given these challenges, I was happy to learn about Community View, a catalogue of data for Saskatoon and area. Developed by Saskatoon Health Region and supported by several local partners including the University of Saskatchewan, it’s an online tool that provides free access to a host of community-level indicators.

Last week I attended a workshop on using the tool, and a couple things stood out for me:

There’s a great range of reliable data sources – mostly census1 but also other sources such as from the Health Region and the school boards

The “data cart” feature sounds like an odd idea at first, but once you start using it to select indicators, it makes perfect sense.

Data can be easily filtered, including down to the neighbourhood level in Saskatoon for most stats. Filters can be applied to individual data sources or to the whole cart of indicators2.

Mapping tools are included, with options to show data (again on the neighbourhood level for most indicators) as either colour gradients or sized circles.

Data can be viewed in tabular form in the web browser as well as exported to Excel.

For the stats nerds among us, there are advanced features to play around with including setting different cutoffs for colour gradients on the map, creating new variables based on mathematical expressions (such as percentages), and uploading your own geocoded data to be mapped.

To bolster the amount of information available, documents such as written reports and articles are also included. Fortunately, it is easy to filter these documents out from listings and search results if you’re just looking for data sets.

Overall, Community View is a great resource for Saskatoon-based community researchers. For someone new to the world of community-level data and the online platforms that house them, it may be a bit intimidating at first: efforts such as the workshop I attended were clearly designed to alleviate the learning curve. Otherwise, I just wish that I had known about this tool sooner, and as one of the workshop attendees commented, that it was available for data across Canada!

What do you think about this tool? Are there similar websites out there that similarly bring together multiple datasets for community researchers? Share in the comments below, through Twitter, or the usual contact means.