Hi, we are James W. Altschuld, Hsin-Ling (Sonya) Hung, and Yi-Fang Lee from The Ohio State University, Virginia Commonwealth University, and National Taiwan Normal University, and have presented on, written about, and been involved in NAs for many years. In reviewing manuscripts for publication, we have observed measurement and analysis issues in double scaled surveys used for identifying discrepancies between what should be (importance) and what is (current) conditions. They include wording, scaling choices, organization, misleading conclusions and interpretations, and others.

If the survey is not sound what comes from it will be the fruit of the poisonous tree. By offering ideas for improving NA surveys some (not all) of the weaknesses can be attended to and the quality of needs work will be enhanced.

Hot Tips:

Read sources about the design and especially strengths and weaknesses in NA surveys (some sources do both, see Rad Resources).

Search for sample NA surveys in the area of concern, examine/critique for strengths and weaknesses as noted in point 1.

See if other techniques have been used with the surveys. (Having multiple sources of information is good practice.)

Conduct pre-interviews (focus group, individual) with a few respondents regarding their thoughts and the language they use for the area of concern. (Will make the instrument more meaningful.)

Cluster items into sections and consider having respondents rank them after completing the survey. (Not all clusters will be equal of value.)

Employ options like don’t know (DN), no information (NI) upon which to decide, not applicable (NA), etc.

Forcing choices without such options, as in point 6, may produce misleading data and additionally the options provide useful information.

Have an undecided (neutral) response on the scale. (Similar rationale to point 6.)

Consider alternatives such as magnitude estimation scaling (MES), fuzzy scales, rank ordering approaches, etc. (Let’s be expansive and innovative in what we do.)

‘There are 95 rules of survey design and after the 95th there are 95 more you don’t know about.’ This applies doubly to double scaled NA surveys and to illustrate the assertion, note that we haven’t touched the gnarly topic of “how to word” items.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Welcome to the Needs Assessment TIG’s week on AEA 365! I’m Lisle Hites, Chair of the Needs Assessment TIG, Associate Professor in Health Care Organization and Policy and Director of the Evaluation and Assessment Unit (EAU) at the University of Alabama at Birmingham (UAB) School of Public Health. On behalf of all of us here at the Needs Assessment TIG, we hope you enjoy this week’s blog entries and to see you at some of our sessions at AEA 2018. Today’s posting is about the complexity of assessing needs in communities and offers a couple of lessons learned that might be of use.

For those of you who have ever tried to assess the needs of a community, especially those communities that are disadvantaged in terms of health, socioeconomic status (SES), etc., with so many apparent needs, researchers, unfortunately, often feel comfortable addressing whatever they want to while couching it as a priority need for the community. I am working on such an effort, here in Birmingham Alabama. Birmingham has 99 distinct neighborhoods, most, if not all of which, are medically underserved and between low and ultra-low on the SES spectrum. Past needs assessments have uncovered a multitude of priorities that vary quite a bit between these tightly grouped urban areas. We have high levels of urban blight in some as indicated in part by abandoned houses, but very few in others. The more a neighborhood is blighted, the fewer resources residents want to put into it, deepening the blight. Many problems contribute to or result from this situation, most of which are identified needs in the eyes of the community and researchers. How then do we decide where and how to intervene?

Working from identified needs, we are involving the communities of interest in planning stages for an intervention. Even so this has resulted in little or no change in the community and they do not trust that this will be any different from previous efforts. Here’s where asset mapping can save the day. By simultaneously assessing needs and mapping assets, we are positioned to identify what is most needed and match that with where we can best impact that need and with what resources. Community, city, county, and even regional resources already exist. By systematically identifying what those resources are, where they are willing to go, what fits within their mission and scope, we can now better select where we can intervene with the most impact, leveraging existing resources to maximize outcomes.

Lessons Learned:

Communities are complex organisms and propinquity does not necessarily mean generalizability of needs assessed.

Community-focused Needs Assessments are a difficult sell without a path forward to improvement.

Asset mapping can complement needs assessment to help form a continuous improvement cycle, each providing input for the next iteration of the other.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I hope you are all enjoying the Translational Research Evaluation TIG week on AEA 365! I’m Lisle Hites, incoming Program Chair of the TRE TIG and also Chair of the Needs Assessment TIG. In my spare time, I’m also an Associate Professor in the School of Public Health and Director of Evaluation for the Centers for Disease Control Prevention Research Center (PRC) and the National Institutes of Health Center for Clinical and Translational Science (CCTS) at the University of Alabama at Birmingham (UAB). Today’s posting is about the use of needs assessment (NA) in supporting translational science.

After more than 17 years of working with researchers, organizations, and communities to assess needs, with 12 of those focusing on translational science with the PRC and Clinical & Translational Science Award (CTSA), it has become clear that there is really no one technique that best suits every situation. As an example, I’m going to draw on a needs assessment my team conducted several years ago to align research investigators with the needs of the communities from which they were drawing their study participants. This was a multi-step process that began with a focus group consisting of a mixed group of community members, interested academic researchers, and guided by the CCTS’ community engagement arm, we call One Great Community. This organization works in tandem with our PRC. We gathered community health concerns, then developed them into a survey based NA protocol that was then utilized to collect data from each of the 99 neighborhoods within Birmingham that surround UAB. The NA utilized scaled response options to collect this neighborhood-level data, allowing us to use the scales to determine community prioritization of their self-reported concerns for each of these health factors. The CCTS then took the most highly prioritized needs and designed a partner funding opportunity that supported community/academic investigator pairs to propose community-driven research pilot projects that addressed these identified top priorities.

We are now in our 5th year of offering these Community Health Innovation Awards and the results have been outstanding, thanks to assessing the community’s needs at the start and asking them what they are most concerned about. Truly, the number of ways to target and conduct NA are nearly infinite. By starting with assessing needs, researchers have the opportunity to gain the support of the communities in which they research and recruit subjects and to use this information to better inform their choices and their application of translational science.

Lessons Learned:

NA can be conducted in a variety of ways to help focus and direct research to meet the perceived needs of targeted populations.

During the conduct of a NA, nothing precludes you from disseminating findings (i.e. lessons learned) and even solutions to needs at the same time.

Sometimes NA are an end as well as a means, reducing the needs they seek to assess.

The American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m Jim Altschuld, an emeritus faculty member at the Ohio State University in the school of educational policy and leadership. Today I’m sharing some tips on asset-based thinking when doing needs assessments.

First, for some premises to note:

Needs, discrepancies between current and desirable conditions, are problems, issues, or difficulties to be resolved.

Assets are resources (strengths) that can be used to improve what we do and programs to be delivered.

Needs and assets are related, yet different, philosophies through which people view, plan, and eventually evaluate efforts.

Most individuals employ/favor one or the other when beginning an endeavor and those lenses color viewpoints.

But, consider using both at the same time in an unbiased manner to optimally develop solutions to problems.

And now for some Hot Tips:

Divide and Conquer

To avoid one philosophy from dominating, divide assessors into teams to independently investigate needs and assets.

Creatively Review Findings

Have those that identified needs review the findings for assets and vice versa for the ones about needs.

Encourage Unique Ways to Think about Assets

Consider assets and how they might be employed not just as solutions for needs, but in distinct ways that capitalize upon them. (Be open to such thinking)

Recognize the Hidden Dimension of Assets

Volunteerism, help in patient care, in-kind community service contributions, the skills and abilities of individuals, etc. may be hard to measure but nevertheless is an important asset and serves an invaluable function. Without them the fabric of our society would be much less rich.

Be Instrumental when Resolving Needs and/or Utilizing Assets

The best solutions for underlying problems are those that would be helpful in dealing several problems at the same time (the instrumental concept).

Examples Illustrating Principles in the Tips:

Use of Public Libraries

Need: enhancing out of school computer learning for grades K-3

Solution: design programs for children that could be adapted to help elderly populations in getting connected to the internet

Fostering Volunteerism through Recreational Businesses

Need: government services will be cut due to reduced budgets

Solution: recreational businesses offer chits for what they provide to incentivize volunteers to cover some service gaps

Community to the Rescue

Need: tiny start-up business is redoing inside of small building and owners find they don’t have the skills to do parts of the work

Solution: ask the community to participate in exchange for small vouchers ($10) to be redeemed when business opens (community eagerly jumps in, vouchers redeemed, business takes off with sense of community ownership).

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Recently in my Virginia neighborhood, community leaders and members held a charrette. You ask, what is a charrette (or charette)? Charrettes are gatherings (often time-bound and intense, with emphasis on a final push) of stakeholders to vision and map out solutions, while at the same time trying to resolve conflicts and obstacles to those solutions. It’s said to be derived from the French word for chariot or cart, and has its history traced to the word used when 19th century Parisian architectural students put their work into a cart – often in a rush – to transport to an exhibition or evaluation. Today charrettes are used in urban planning and design processes, usually with communities, and with groups ‘carting’ ideas and information to one another. The charrette process will often bring people and groups together with the goal of getting them moving quickly from investigation to decisions. This is especially helpful when they might otherwise get stuck in the process – sometimes for months or years.

How does it work? While there are many commonalities, there is no standard structure for charrettes. They are unique to the problem and group. They usually involve a facilitated process with larger groups breaking into subgroups. The subgroups work separately to understand the issues, brainstorm, and eventually come to decisions and design solutions. The subgroups then rejoin in a large group to share the subgroups’ work and generate dialogue and some initial decisions. Through this process, multiple interests and information are shared and explored. The process then continues, potentially over multiples sessions. Together members “work on the work” together, which is meant to enhance ownership, move past obstacles, and come to joint decisions. In community and urban planning situations, it’s not surprising that charrettes are seen to be a good tool where consensus is needed to get to workable solutions.

For those working on needs assessments – which emphasizes determining gaps, causes to those gaps, optimal results, and potential solutions – the charrette process can be another ‘tool’ in the toolbox to explore. In my research on charrettes and related community visioning exercises, I’ve gathered a range of resources from those who have used them and hope you find them useful in your work.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Don’t scroll down! Don’t hit back button yet, read this first! My name is Ryan Watkins and I am a professor at George Washington University. Whereas many aspects of a needs assessment parallels an evaluation, the making of recommendations about what to do next is often a new challenge for us when we move from the back-end to the front-end of project (or program) design. Needs assessments, by definition, do however require that we not only identify and measure the “needs”, but that we also make recommendations about what to in order to satisfy (or partially satisfy) those needs in the future.

The good news is that there are many tools that can help us through the challenges of making these difficult choices. And yes, these are typically very complex decisions without any clear “best” options. The nature of these decisions is that we must forecast the future and what results we can reasonably expect of various alternatives. Foresight is never an exact science and it does present some risks to our projects and institutions. But making recommendations is a distinguishing characteristic of needs assessments.

The systems theory principle of equifinality tells us, however, that for any result we want to achieve (or need we want to satisfy) there is always more than one way to get there. Therefore we must know about our options and make choices among competing alternatives – which is typically not an easy task. To succeed, and to end our needs assessment with valuable recommendations about what to do next, we should balance our rational decision-making with our value-based decision-making.

The 2015 World Development Report (published by the World Bank and available free) is a comprehensive guide to behavior economics and provides an excellent guide to understanding the complexities of how we make decisions.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Ingrid Guerra-Lopez, Professor at Wayne State University and CEO of the Institute for Needs Assessment & Evaluation, and I want to help you through the transition from problem identification to solution design. Needs assessment give us an excellent process to identify the critical gaps in results (what), analyze contributing root causes or factors (why), and then make suggestions on (how) to address the problems and opportunities. A useful needs assessment includes recommendations that address problems and opportunities from a systemic perspective. However, the level of detail with which those recommended solutions should be delivered to stakeholders can vary greatly, and this in turn impacts the focus and scope of work.

A design-oriented needs assessment is focused on analysis for the purposes producing specifications for designing, developing, and implementing solutions effectively. For example, needs assessment with a knowledge and learning design orientation may be triggered by the desire to develop a learning program for a particular target group. The assumption is that the learning program is a solution to a given problem, hopefully identified through a previous and rigorous needs assessment. The focus in this case, is primarily to investigate what the solution should look like, how it should be implemented, and what will enable it to achieve useful results.

Hot Tip: Collaborate with those who will be designing, developing, and implementing solutions to understand their decision-making and information requirements. Clarifying the specific questions that must be answered for solution designers and implementers will ensure you focus on concrete and ‘actionable’ information.

Hot Tip: Ensure that you gather information from a cross-section of stakeholders to understand the specific requirements of the solution. For example, in order to define what a particular educational program should look like, include potential participants, employers of program completers, subject matter experts, instructors, administrators, etc.

Hot Tip: Balance perspectives of “must have” and “nice to have.” Our process must help us understand in concrete terms what the solution must help participants/users do and achieve, not only what they may want as solution features, characteristics, or content.

Hot Tip: Agree ahead of time on a process and criteria for integrating, and potentially weighing, the solutions requirements of the various stakeholders. At the very least, this will help you make recommendations about implementing phases in light of utility, readiness, feasibility, acceptability, resources, and other agreed upon criteria.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Hsin-Ling (Sonya) Hung, Program Co-chair for the Needs Assessment (NA) TIG. Because of my involvement in the TIG, I have reviewed annual conference proposals since 2008. Over the years I found that some proposals only had a title associated with needs or needs assessment, but these are were not needs assessments. Since needs assessments are often misunderstood, here I share what I think are the key elements constituting a needs assessment.

Using cooking as an metaphor, even a skilled chef would not be able to prepare a tasty dish without the necessary ingredients. So, if you are going to do a basic needs assessment, certain elements must be included. To create a simple ‘recipe’ for planning and reviewing needs assessments, I’m starting with Altschuld and Kumar’s definition of needs assessment.

The fundamental of a needs assessment is to assess need (attend or resolve a problem) for improvement of organizations or systems. A need is the measurable discrepancy between two conditions—“what is” and “what should be.” Without assessing a ‘gap’ it is not a NA. A genuine needs assessment project would describe needs and the conditions associated with them.

So how do we make this dish? After problems/issues have been depicted, you go through a process to understand the situation, the nature and the causes of the gap(s), prioritizing needs, making decisions about their resolution, and finally developing an action plan for improvement. All of these procedures would engage many constituencies and involve collecting much information. It might include organizing a needs assessment committee (NAC), examining root causes, prioritization, making needs-based decisions, and implementing action plan–all key parts of a simple needs assessment recipe. Detailed of all these can be found in the needs assessment kit edited by James W. Altschuld.

Lesson Learned: A needs statement presented in discrepancy form is essential, as well as other components presented above. Without these, the needs assessment recipe will produce an unappetizing product

Rad Resources: Check out the Needs Assessment Kit edited by James Altschuld.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Welcome to Needs Assessment TIG week on AEA 365! I’m Lisle Hites, Chair of the Needs Assessment TIG, Associate Professor in Health Care Organization and Policy and Director of the Evaluation and Assessment Unit (EAU) at the University of Alabama at Birmingham (UAB) School of Public Health. We hope you enjoy this week’s TIG contributions and look forward to seeing you at our sessions in Atlanta. Today’s posting is about the use of a Scenario Based Needs Assessment to Assess Community Needs.

In over 15 years of working with agencies and communities to assess needs, I’ve learned there’s really no one technique that suits every situation. To illustrate this point I’m sharing a recent, somewhat unusual, community-based needs assessment we conducted that was very specifically focused on the needs of daycare centers to prepare for a potential “active shooter”. While this unfortunate scenario has been fairly well assessed for public schools and other agencies (and considerable resources have been applied to follow-up on identified needs), little attention has been payed to this highly vulnerable daycare group.

We drew our methods from the disaster and emergency preparedness field. Both Federal Emergency Management Agency and Homeland Security have developed evaluation protocols (Homeland Security Exercise and Evaluation Program) to help prepare and test plans to assess needs and prepare for disaster events. With our county’s Children’s Policy Council and local law enforcement we developed a scenario that would initiate conversations and encourage representatives of daycare centers and law enforcement agencies to identify, discuss, and capture needs. By planting evaluators within each discussion group, we captured identified needs, found solutions in some cases, developed plans for finding these, and gathered lessons learned. Altogether, we acquired a reasonably comprehensive set of immediate needs for this non-homogenous group of small daycare businesses. As a result of this scenario based needs assessment and the new connections among daycare center teams and law enforcement officers, daycare centers have a better idea of what they need to do to prepare for an active shooter event and now many have relationships with local law enforcement to begin this preparation.

Lessons Learned:

Needs Assessments can be conducted in a variety of ways using existing data in new and innovative ways.

During the conduct of a Needs Assessment, nothing precludes you from disseminating findings (i.e. lessons learned) and even solutions to needs at the same time.

Sometimes Needs Assessments are an end as well as a means, reducing the needs they seek to assess.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Natalia Woolley. I am a GEDI scholar from the 2014-15 cohort, and a graduate student in the Community Health Sciences department at UCLA. As part of the GEDI program, I interned at Kaiser Permanente (KP), in the Community Benefit Department’s evaluation unit.

At KP, I provided support for the Community Health Needs Assessment (CHNA), a federally mandated process all non-profit hospitals must conduct every three years. During the internship I focused on the methods used to collect and analyze primary and secondary data. I also collaborated in the department’s efforts to ensure primary data collection methods were systematically responsive to the cultural diversity found in the communities served by KP.

Lessons Learned:

Operationalizing culturally responsive practices is a challenge. Although many scholars have defined culture and articulated its importance when conducting evaluations, it is still a challenge to operationalize some cultural concepts. Nevertheless, I believe acknowledging the challenge is an important step into making needs assessments more culturally responsive.

Successful primary data collection should be culturally responsive. Hospitals must collect primary data as part of the CHNA. This process allows hospitals to better understand the communities’ main health issues, priorities and resources. To successfully connect with community members, hospitals should ensure their outreach and engagement are culturally responsive.

Hot Tips:

Helpful Hints: Secondary data can inform culturally responsive primary data collection. Secondary data provides a great deal of information about the groups living in each community. For example, secondary analysis results can provide a snapshot of the community demographics, including the population percentage with limited English proficiency. Evaluators can use this information to include language appropriate resources in the data collection process.

However, secondary data might miss some marginalized groups. To go beyond the secondary data, it is helpful to identify and contact organizations working with marginalized groups. For instance, Los Angeles County has an extensive database of organizations providing services to groups in need (https://www.211la.org/). Another possible option is to solicit input from community health workers servicing these groups.

The American Evaluation Association is celebrating Graduate Education Diversity Internship (GEDI) Program week. The contributions all this week to aea365 come from AEA’s GEDI Program and its interns. For more information on GEDI, see their webpage here: http://www.eval.org/GEDI Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.