“We left with a plan that we feel will be feasible, accepted, and actionable. Fabulous experience!”
–TNLI participant

The TLNI experience fosters data-informed change initiatives that improve the quality of students’ experiences in courses and educational activities across the curriculum—leading to improvements in student engagement, retention and graduation, and learning outcomes. With a mix of plenaries, structured concurrent sessions, team time, and small- and large-group sessions facilitated by experienced educators from across the country, this residential institute, held at The Evergreen State College, provides a learning space for campus teams to use evidence to develop plans that can guide the work of examining, improving, and supporting what happens in learning contexts across their campuses. The institute fee covers on-campus housing, meals, workshops, sessions, plenaries, and materials.

Teams are encouraged to use a variety of forms of evidence including their NSSE results in their action plans. In TLNI’s previous years, teams have used NSSE data and other evidence to inform a comprehensive communication strategy for new students and to develop a first-year experience, to bring together student and academic affairs to develop an effective and equitable first-year experience, to focus on faculty development and the use of assessment data to improve teaching and learning, to examine students’ participation in and faculty emphasis of high-impact practices (HIPs) so as to design enriched HIPs, to develop new approaches to share assessment data with faculty, and to coordinate institutional efforts to use data to inform student success efforts.

Co-sponsors of the institute, along with the National Survey of Student Engagement (NSSE), are Achieving the Dream, Inc. (AtD), the National Institute for Learning Outcomes Assessment (NILOA), the Washington Center for Improving Undergraduate Education, and the Washington State Board for Community and Technical Colleges (WSBCTC).

Using NSSE Data to Predict First-to-Second-Year Student Retention

NSSE’s Psychometric Portfolio is comprised of a series of studies that demonstrate the project’s validity, reliability, and other data quality indicators. One specific type of psychometric study—a predictive validity study—assesses the extent to which a measure can predict future outcomes. Demonstrating predictive validity between NSSE’s Engagement Indicators (EIs) and student retention, for example, can help support efforts by educators trying to promote the use of effective educational activities across campus.

In a soon-to-be-released study, NSSE researchers document the relationship between both EIs and selected High Impact Practices (HIPs) and first-to-second-year student retention. Based on the responses of approximately 10,000 first-year students from 47 colleges and universities, the study shows the predicted probability of students being retained increases by 2 to 3 percentage points, on average, as responses to sets of EI survey questions change from doing activities “Sometimes” to “Often.” The predicted impact is even more substantial for students who report “Never” having done a set of EI activities and students who report having done them “Very often”—with the increase in retention probability ranging between 5 to 14 percentage points depending on the EI. As for the impact of participation in HIPs, first-year student participation in a learning community, research with faculty, and service-learning also increases the probability of being retained by about 2 to 3 percentage points, particularly among students entering college with lower standardized test scores.

Why does this matter? Results like these demonstrate that efforts at increasing student engagement have the potential to yield positive outcomes at both the student and the institution level. A win-win!

The Beginning College Survey of Student Engagement (BCSSE) measures entering first-year students’ high school academic and co-curricular experiences as well as their expectations for participating in educationally purposeful activities during the first college year.

New for BCSSE 2018: An online winter administration of BCSSE for new first-year students enrolling in January is now available to supplement the institution’s summer/fall administration of the survey.

Participating institutions receive student advising reports, a preliminary data file, and a final SPSS data file, in addition to their Institutional Report.

Canadian Institutions a Strong Presence in NSSE

Canadian institutions of higher education have participated in the National Survey of Student Engagement (NSSE) since 2004. While the number of institutions and students that participate in the survey varies from year to year, Canadians have been a strong presence in the NSSE milieu. To illustrate the Canadian contribution to NSSE, the graphic below shows the number of Canadian institutions and students participating in NSSE since the survey’s 2013 update.

Canadian institutions have also participated in NSSE through consortia, groups of at least six institutions that participate in NSSE during the same year and who append additional questions of mutual interest to the core survey. Over the years, Canadians have created several consortia: Canadian Research Universities, Canadian U4, G13-x-Ontario, New Western Canadian Universities, and Ontario Universities.

As a new feature on the NSSE website, results from participating Canadian institutions are now available on the Summary Tables web page. These include combined results from 2016 and 2017 Canadian respondents featuring frequencies, means, and Engagement Indicator statistics by sex and major as well as a table showing participation in High-Impact Practices by various student characteristics.

Inquiry Scholars Add Value at Southern New Hampshire University

At the Southern New Hampshire University (SNHU) University College campus, NSSE results and questions that arise from these results serve as the content of a one-credit School of Education course called “Inquiry Scholars.” Each semester, the students taking this course are asked to take up an authentic problem related to improving student learning that can be illuminated with data from their campus. Following an inquiry-based process, the Inquiry Scholars gain experience formulating and refining research questions, designing research studies, collecting and analyzing data, and communicating results. The stated outcomes for this course utilize language from the Association of American Colleges and Universities (AAC&U) VALUE Rubrics for Inquiry and Analysis, Information Literacy, and Teamwork.

After the administration of NSSE 2017 at SNHU, eight Inquiry Scholars classmates analyzed their institution’s results in several projects related to improving student engagement. The first project they completed was an analysis of the open-ended NSSE item, “What one change would you most like to see implemented that would improve the educational experience at this institution, and what one thing should not be changed?” The Inquiry Scholars literally put each comment from the 270 respondents who answered this question on a strip of paper and sorted them all into thematic affinity groups. They analyzed the results by gender and by year and then shared their findings with more than 150 faculty and staff members. Faculty, in turn, were asked to answer the same prompt during this event, and the Inquiry Scholars analyzed the faculty’s results as well.

Another project using NSSE data undertaken by the Inquiry Scholars centered on the larger body of NSSE results. From their reading of SNHU’s Snapshot report and the reports on Engagement Indicators and High-Impact Practices, the Inquiry Scholars generated and prioritized questions for further study. They applied to their campus Institutional Review Board (IRB) to run student focus groups for the project and, with IRB approval, worked in teams of two conducting focus groups of eight to ten students to learn more about the student engagement experience. From these focus groups, once again, the Inquiry Scholars generated common themes and shared their findings with university stakeholders including academic affairs staff, deans, and faculty leaders.

Check out NSSE Sightings, and if you sight good engagement-related research and findings, please let us know. As always, we welcome your ideas and feedback!

NSSE at AIR Forum 2018

NSSE hopes to see you at the upcoming Association for Institutional Research (AIR) Annual Forum in Orlando, Florida. Our staff will be well represented in the conference program book with various paper and poster presentations, a panel session, a half-day workshop, and, of course, our customary special interest group meeting. NSSE events are among many that institutions should find useful and interesting at AIR Forum 2018.

Academic Dishonesty Among Undergraduates
James Cole, Robert Gonyea, and Ryan MerckleAbstract:

That academic dishonesty occurs on campuses across the country is not new. Nonetheless, understanding how academic dishonesty relates to engagement and student success is important. Using data from more than 14,000 first-year and senior students enrolled at 32 bachelor’s degree-granting U.S. institutions, this poster presentation focuses on who cheats on college campuses and how cheating correlates with effective educational practices, time on task, academic achievement, and other factors important for student success. Group differences in the frequency and severity of academic dishonesty will also be presented. Finally, regression analysis shows which students’ experiences and engagement predict academic dishonesty. Audience members will be challenged to consider how understanding academic dishonesty can inform the programs and resources on their own campuses.

Quantitative and survey research depend heavily on large sample sizes, but there are a variety of reasons why larger sample sizes may not be possible. Participants in this presentation will discuss common issues and solutions associated with assessing small populations of college students and instructors. Examples will focus on the experiences of gender variant and LGBQ+ students and faculty. Participants will also learn about and discuss administration issues related to small populations such as increasing response rates and identifying special subpopulations, then, strategies for analyzing and communicating the results from small populations, and finally, approaches for communicating the validity and data quality from small sample sizes.

This poster presentation will present findings from NSSE, investigating whether High-Impact Practice (HIP) participation is more common in students with certain personality traits. In addition to the core NSSE items, this study uses responses from experimental items on the Five-Factor Model of personality. A series of exploratory t-tests, looking at the different HIPs included on NSSE, suggests several significant differences. Understanding of these differences can help institutions develop new HIP programming that is more appealing and inclusive for all students, regardless of personality traits.

Learning Strategies in High School and First Year in College
Lanlan Mu and James ColeAbstract:

This poster presentation will focus on the importance of learning strategies for student success. New college students enter higher education institutions with a habitual way of using learning strategies. The current study explores the association of learning strategy use in high school on that in the first year of college and examines how the environmental factors affect the association. The preliminary results showed that the use of learning strategies in high school and an institution’s academic challenge in course work had a positive effect on students’ use of these strategies in college. These findings are meaningful, as they show that the institution-wide learning and teaching atmosphere does influence individual students’ learning.

Lost in Translation: Transforming Survey Data to Get Started with Tableau
Allison BrckaLorenz, Amy Ribera, and Brendan DuganAbstract:

IR professionals must not only collect, analyze, disseminate, and archive information about their institutions; they must also report findings to a wide variety of stakeholders. One way to communicate information to a broad audience is through interactive data visualizations. Data visualization tools, such as Tableau, are often built with a mindset that may not be as familiar to today’s IR professional, and learning a new tool from a different educational culture may be daunting. This presentation aims to bridge the gap between traditional-IR and data-analyst thinking with tools such as SPSS and business intelligence tools such as Tableau. Participants will learn about the merits of “long” versus “wide” data and tips for converting from one format to another. Participants will learn how to use these converted data in Tableau, how Tableau jargon maps to traditional data-analyst terminology, and ways to get started using survey data in Tableau.

This presentation will provide a variety of real-life examples of how institutions have used survey data collected from students, faculty, and alumni within multiple contexts. Examples will be drawn from institutions participating in the National Survey of Student Engagement (NSSE), the Beginning College Survey of Student Engagement (BCSSE), the Faculty Survey of Student Engagement (FSSE), and the Strategic National Arts Alumni Project (SNAAP). The types of data use cover numerous categories: sharing on campus; recruitment; academic and career advising; publicity, alumni relations, and donor outreach; planning, assessment, and accreditation; program and curricular change; and advocacy and public policy.

R for IR: A Brief Introduction for Institutional Research Professionals
Brendan Dugan and Kevin Fosnacht Abstract:

This half-day workshop kindly and briefly introduces analysts to the open-source statistical programming language, R. R’s capacity to flexibly handle a wide variety of common IR tasks, from data cleaning to reporting, makes it an ideal tool for analysts. Attendees will work through foundational concepts for working with R, from vectors and environments to simple functions and mathematical operations to control structures. Attendees will also work through the basic stages of research: data import, cleaning, and manipulation; hypothesis testing; reporting and visualization. R syntax, data files, and resources for further learning will be provided. Participants do not need prior R or other programming experience, but familiarity with Excel functions and other statistical software syntax (SPSS, MATLAB, SAS, Python, Stata) is beneficial. R and Rstudio should be installed prior to the workshop.

Survey analysis and presentation can become difficult as surveys grow more complex, covering multiple constructs. Diagnostic Classification Models (DCMs) provide a means of simplifying survey data while yet enabling rich analytical possibilities. DCMs approach latent constructs similarly to item response theory but provide more fine-grained information by producing categorization of respondents across several attributes of interest. Using data from the Beginning College Survey of Student Engagement (BCSSE), this study demonstrates how DCMs provide easily understood and contextualized categorizations that can lead to better understanding of student and institutional needs.

High quality data and assessment instruments have become essential for institutional researchers to take a data-driven approach to informing decision-making and strategic planning. Instruments and the resulting data they collect can be studied for different aspects of validity and reliability as well as the procedures and standards used to reduce error, bias, and increase the rigor of the data. This presentation will focus on a framework for operationalizing and organizing a wide variety of studies to investigate data quality. Participants in this session will see how a large-scale quantitative survey project designed and created a psychometric portfolio with studies designed to make survey instrument and data quality transparent so that higher education leaders, researchers, and professionals can trust the results. Challenges and potential solutions, including thinking about strategies for conducting studies of data quality with limited time and resources, will be discussed.

Undergraduates typically enter college ill-prepared to manage their own finances, and so preparing students to be financially literate and knowledgeable should be a liberal learning outcome. This poster presentation examines the relationship between students’ financial knowledge and participation in financial education programs. Using data collected by the National Survey of Student Engagement (NSSE), the results indicate that the efficacy of financial education programming varies by the method of delivery. Implication for future efforts are discussed.

VALUE Institute Launched

The Center for Postsecondary Research, NSSE’s home unit at Indiana University, has launched the VALUE Institute in collaboration with the Association of American Colleges and Universities (AAC&U). Through this project, higher education institutions and other postsecondary education providers can have authentic student work scored by trained raters using VALUE rubrics. For more information, visit valueinstituteassessment.org.