Mattermost Pilot

The Mattermost pilot took place at UBC during the 2017/18 academic year. After the pilot concluded, the LT Hub gathered feedback from instructors, teaching assistants (TAs), and students to evaluate the tool’s perceived strengths and weaknesses, and to make implementation recommendations.

Mattermost is a self-hosted open-source chat tool for facilitating online communication and collaboration. In the context of teaching and learning, Mattermost allows for the creation of course- or cohort-based “teams.” Each team receives access to an online space where instructors and students can freely share messages and files. Conversations can occur in public channels, private channels, and direct one-on-one messages. All posts are open-ended and searchable, allowing individuals to follow conversations as they happen or return later on.

Mattermost was released and supported for a trial run in more than 11 courses, including Arts and Science, but predominantly from Education. Half were at the master’s level, with first-year, third-year, and fourth-year courses comprising the rest. Most of the courses were online, with only a handful offering an in-person classroom component. Interviews took place with seven instructors and one teaching assistant, and 202 students responded to surveys.

Instructors taking part in the pilot were interested in improving efficiency in communicating with students and enhancing the student experience, especially for distance courses.

Perceived strengths

Good one-on-one instructor-student communication: Mattermost helped filter course-specific messages for instructors into one space and provided new useful venues (e.g., private channels, direct messages) for checking in with individuals.

“That was a great way to chat with them.”
“We could just have a quick dialogue connection and move forward.”

Ability for students to answer each other’s questions: Instructors could crowd-source answers to common questions in the public channels.

“In many cases, they will ask questions in the open town square environment.”
“Someone else could provide the answer without me.”

Stronger sense of online community, when uptake is high: With the active participation of peers, students seemed to engage each other more.

“They have super rich conversations that had nothing to do with me.”
“Asking each other ‘hey what are you doing with regards to this activity?’ or ‘I just saw this and I thought I would share it with you.'”

Modern, relevant interface and interactivity: Using Mattermost was viewed as a way to prepare students for communication spaces used in modern workplaces.

“If people can get familiar with these kind of tools that are actually being used in real environments, that’s kind of meta teaching.”

Potential of persistent learning space: Mattermost teams can exist outside an individual course so that students can maintain access to the same shared space over time.

Perceived weaknesses

Pressure on instruction team work-life boundaries: Real-time chat presented more opportunities to engage at any time from any place. Not all instructors/TAs welcomed this constant connection.

“When you’ve got 150 students, there’s an expectation to always be around.”
“I’m really really responding to them all the time.”

Unclear how much to encourage student uptake: Students already use many tools for communication and not all will gravitate toward this format. Instructors were unsure how much to push them.

“Some students didn’t want to engage that way at all.”
Some students didn’t…”feel uncomfortable with the trend of more and more and more.”

In some cases, there was a need for more functionality: To help assign marks, some instructors required better participation and activity statistics.

Student experience was largely reported positive (45%) or neutral (43%). Open-ended responses implied that, rather than neutral ratings indicating apathy about the tool, students genuinely “feel mixed about it” and often saw “there are upsides and downsides” to using Mattermost.

Very negative (4.4%)

Somewhat negative (7.55%)

Neutral (43.4%)

Somewhat positive (23.27%)

Very positive (21.38%)

Quantitative benefits

Opinions varied regarding the benefits to specific learning activities, though students agreed with the instructor-reported benefits to communicating with instructors and getting quick answers to questions.

Did not benefit

Somewhat benefited

Neutral

Benefited

Greatly benefited

Never used

Communicating with the instructor(s)

Did not benefit (13.01%)

Somewhat benefited (6.85%)

Neutral (14.38%)

Benefited (18.49%)

Greatly benefited (25.34%)

Never used (21.92%)

Getting quick answers to course-related questions

Did not benefit (10.96%)

Somewhat benefited (12.33%)

Neutral (15.07%)

Benefited (19.86%)

Greatly benefited (25.34%)

Never used (16.44%)

Understanding the course topics

Did not benefit (16.44%)

Somewhat benefited (9.59%)

Neutral (23.97%)

Benefited (11.64%)

Greatly benefited (12.33%)

Never used (26.03%)

Completing individual assignments

Did not benefit (15.75%)

Somewhat benefited (8.22%)

Neutral (22.60%)

Benefited (12.33%)

Greatly benefited (13.01%)

Never used (28.08%)

Completing group assignments

Did not benefit (16.44%)

Somewhat benefited (6.16%)

Neutral (11.64%)

Benefited (6.85%)

Greatly benefited (8.22%)

Never used (50.68%)

Preparing for quizzes or assessments

Did not benefit (18.49%)

Somewhat benefited (10.27%)

Neutral (17.81%)

Benefited (4.11%)

Greatly benefited (7.53%)

Never used (41.78%)

Connecting generally with other students

Did not benefit (18.49%)

Somewhat benefited (10.96%)

Neutral (16.44%)

Benefited (16.44%)

Greatly benefited (7.53%)

Never used (30.14%)

Contributing my own thoughts and opinions

Did not benefit (19.18%)

Somewhat benefited (8.90%)

Neutral (16.44%)

Benefited (13.01%)

Greatly benefited (6.85%)

Never used (35.62%)

Perceived strengths

Helpfulness of peer interactions and connection: Students noted the overall benefit of being able to interact with each other through this medium.

“I liked the ability to connect and chat with classmates in real time.”
“Public discussions were very helpful in directing me.”

Good overall ease-of-use and features: About 59% of students agreed with the statement “Mattermost was easy to use” and qualitative comments highlighted specific useful features such as notifications.

Mattermost was easy to use

Strongly disagree (5.66%)

Somewhat disagree (8.18%)

Neutral (27.04%)

Somewhat agree (32.70%)

Strongly agree (26.42%)

Immediacy of interactions removed blockages quickly: The ability for students to reach their instructional teams and/or peers and hear back immediately helped resolve issues efficiently, so they could get back to work.

“The ease of being able to have quick communication with my professor and TAs made me more inclined to ask questions.”

Nice to share informal space with peers, when uptake is high: The informal feel of Mattermost led to the sharing of more and different content than static discussions boards might invite.

“Lots of valuable additional information that…wouldn’t be in a more formal discussion.”

Perceived weaknesses

Platform disconnect from primary course sites: The lack of integration for Mattermost made it feel inconveniently isolated from the rest of the course.

“It was on yet another platform.”
“Made it more difficult to make the effort each week to check up.”

Information overload with existing communications commitments: As instructors suspected, not all students welcomed another online space to monitor.

“Many other modes of communication to keep up with.”
“Checking it on a frequent basis makes me feel overwhelmed and distracted from other things.”

Usability issues, mostly as more content added: With lots of activity, channels can quickly become clogged.

“Overwhelming and time consuming at times to be interactive.”
“It’s not easy to sort through and find what’s relevant to me.”

Feeling of being left out at times: Students were not always available or ready to engage when their peers were online.

“When I get a chance to log-in, often I feel very ‘out of the loop’ of the discussions.”

The following are recommendations for how Mattermost could best be implemented to maximize perceived benefits and minimize perceived shortcomings as a pedagogical tool.

Set up channels for smaller groups of students (approx. 10-30): Help students meaningfully contribute and interact without feeling drowned out or overwhelmed, especially in more extensive courses. Learn more »

Organize content in clear channels and guide organization: Model and explain to students how best to use areas and features to create a shared understanding of effective communication. Learn more »

Set explicit expectations around availability of instructional team: Tell students how and when instructors/TAs will interact in real-time vs. with delay. Learn more »

Integrate with the Learning Management System or regularly prompt use from other course sites: Make chat feel like an integral and useful part of the course, even if it’s external to the primary course site. Learn more »

Set loose guidelines for student participation: Decrease student stress and disengagement with norms other than 24/7 participation, including:

Scheduling specific times for real-time discussions that work for the majority.

Highlighting a single channel, hashtag, or thread weekly to help students who aren’t broadly engaged to focus on one conversation.

Adding marks or badges to reward participating.

Assuring students they can meaningfully contribute later (not everything has to be real-time).

Explain why Mattermost over other more established chat tools: Justify students learning another interface while explaining and educating them on the importance of privacy in social media platforms. Learn more »