Setting up Mattermost

When you live far away from some of your best friends, you want to find some reasonable ways to stay in touch. After a while, we found out that social media and instant messaging apps are not really an answer to our needs, so we decided to give a chance to something that easily engages everyone. Most, if not all, of us know about Slack and how it (usually) improves communication within teams that use it. However, if you want to fully own your data and e.g. have access to all archives without high pricing, you might start with some cheaper alternative. It appears there is one - Mattermost Team Edition. And that’s what we decided to try out.

In this post I want to describe how I configured my Mattermost server step by step. I made some assumptions about used hosts, distros and setup that simplified the whole process. With those assumptions in mind, I compiled knowledge about basic server security guidelines from Mattermost’s documentation and several tutorials. I’ll show that, as long as you are not afraid of command line, you can quickly setup your own secured chat server from scratch. After that, I will describe how its UX compares to Slack’s and whether or not it’s something right for you.

The setup we decided to try out is a single Debian server with TLS secured connection and daily backups of all configs and user data in the cloud. We decided to go with the cheapest Linode server, obtain a TLS certificated from Let’s Encrypt and back up our data using both Linode backup functionality and AWS S3 storage. Altogether, it should be under $13 per month. Since Let’s Encrypt won’t allow you to register a certificate to domain like li[number]].members.linode.com, you should also obtain some nice domain, if you don’t have one already.

I was honestly surprised, how easy it was to setup a Mattermost server to the point where it just works. Merely following the tutorial let me set things up in about 2 hours, from the moment I registered a Linode account to the moment when I registered as the first user on working site. Most of that time I spend reading comprehensive tutorials to learn how to tweak things in the future (links to useful materials at the end of the post).

Preparing a server

First, I registered at the Linode site and configured my planning - 1 node with 2 GB of RAM, 1 CPU Core, 24 GB of space and 2TB of transfer. With that, I created a node with a Debian Jessie distribution, set up root password and booted it.

From Remote Access tab, I read the server’s IP and used it to log in via ssh (for the sake of this post, let’s assume that the server’s IP is 1.1.1.1):

I chose a password that is easy to remember, but long enough that no one should hack into my server during the next hour or so - I wanted to block logging in with root and password as soon as possible.

Next, I created a new admin account on the server and added it to the sudoers group. Then I logged out of the root account:

Then I generated a RSA key and set up ssh to use it when connecting to my server. I connected to server using the admin user (ssh gave me a warning about not being able to connect using RSA key - that is expected as I was just about to configure it):

Then I added my public key on the server:

I logged out and logged in to check if logging with the RSA key works as expected. Once I confirmed that I can do that, I could apply first security measures: blocking login by password and logging into the root account.

I changed the default port, let’s say to 222 (any port < 1024 and != 22should do) and checked that some options are set like below:

To make sure that everything worked as expected I checked that:

Finally, I made sure that the server is up to date:

That is a nice moment to take care about configuring the domain. Usually DNS propagation can take up to 72 hours and Let’s Encrypt would require us to have it already configured. Considering that I wanted my connection to be secure, outdated DNS would be a blocker. I assume that it is chat.mypage.com.

And now Mattermost!

Setup database

Official guide for Debian suggests using 3 machines:

database and files storage,

Mattermost node,

nginx load balancer.

However, if your team is really small (as mine), you can get away with a simpler single-server setup. Here nginx will be used to handle SSL termination, http to https redirection and port mapping.

I decided to go with a PostgreSQL database. I installed it and logged into postgres:

and created a database:

then quit the postgres account (and a postgres REPL) with exit.

You don’t have to name your database mattermost, your user mmuser and make password mmuser_password. And I strongly suggest you change them.

Official guide suggest you to make some changes to the PostgreSQL configuration, but as in this example I’m not setting up the DB as a separate server, it is not needed.

Installing Mattermost

I downloaded and extracted the Mattermost installation inside /opt with the data directory under /opt/mattermost/data:

According to good practices, I created a dedicated account for the Mattermost service (since I run it as a service) and made my admin part of the group allowed to mess with the installation:

Halfway through the setup! I opened the settings with en editor:

to change "DataSource"’s value into "postgres://mmuser:mmuser_password@127.0.0.1:5432/mattermost?sslmode=disable&connect_timeout=10". At this point I could check if the server could be actually be run:

When I connected to 1.1.1.1:8065 I could see that the server was working as expected. I used that opportunity to quickly register the first user (which automatically becomes server administrator).

I got back to ssh and terminated the server with ctrl+c. The only thing left was turning it into service. I created the file /etc/systemd/system/mattermost.service and filled it with:

and then loaded my newly created service with:

With that I had Mattermost running at (default) port 8065. Personally I preferred to change that port (because why not) by editing /etc/systemd/system/mattermost.service and restarting the service.

Configure nginx

Well, actually at this point I already had some functioning chat service, right? But I didn’t intend to stop on that - it still required passing the port number and the connection was not secure. To address these issues, first I had to install nginx.

After accessing http://1.1.1.1 I could confirm that nginx was working. So now I had to configure it to redirect :80 calls to Mattermost. I’ve created file /etc/nginx/sites-available/mattermost:

Now I had to link it as an enabled site, remove default config and restart nginx:

With that I had a nicely working server… which still needed some security tweaks.

At that moment I’d only adjusted ssh logging in a little bit. I had to add a few more things to introduce minimal security of the server. First I started by making sure that one won’t try to attack ssh by brute force. Such an attacker could be banned with fail2ban.

fail2ban

Now we would like to make sure that ssh configuration point to the right port:

AFAIR the rest of the default setup on Debian Jessie was good enough. Then I needed to adjust iptables to filter out any other port scanning attempts.

iptables

I created a file with iptables rules with:

and filled it with the following content:

It is slightly different from the original file from the tutorial I based my setup on - I had to uncomment outbound DHCP request as I used Linode host and it was essential for server to work. Another change was allowing inbound HTTP - without it some Mattermost requests were misbehaving: no message send confirmation, etc.

I had to test these rules before accepting them permanently:

With command above I had 60 seconds to check if I could create new connections to the server. I recommend checking:

whether a new ssh connection can be made (in another terminal/console),

whether one can establish a connection to Mattermost server, log in and send message with confirmation.

If the answer to both questions was true, then I could confirm that iptables were set up correctly and apply these settings permanently. I needed a few tries before it worked as intended, but once I introduced changes described above anything worked as expected. I only needed to add a hook to make sure settings would be applied on network startup:

where /etc/network/if-pre-up.d/iptables was filled with:

TSL and nginx again

Now the only thing I had to take care of was creating a SSL (TLS) certificate and using it to secure the connection. Let’s Encrypt allows us to get a certificate for free so I gave it a shot. First, I added jessie-backports to /etc/apt/sources.list and updated apt cache. With that I was able to install certbot from repositories:

At this point I already had my domain - I really recommend setting it up early on as I REALLY had to wait about 72 hours before my ISP’s DNS cache had it. And with the settings I want to describe here, I could no longer access the site through IP or Linode’s domain. (A workaround was to move away from my ISP’s DNS cache as it surprisingly worked more reliably and faster, but that’s a tale for another time). I’d like to remind that I assume here that the domain is chat.mypage.com.

Since certbot’s support for nginx is experimental and not turned on by default I used standalone routine - it required me to turn off nginx for a moment, so that certbot could set up a temporary server, and use it for serving content to Let’s Encrypt servers to prove that the domain truly belongs to me (or specifically that the domain points to the server that cerbot is currently run on).

Sometimes running certbot requires several attempts - to avoid spamming with unnecessary requests, their server has quotas for certificate requests. If your request was denied, just try again later on. At some point you will succeed. Once I obtained the certificate, I was able to access it in /etc/letsencrypt/live/chat.mypage.com. With that I could finally make nginx use it to secure HTTP(S) connections. I changed the content of /etc/nginx/sites-available/mattermost to:

Finally I started nginx again:

to confirm that http://chat.mypage.com redirects to https://chat.mypage.com which in turn uses secure TLS connections.

To make sure that certificate would be renewed every 30 days (Let’s Encrypt gives certificates valid for only 90 days) I also created a cron job:

While the application was ready to use and I could actually start allowing users on the server (after I played around with Mattermost’s settings), I needed to make sure that whatever data would be generated - user’s archives, uploaded files and so on - would be preserved for certain. First thing I could do was enabling backup on Linode - it costs me $2.5 a month and creates a snapshot of my whole node once a week.

But once a week is not enough. What if I wanted to restore content 6 days after a last backup? What if I wanted to move it to another node/server? For that I needed to find some additional solution.

Upload to AWS S3

I decided to go with AWS S3 storage. Uploading all the config files and user data once a day should give me enough confidence and with AWS’ prices it is even cheaper than Linode’s backup.

Installation of AWS command line tools is rather simple:

For AWS S3 I needed an Amazon account paired with my credit card (for billings). There, in S3 services settings, I created a new bucket for my backups. Finally, I used Security Credentials for creating keys just for my backups. I used them for creating an aws_upload bash script:

After sourcing it into my backup script I would be able to upload backup file as backup-from-${current-date}.tar.gz with command like:

Since it keeps credentials in a separate file I could check backup script into git ignoring aws_upload and avoiding credentials exposure.

Encryption

Because I have limited trust to third-party storages I would still encrypt the backup with something like mcrypt before the upload.

Installation of mcrypt was even faster:

Since it requires some key for encryption I decided to generate a random one:

Then I could encrypt/decrypt backups with something like:

Similarly like with credentials I added the keyfile to ignored so that the key would not be published in a repository.

My backup scripts

Long story short - backup scripts would just copy all config/data files into one place, archive them and publish on AWS S3. The only exception would be Mattermost DB which would be backed up with pqdump to create a dump file which can just be read into the DB like any other PostgreSQL script. To avoid typos in hardcoded paths, I extracted all paths into variables and wrote a few helpers to make the scripts more readable (to me :D).

.common.sh:

Then backup is just a matter of defining tasks:

Similarly restoring:

Last thing to do is write a cron job to perform backup once a day:

Mattermost tweaks

Out of the box Mattermost would have some generic site name. Of all settings that we would like to tweak after installation, that is the only one we can only access by editing /opt/mattermost/config.config.json (TeamSettings/SiteName). Any other config that would be interesting to us could be accessed via the System Console window of Mattermost. That change would require us to restart Mattermost manually to take effect.

Once we change the site name to something more preferable we can also change title of notification emails (if we’ll use them) and admin’s email address. If we decide to allow push notifications for mobile devices (Android and iOS clients), then we’ll have to decide on the server - we can use test server (TPNS) which does not encrypt our messages (then I would recommend sending generic description with user and channel names) or setting up an encrypted push notification server manually - problem with the later is
that official clients only support Mattermost servers. So one has to either pay for Mattermost’s encrypted push notification service, or fork clients and release one’s own version.

I would also recommend skimming through other options to e.g. regenerate hashes for links and turning off public user registration. If we decide to have several unrelated teams on one server, we might also disable option for cross-team communication.

As for customization, for now we can only set up global default localization options. An option to change default fonts or themes is still an open ticket on the issue tracker.

How does Mattermost actually compare to Slack in action? I won’t deny it - for me it looks less polished. Of 10 possible fonts, only 2 (Open Sans and Roboto) seems to handle Polish characters correctly and 1 of them (Roboto) seems to fail when we use bold (e.g. when in notifications from some channel). We cannot change the size of the font, and by default they seem a little bit too small (after zooming in, they seem to be a bit too big). The default theme has annoying sets of colors and I am unable to change these settings without manually editing source code files and preparing my own build.

Some other potentially useful options are also lacking. I am unable to configure the server so that only admin could invite users. Instead, there is a registration link with a hash which could be hidden from the landing page and (optionally) requiring email registration from fixed domain. As an admin I can only block any registration to prevent users from sending it to whoever they want. Alternatively, I can disable any registration and use Mattermost command line interface for creating my users (probably best solution for enterprises). And newly registered users cannot be automatically invited into a predefined list of channels.

Mobile clients seems to be build using an embedded browser - I only tested the Android client, and it has significant startup overhead before I’ll see that my chat is even loading. I attribute it to the browser initialization. When I receive a notification and click it to go to the message, client reloads browser, even if I just have the said conversation opened. Currently the client does not have an option to open menus by sliding screen to the side, though that’s exactly how they open once we click on a menu icon. You can also only have one server configured in your app. (And you cannot remove it easily - when I switched from HTTP to HTTPS the app stuck on an infinite server load and I had to reinstall it in order to enter a new URL). Overall, I got the feeling that the mobile clients are slow and heavy (at least Android one is. AFAIK iOS client is build the same way so I assume it shares it’s brother’s issues).

Other quirks that brought my attention was no ability to add reaction emoticons (we use them on Slack heavily) and worse, viewing thumbnails for assigned files - however the former should be solved with 3.5 release. When you remove a message ugly message removed notice would stay (it disappears after reloading but still). If you spent a lot of time on Slack, these small quirks will add up and you won’t be able to help but notice the difference in the user experience. So does Mattermost have any advantages beside the pricing?

I’d say yes. Even if currently each user would have to apply customization settings manually, they are still able to change color of almost everything. (I weren’t able to locate the option to disable my own messages’ highlighting - each message you send has a different background than all the rest. This color is also used for highlighting the message your cursor is over it.) Having the ability to respond to a message directly (kind of like in Reddit) might also be a nice thing if 2 parallel discussions emerge on one channel.

When looking at the message bar you’ll notice that it lacks Slack’s plus button, it only has assignment. There is no distinction on photo or generic file upload (which is a minus) and there is no option to publish a snippet. The latter is not that much of a problem, though, as Mattermost has better markdown support than Slack, so you can paste you code like on GitHub:

and with the ability to respond to a message it works similar to commenting assignments.

It is also useful to share server for several teams, because if you want those teams to communicate, you can still allow that. And anyone on the server could then talk to anyone else, without the need to register on each team separately.

Having access to the whole archive out-of-the-box and without additional fees is also a big plus. And if you are really concerned about the security, privacy and ownership of your data, having your own secured server will always beat trusting third-party vendor locked-in solutions.

Setting up my own Mattermost server was a great learning experience. And a lot of fun. Though the service is less polished than a Slack, it suits my use case and my pocket. I am looking forward to new releases which will bring new functionalities and get rid of aforementioned quirks.

I cannot recommend this solution to everyone - limited ability to control who can register might be a deal breaker for many companies. However, if you are managing a small team (of friends?) or your company is OK with manually managing users via command line, or the Mattermost server is only available in the intranet or via VPN it might br a quite good Slack alternative. In such a case, though, you should consider subscribing for paid push-notifications to make sure they are distributed via secured connections. Or, if your company is reaally concerned about owning everything, forking mobile clients to use your own dedicated push-notification server (for a while forking the platform might be the only way of providing your own color-scheme defaults). Or perhaps just go straight to Mattermost Enterprise Edition which should ease all those issues and add some additional perks like rebranding. But here I wanted to focus on Team Edition.

In the end it’s all about the tradeoffs. If your primary concerns are flawless user experience and polished UI - go with Slack. If you want to go an extra mile to make sure that you actually own your data - Mattermost is the way to go.

These are the sources I read while configuring my server. I admit that a major part of my text is just a creative ripoff ;)

Production Install on Debian Jessie - complete manual on setting up Mattermost on a server. I just skipped all settings required by 3-servers’ setup and replaced cloning Let’s Encrypt with certbot already available in jessie-backports.