Mender Server

This document details troubleshooting steps for the most common problems with the Mender server.
The first part applies to all installations, while the section below on Production installations
only applies when the Mender server is installed for production.

Cleaning up the deviceauth database after device decommissioning

It is possible that after a failed device decommissioning operation there will be some unaccessible and unnecessary data in the deviceauth database. In this case, you should clean the database manually.

Is is recommended to backup your data before performing the clean up operation.
The Backup and restore chapter provides examples and
introduces example tools provided in Mender integration repository.

To clean up the deviceauth database, run the following from within the integration repository:

The virtual QEMU device is not showing up in demo mode

When running the Mender server in demo mode, as described in the getting started tutorial,
a virtual qemux86-64 device should connect to and ask to join the server.

If this does not happen, please make sure your environment meet the resource requirements
to run the Mender Server. In particular, it is known that the virtual device will not
start if you do not have enough memory.

A device shows up as pending after preauthorizing it

If you see your device gets the pending status after preauthorizing it, something went wrong. Most likely there is a mismatch between the identity and public key you preauthorized and what your Mender client is actually using.

To diagnose this, look for the device identity in the Device Authentication service, for example:

In this case you can see that there are two authentication sets with the exact same device identity: {"mac":"52:54:00:50:9b:84"}, one preauthorized and one pending. So the device reported (see the pending set) the exact same identity as we preauthorized; however, there is a mismatch between the public keys.

The problem here is most likely that your server CPU does not support the SSE 4.2
instruction set, while openresty is compiled assuming SSE4.2 support.
Try again with a more modern CPU (from 2009 or later), or consider building
openresty from source for your architecture.

docker-compose may show Restating status for containers that are restarting in a quick succession, if containers restart after a longer while, they may appear as Up

In situations where some containers are restarting after running for a longer
while, docker ps will show the containers having a shorter lifetime than
others. In the listing show below, mender-deployments service uptime is
shorter than that of the other containers:

Docker event log

Docker event monitor provides a dynamic view of events. This feature can be
helpful, especially with filters applied. Example view of events registered when
a mender-deployments container was restarting (output edited for clarity):

Inspecting containers

As seen in container logs section, mender-deployments
service is restarting. The logs suggest there might be missing credentials for
an AWS related service.
From the production installation guide, we can recall
that
mender-deploymentsservice configuration contains
credentials for artifact storage service.

Configuration of current instance of mender-deployments can be viewed using
docker inspect command. Looking for Env (container environment
configuration) in inspect output reveals that DEPLOYMENTS_AWS_AUTH_KEY and
DEPLOYMENTS_AWS_AUTH_SECRET are unset: