Juju logs

Model logs

Model logs can be considered as Juju's "regular logs" and are intended to be
inspected with the juju debug-log command. This method provides logs on a
per-model basis and is therefore more convenient than reading individual logs
on multiple (Juju) machines directly on the file system. The latter can
nonetheless be done in exceptional circumstances and some explanation is
provided here.

So there are two agents running on this Juju machine. One for the machine
itself and one for a service unit.

The contents of one of these directories

juju ssh 2 ls -lh /var/lib/juju/agents/machine-2

reveals the agent's configuration file:

-rw------- 1 root root 2.1K Apr 28 00:42 agent.conf

Consider keeping backups of these files, especially prior to upgrading the
agents. See Upgrading models.

The debug-log command

The juju debug-log command shows the consolidated logs of all Juju agents
(machines and units) running in a model. The juju switch command is used
to change models. Alternatively, a model can be chosen with the '-m' option.
The default model is the current model.

The 'controller' model captures logs related to internal management (controller
activity) such as adding machines and services to the database. Whereas a
hosted model will provide logs concerning activities that take place post-
provisioning.

Due to the above, when deploying a service, it is normal for there to be an
absence of logging in the hosted model since logging first takes place in the
'controller' model.

The output is a fixed number of existing log lines (number specified by
possible options; the default is 10) and a stream of newly appended messages.
Both existing lines and appended lines can be filtered by specifying options.

The exception to the streaming is when limiting the output (option '--limit';
see below) and that limit is attained. In all other cases the command will need
to be interrupted with 'Ctrl-C' in order to regain the shell prompt.

For complete syntax, see the Command reference page. The juju help
debug-log command also provides reminders and more examples.

Examples:

To begin with the last ten log messages:

juju debug-log

To begin with the last thirty log messages:

juju debug-log -n 30

To begin with all the log messages:

juju debug-log --replay

To begin with the last twenty log messages for the 'lxd-pilot' model:

juju debug-log -m lxd-pilot -n 20

To begin with the last 500 lines. The grep utility is used as a text filter:

juju debug-log -n 500| grep amd64

Logging levels

You can verify the current logging level like this:

juju model-config logging-config

Output will be similar to the following:

<root>=WARNING;unit=DEBUG

The above is the default configuration. It sets the machine agent log level at
'WARNING' and the unit agent log level at 'DEBUG'.

The logging levels, from most verbose to least verbose, are as follows:

TRACE

DEBUG

INFO

WARNING

ERROR

Change the logging level

When diagnosing an issue (and possibly filing a bug), the first step is to make
logging more verbose. For instance, to increase the logging level of the unit
agents to 'TRACE' you would enter the following command:

juju model-config logging-config="<root>=WARNING;unit=TRACE"

To avoid filling up the database unnecessarily, when verbose logging is no
longer needed, do not forget to return logging to normal levels.

Advanced filtering

A Juju log line is written in this format:

<entity> <timestamp> <log-level> <module>:<line-no> <message>

The --include and --exclude options select and deselect, respectively, the
entity that logged the message. An entity is a Juju machine or unit agent. The
entity names are similar to the names shown by juju status.

Similarly, the --include-module and --exclude-module options can be used to
influence the type of message displayed based on a (dotted) module name. The
module name can be truncated.

A combination of machine and unit filtering uses a logical OR whereas a
combination of module and machine/unit filtering uses a logical AND.

The --level option places a limit on logging verbosity (e.g. --level INFO
will allow messages of levels 'INFO', 'WARNING', and 'ERROR' to be shown).

Examples:

To begin with the last 1000 lines and exclude messages from machine 3:

juju debug-log -n 1000 --exclude machine-3

To select all the messages emitted from a particular unit and a particular
machine in the entire log:

juju debug-log --replay --include unit-mysql-0 --include machine-1

Note: The unit can also be written 'mysql/0' (as shown by juju status).

Agent logging override

As we've seen, the logging level for machine agents and unit agents are
specified as a single model configuration setting. However, in some situations
(e.g. targeted verbose debugging) it may be desirable to increase the logging
level on a per-agent basis. This can also be done after having reduced the
model-wide log level (as explained in section
Change the logging level above).

For example, let's enable 'TRACE' logging level to a unit of MySQL. We begin by
logging in to the unit's machine ('0' in this example):

juju ssh mysql/0

File /var/lib/juju/agents/unit-mysql-0/agent.conf is then edited by adding
the line LOGGING_OVERRIDE: juju=trace to the 'values' section.

File logsink.log contains logs for all models managed by the controller. Its
contents get sent to the database where it is consumed by the debug-log
command.

Note: In a High availability scenario, logsink.log is not guaranteed to contain all messages since agents have a choice of several controllers to send their logs to. The debug-log command should be used for accessing consolidated data across all controllers.

Remote logging

On a per-model basis log messages can optionally be forwarded to a remote
syslog server over a secure TLS connection.

See Rsyslog documentation for help with
security-related files (certificates, keys) and the configuration of the
remote syslog server.

Configuring

Remote logging is configured during the controller-creation step by supplying
a YAML format configuration file:

Audit logging

Juju audit logging provides a chronological account of all events by capturing
invoked user commands. These logs reside on the controller involved in the
transmission of commands affecting the Juju client, Juju machines, and the
controller itself.

The audit log filename is /var/log/juju/audit.log and contains records which
are either:

a Conversation, a collection of API methods associated with a single
top-level CLI command