Log and trace collection

Overview

IBM Containers for Bluemix provides some built-in logging and monitoring capabilites around container CPU, memory, and networking. You can optionally change the log levels for your MobileFirst containers.

The option to create log files for the MobileFirst Server and MobileFirst Analytics containers is enabled by default (using level *=info). You can change the log levels by either adding a code override manually or by injecting code using a given script file. Both container logs and server or runtime logs can be viewed from a Bluemix logmet console by means of the Kibana visualization tool. Monitoring can be done from a Bluemix logmet console by means of Grafana, an open source metrics dashboard and graph editor.

When your MobileFirst container is created with a Secure Shell (SSH) key and bound to a public IP address, a suitable private key can be used to securely view the logs for the container instance.

Logging overrides

You can change the log levels by either adding a code override manually or by injecting code using a given script file. Adding a code override manually to change the log level must be done when you are first preparing the image. You must add the new logging configuration to the package_root/mfpf-[analytics|server]/usr/config folder as a separate configuration snippet, which gets copied to the configDropins/overrides folder on the Liberty server.

Injecting code using a given script file to change the log level can be accomplished by using certain command-line arguments when running any of the start*.sh script files provided in the V8.0.0 package (startserver.sh, startanalytics.sh, startservergroup.sh, startanalyticsgroup.sh). The following optional command-line arguments are applicable:

[-tr|--trace] trace_specification

[-ml|--maxlog] maximum_number_of_log_files

[-ms|--maxlogsize] maximum_size_of_log_files

Container log files

Log files are generated for MobileFirst Server and Liberty Profile runtime activities for each container instance and can be found in the following locations:

/opt/ibm/wlp/usr/servers/mfp/logs/messages.log

/opt/ibm/wlp/usr/servers/mfp/logs/console.log

/opt/ibm/wlp/usr/servers/mfp/logs/trace.log

/opt/ibm/wlp/usr/servers/mfp/logs/ffdc/*

You can log in to the container by following the steps in Accessing log files and access the log files.

To persist log files, even after a container no longer exists, enable a volume. (Volume is not enabled by default.) Having volume enabled can also allow you to view the logs from Bluemix using the logmet interface (such as https://logmet.ng.bluemix.net/kibana).

Enabling volume
Volume allows for containers to persist log files. The volume for MobileFirst Server and container logs is not enabled by default.

You can enable volume when running the start*.sh scripts by setting ENABLE_VOLUME [-v | --volume] to Y. This is also configurable in the args/startserver.properties and args/startanalytics.properties files for interactive execution of the scripts.

The persisted log files are saved in the /var/log/rsyslog and /opt/ibm/wlp/usr/servers/mfp/logs folders in the container.
The logs can be accessed by issuing an SSH request to the container.

Accessing log files

Logs are created for each container instance. You can access log files using the IBM Container Cloud Service REST API, by using cf ic commands, or by using the Bluemix logmet console.

IBM Container Cloud Service REST API

For any container instance, the docker.log and /var/log/rsyslog/syslog can be viewed using the Bluemix logmet service. The log activities can be seen using the Kibana dashboard of the same.

Enabling SSH

To enable SSH, copy the SSH public key to the package_root/[mfpf-server or mfpf-analytics]/usr/ssh folder before you run the prepareserver.sh or the prepareanalytics.sh scripts. This builds the image with SSH enabled. Any container created from that particular image will have the SSH enabled.

If SSH is not enabled as part of the image customization, you can enable it for the container using the SSH_ENABLE and SSH_KEY arguments when executing the startserver.sh or startanalytics.sh scripts. You can optionally customize the related script .properties files to include the key content.

The container logs endpoint gets stdout logs with the given ID of the container instance.

Example: GET /containers/{container_id}/logs

Accessing containers from the command line

You can access running MobileFirst Server and MobileFirst Analytics container instances from the command line to obtain logs and traces.

Create an interactive terminal within the container instance by running the following command: cf ic exec -it container_instance_id "bash".

Accessing containers using SSH

You can get the syslogs and Liberty logs by using Secure Shell (SSH) to access your MobileFirst Server and MobileFirst Analytics containers.

If you are running a container group, you can bind a public IP address to each instance and view the logs securely using SSH. To enable SSH, make sure to copy the SSH public key to the mfp-server\server\ssh folder before you run the startservergroup.sh script.

Make an SSH request to the container. Example: mylocal-workstation# ssh -i ~/ssh_key_directory/id_rsa root@public_ip