Produce and deliver reports on system utilization (processor, memory, disk, and network)

Use shell scripting to automate system maintenance tasks

Configure a system to log to a remote system

Configure a system to accept logging from a remote system

How do you know when something goes wrong on your system? You look at the logs, of course! In all seriousness, understanding system logging is important so that you can troubleshoot when something goes wrong. Logs are kept for almost everything that you can think of, including the kernel, boot process, and different services that are running. Aside from logs, there are also utilities you can use to monitor and check the status of your system. This chapter looks at how you can use these tools and logging to troubleshoot anything that is thrown your way. The chapter also touches on how to automate some of these tasks.

Working with Syslog

Whenever something goes wrong—and sometimes when things go right—on the system, a message is generated by the syslog service. These messages are used to keep track of how the system is performing, what actions are taking place, and if there is a problem with something. Red Hat comes with a syslog service built in and already set to start at boot. In RHEL6, the rsyslog package is installed by default and is the primary logging service used.

Migration Tip

In RHEL5, the default logging service was split between two daemons: syslogd and klogd. Syslogd was used for system messages and klogd for kernel messages. In RHEL6, these have both been replaced with the rsyslogd daemon, which handles both types of messages. If you are still on RHEL5, you can install the syslog service via the sysklogd package.

The rsyslog service has a main config file, /etc/rsyslog.conf, that controls where messages are sent when they are generated. Because logging is a critical piece of the operating system, it is installed by default as part of the core packages.

Step 1. Just out of good habit, you should verify that the package is installed:

# rpm -qa | grep syslog
rsyslog-4.6.2-2.el6.x86_64

Because you can see that it is installed, you should also check on two other issues.

Step 2. First, make sure that the service is set to start when the system boots:

As you can see here, there are some slight differences between logging on RHEL5 and RHEL6 (such as the naming and single versus double daemons). The rsyslog service provides better logging features, so it is recommended to upgrade if possible.

All the messages sent to the rsyslog service are stored in the /var/log directory, and each message section has its own file or subdirectory. You can also define your own log files or directories. The biggest problem that arises from using any logging service is the files can become uncontrollable if left unchecked. Being able to log different alerts and to obtain information from logs, in my opinion, is an art all unto itself. There are so many ways to customize logs, and each network setup is different. You should know that there are nine different alerts that the syslog service uses. These alerts can be produced by different network devices and Red Hat systems.

You can see from the output where some of the files are stored. Also shown is a sample rule.

Log Rotation

If left alone, logs can grow to enormous size. Luckily for you, the logrotate command allows you to rotate logs before they become too big. By default, the logrotate command is called daily to cycle the log files into a new file. The parameters of the command are defined in its config file, /etc/logrotate.conf, and the /etc/logrotate.d directory. However, the command itself is called from /etc/cron.daily/logrotate, a cron job that is run daily (as already mentioned). You can always call the logrotate command yourself if you'd like to rotate your logs, but don't do this too frequently because you'll end up with tons of rotated logs and you'll be right back to square one with a mess of unorganized log files.

Syntax: logrotate [options] configfile

Options:

-d

Debugs; doesn't do anything but test

-f

Forces file rotation

-v

Provides verbose output

The following example rotates the current logs as defined in the config file:

# logrotate /etc/logrotate.conf

Centralized Logging

If you'd like to set up a centralized syslog server, you need to choose which server you'd like to store all your logs on. For this example, you can use the RHEL01 system as the centralized server and configure it to receive logs from the other systems on the network.

In the modules section, you can see two areas that will provide the capability to receive logs from other systems. The standard is to use the UDP protocol on port 514. By uncommenting the section related to UDP reception, you essentially can provide a way for the syslog server to become the centralized server for the network.

Step 2. Uncomment the following two lines:

$ModLoad imudp.so
$UDPServerRun 514

Step 3. When you're finished, you need to save the file, exit, and restart the syslog service:

The firewall rules might not make sense at the moment because we haven't covered iptables or firewall rules yet. After you read Chapter 12, "System Security," return here to make sure you understand what is happening in the creation of firewall rules.

Migration Tip

Unfortunately, due to the revamp of the syslog service, centralized logging for RHEL5 is quite different. On RHEL5, the syslog config file doesn't contain a modules section to enable remote management. Instead, you need to edit the daemon config file. We cover centralized logging for RHEL5 in the next section.

Now that the security requirements have been addressed and the centralized server is configured, you can shift your attention to the client systems. Let's look at RHEL02 for the client system. You can config RHEL02 to send some or all of its log files to the centralized syslog server (RHEL01).

You can use the following syntax when configuring log files to be sent to the centralized server:

<log file> @<hostname or IP of system (local or remote)>

You can edit the RHEL02 system to point all security logs to the central server:

With the syslog service now accepting remote logs, you need to create a firewall rule. This process is the same as on RHEL6. One last modification that needs to be made on RHEL5 is the adjustment of some SELinux protections.

Although we haven't covered SELinux yet (it is covered in Chapter 11, "SELinux"), the Boolean values are required here if you are still using RHEL5. After you read through Chapter 11, you can refer back here to make sure you understand the changes being made.

User Login Events

Aside from the normal logs generated and used by the syslog service, there are two special commands that deal with system logins. These two commands have special logs that can be read only through the use of these commands:

lastlog

Lists login records

faillog

Lists failed login attempts

You can use these two commands for viewing login events. These two commands are useful for hunting down attacks where users are trying to force themselves into other users' accounts (called a brute-force attack).

The ability to work with syslog is important to being able to help troubleshoot any issue on Linux, not just Red Hat. When you're trying to find an issue or resolve one, the log files should always be your first stop.