This article is focused on providing developer guidance on building application logging mechanisms. Many systems enable operating system, web server, mail server and database server logging, but often custom application event logging is missing, disabled or poorly configured. It provides much greater insight than infrastructure logging alone. Web application (e.g. web site or web service) logging is much more than having web server logs enabled.

+

This article is focused on providing developers with concentrated guidance on building application logging mechanisms, especially related to security logging. Many systems enable network device, operating system, web server, mail server and database server logging, but often custom application event logging is missing, disabled or poorly configured. It provides much greater insight than infrastructure logging alone. Web application (e.g. web site or web service) logging is much more than having web server logs enabled (e.g. using Extended Log File Format).

−

Application logging should be consistent within the application, consistent across an organization's application portfolio and use industry standards where relevent, so the logged event data can be consumed, correlated, analyzed and managed by a wide variety of systems.

+

Application logging should be consistent within the application, consistent across an organization's application portfolio and use industry standards where relevant, so the logged event data can be consumed, correlated, analyzed and managed by a wide variety of systems.

= Purpose =

= Purpose =

−

Application logging should be always be included for security events. Application logs are invaluable data for identifying security incidents, monitoring policy violations, establishing baselines, providing information about problems and unusual conditions, contributing additional application-specific data for incident investigation which is lacking in other log sources, and helping defend against vulnerability identification and exploitation through attack detection. Application logging might also be used to record other types of events too. Thus application logging could include:

+

Application logging should be always be included for security events. Application logs are invaluable data for:

+

+

* Identifying security incidents

+

* Monitoring policy violations

+

* Establishing baselines

+

* Providing information about problems and unusual conditions

+

* Contributing additional application-specific data for incident investigation which is lacking in other log sources

Process monitoring, audit and transaction logs/trails etc are usually collected for different purposes than security event logging, and this often means they should be kept separate. The types of events and details collected will tend to be different. For example a PCIDSS audit log will contain a chronological record of activities to provide an independently verifiable trail that permits reconstruction, review and examination to determine the original sequence of attributable transactions. It is important not to log too much, or too little. Use knowledge of the intended purposes to guide what, when and how much. The remainder of this cheat sheet primarily discusses security event logging.

Line 25:

Line 35:

== Event data sources ==

== Event data sources ==

−

The application itself has access to a wide range of information events that should be used to generate log entries. Thus, the primary event data source is the application code itself. Other sources that could also be considered are:

+

The application itself has access to a wide range of information events that should be used to generate log entries. Thus, the primary event data source is the application code itself. The application has the most information about the user (e.g. identity, roles, permissions) and the context of the event (target, action, outcomes), and often this data is not available to either infrastructure devices, or even closely-related applications.

+

+

Other sources of information about application usage that could also be considered are:

−

* Client software e.g. actions on desktop software and mobile devices in local logs or using messaging technologies, web browser such as using Content Security Policy (CSP) reporting mechanism

The degree of confidence in the event information has to be considered when including event data from systems in a different trust zone. Data may be missing, modified, forged and could be malicious – it must always be treated as untrusted data. Consider how the source can be verified, and how integrity and non-repudiation enforced.

+

The degree of confidence in the event information has to be considered when including event data from systems in a different trust zone. Data may be missing, modified, forged, replayed and could be malicious – it must always be treated as untrusted data. Consider how the source can be verified, and how integrity and non-repudiation can be enforced.

−

+

−

Note that the application itself has the most information about the user (e.g. identity, roles, permissions) and the context of the event (target, action, outcome), and often this data is not available to infrastructure devices.

+

== Where to record event data ==

== Where to record event data ==

−

Applications most commonly write event log data to the file system or a database (SQL or NoSQL). Applications installed on desktops and on mobile devices may use local storage and local databases. All types of applications may send event data to remote systems (instead of or as well as more local storage). This could be a centralized log collection and management system (e.g. SIEM or SEM) or another application elsewhere.

+

Applications most commonly write event log data to the file system or a database (SQL or NoSQL). Applications installed on desktops and on mobile devices may use local storage and local databases. Your selected framework may limit the available choices. All types of applications may send event data to remote systems (instead of or as well as more local storage). This could be a centralized log collection and management system (e.g. SIEM or SEM) or another application elsewhere.

* When using the file system, it is preferable to use a separate partition than those used by the operating system, other application files and user generated content

* When using the file system, it is preferable to use a separate partition than those used by the operating system, other application files and user generated content

** For file-based logs, apply strict permissions concerning which users can access the directories, and the permissions of files within the directories

** For file-based logs, apply strict permissions concerning which users can access the directories, and the permissions of files within the directories

−

** In web applications, the logs must never be exposed in web-accessible locations

+

** In web applications, the logs should not be exposed in web-accessible locations, and if done so, should have restricted access and be configured with a plain text MIME type (not HTML)

* When using a database, it is preferable to utilize a separate database account that is only used for writing log data and which has very restrictive database , table, function and command permissions

* When using a database, it is preferable to utilize a separate database account that is only used for writing log data and which has very restrictive database , table, function and command permissions

−

* Use standard formats over secure protocols to send event data, or log files, to other systems

+

* Use standard formats over secure protocols to record and send event data, or log files, to other systems e.g. Common Log File System (CLFS), Common Event Format (CEF) over syslog, possibly Common Event Expression (CEE) in future

Consider separate files/tables for extended event information such as error stack traces or a record of HTTP request and response headers and bodies.

Consider separate files/tables for extended event information such as error stack traces or a record of HTTP request and response headers and bodies.

Line 53:

Line 65:

== Which events to log ==

== Which events to log ==

−

The level and content of security monitoring, alerting and reporting needs to be set during the requirements and design stage of projects, and should be proportional to the information security risks. This can then be used to define what should be logged. There is no one size fits all solution, and a blind checklist approach can lead to unnecessary "alarm fog" that means real problems go undetected. Always log:

+

The level and content of security monitoring, alerting and reporting needs to be set during the requirements and design stage of projects, and should be proportionate to the information security risks. This can then be used to define what should be logged. There is no one size fits all solution, and a blind checklist approach can lead to unnecessary "alarm fog" that means real problems go undetected. Where possible, always log:

Optionally consider if the following events can be logged and whether it is desirable information:

Optionally consider if the following events can be logged and whether it is desirable information:

Line 75:

Line 87:

* Application code file and/or memory changes

* Application code file and/or memory changes

−

== Event properties ==

+

== Event attributes ==

−

Each log entry needs to include sufficient information for the intended subsequent monitoring and analysis. The application logs must record for each event:

+

Each log entry needs to include sufficient information for the intended subsequent monitoring and analysis. The application logs must record "when, where, who and what" for each event. The properties for these will be different depending on the architecture, class of application and host system/device, but often include the following:

−

* Log date and time

+

* When

−

* Event date and time - the event time stamp may be different to the time of logging e.g. server logging where the client application is hosted on remote device that is only periodically or intermittently online

+

** Log date and time (international format)

−

* Interaction identifier [Note A]

+

** Event date and time - the event time stamp may be different to the time of logging e.g. server logging where the client application is hosted on remote device that is only periodically or intermittently online

−

* Application identifier e.g. name and version

+

** Interaction identifier [Note A]

−

* Application address e.g. host name and port, or server IPv4 or IPv6 address, or local device identifier

+

* Where

−

* Service e.g. name and protocol

+

** Application identifier e.g. name and version

−

* Window/form/page e.g. entry point URL and HTTP method for a web application, dialogue box name

For more information on these, see the "other" related articles listed at the end, especially the comprehensive article by Anton Chuvakin and Gunnar Peterson.

Note A: The "Interaction identifier" is a method of linking all (relevant) events for a single user interaction (e.g. desktop application form submission, web page request, mobile app button click, web service call). The application knows all these events relate to the same interaction, and this should be recorded instead of losing the information and forcing subsequent correlation techniques to re-construct the separate events. For example a single SOAP request may have multiple input validation failures and they may span a small range of times. As another example, an output validation failure may occur much later than the input submission for a long-running "saga request" submitted by the application to a database server.

Note A: The "Interaction identifier" is a method of linking all (relevant) events for a single user interaction (e.g. desktop application form submission, web page request, mobile app button click, web service call). The application knows all these events relate to the same interaction, and this should be recorded instead of losing the information and forcing subsequent correlation techniques to re-construct the separate events. For example a single SOAP request may have multiple input validation failures and they may span a small range of times. As another example, an output validation failure may occur much later than the input submission for a long-running "saga request" submitted by the application to a database server.

Line 111:

Line 134:

== Data to exclude ==

== Data to exclude ==

+

+

Never log data unless it is legally sanctioned. For example intercepting some communications, monitoring employees, and collecting some data without consent may all be illegal.

+

+

Never exclude any events from "known" users such as other internal systems, "trusted" third parties, search engine robots, uptime/process and other remote monitoring systems, pen testers, auditors. However, you may want to include a classification flag for each of these in the recorded data.

The following should not usually be recorded directly in the logs, but instead should be removed, masked, sanitized, hashed or encrypted:

The following should not usually be recorded directly in the logs, but instead should be removed, masked, sanitized, hashed or encrypted:

In some systems, sanitization can be undertaken post log collection, and prior to log display.

In some systems, sanitization can be undertaken post log collection, and prior to log display.

Line 145:

Line 174:

== Event collection ==

== Event collection ==

−

Implement an application-wide log handler which can be called from other modules/components. Document the interface referencing the organisation-specific event classification and description syntax requirements. If possible create this log handler as a standard module that can is thoroughly tested, deployed in multiple application, and added to a list of approved & recommended modules.

+

If your development framework supports suitable logging mechanisms use, or build upon that. Otherwise, implement an application-wide log handler which can be called from other modules/components. Document the interface referencing the organisation-specific event classification and description syntax requirements. If possible create this log handler as a standard module that can is thoroughly tested, deployed in multiple application, and added to a list of approved & recommended modules.

* Perform input validation on event data from other trust zones to ensure it is in the correct format (and consider alerting and not logging if there is an input validation failure)

* Perform input validation on event data from other trust zones to ensure it is in the correct format (and consider alerting and not logging if there is an input validation failure)

* Ensure failures in the logging processes/systems do not prevent the application from otherwise running

+

* Ensure failures in the logging processes/systems do not prevent the application from otherwise running or allow information leakage

* Synchronize time across all servers and devices [Note C]

* Synchronize time across all servers and devices [Note C]

Note C: This is not always possible where the application is running on a device under some other party's control (e.g. on an individual's mobile phone, on a remote customer's workstation which is on another corporate network). In these cases attempt to measure the time offset, or record a confidence level in the event time stamp.

Note C: This is not always possible where the application is running on a device under some other party's control (e.g. on an individual's mobile phone, on a remote customer's workstation which is on another corporate network). In these cases attempt to measure the time offset, or record a confidence level in the event time stamp.

−

== Testing ==

+

Where possible record data in a standard format, or at least ensure it can be exported/broadcast using an industry-standard format.

−

Logging functionality and systems must be including in application testing:

+

In some cases, events may be relayed or collected together in intermediate points. In the latter some data may be aggregated or summarized before forwarding on to a central repository and analysis system.

+

+

== Verification ==

+

+

Logging functionality and systems must be included in code review, application testing and security verification processes:

* Ensure the logging is working correctly and as specified

* Ensure the logging is working correctly and as specified

−

* Check events are being classified consistently and the field names, types and lengths are correctly defined to an agreed standard.

+

* Check events are being classified consistently and the field names, types and lengths are correctly defined to an agreed standard

* Check the effect on the logging mechanisms when external network connectivity is lost (if this is usually required)

* Ensure logging cannot be used to deplete system resources, for example by filling up disk space or exceeding database transaction log space, leading to denial of service

* Ensure logging cannot be used to deplete system resources, for example by filling up disk space or exceeding database transaction log space, leading to denial of service

+

* Test the effect on the application of logging failures such as simulated database connectivity loss, lack of file system space, missing write permissions to the file system, and runtime errors in the logging module itself

+

* Verify access controls on the event log data

+

* If log data is utilized in any action against users (e.g. blocking access, account lock-out), ensure this cannot be used to cause denial of service (DoS) of other users

* Brief the application/process owner about the application logging mechanisms

+

* Ensure the outputs of the monitoring (see below) are integrated with incident response processes

== Operation ==

== Operation ==

−

Enable processes to detect whether logging has stopped, and to identify tampering or unauthorized access and deletion.

+

Enable processes to detect whether logging has stopped, and to identify tampering or unauthorized access and deletion (see protection below).

== Protection ==

== Protection ==

−

The logging mechanisms and collected event data must be protected from mis-use such as tampering in transit, and unauthorized access, modification and deletion once stored. Logs may contain personal and other sensitive information, the data may contain information regarding the application's code and logic, and the aggregated information in the logs may itself have business value such as allowing the estimate of revenues, or proving performance information about employees. Consider whether parts of the data may need to be excluded, masked, sanitized, hashed or encrypted during examination or extraction.

+

The logging mechanisms and collected event data must be protected from mis-use such as tampering in transit, and unauthorized access, modification and deletion once stored. Logs may contain personal and other sensitive information, or the data may contain information regarding the application's code and logic. In addition, the collected information in the logs may itself have business value (to competitors, gossip-mongers, journalists and activists) such as allowing the estimate of revenues, or providing performance information about employees. This data may be held on end devices, at intermediate points, in centralized repositories and in archives and backups. Consider whether parts of the data may need to be excluded, masked, sanitized, hashed or encrypted during examination or extraction.

At rest:

At rest:

Line 184:

Line 226:

* Store or copy log data to read-only media as soon as possible

* Store or copy log data to read-only media as soon as possible

* All access to the logs must be recorded and monitored (and may need prior approval)

* All access to the logs must be recorded and monitored (and may need prior approval)

−

The privileges to read log data should be restricted and reviewed periodically

+

* The privileges to read log data should be restricted and reviewed periodically

In transit:

In transit:

Line 191:

Line 233:

* Consider whether the origin of the event data needs to be verified

* Consider whether the origin of the event data needs to be verified

* Perform due diligence checks (regulatory and security) before sending event data to third parties

* Perform due diligence checks (regulatory and security) before sending event data to third parties

+

+

See NIST SP 800-92 Guide to Computer Security Log Management for more guidance.

== Monitoring of events ==

== Monitoring of events ==

−

The logged event data needs to have appropriate monitoring, alerting and reporting:

+

The logged event data needs to be available to review and there are processes in place for appropriate monitoring, alerting and reporting:

−

* Ensure the application/process owner and incident response team is aware of the application logging mechanisms

* Enable alerting and signal the responsible teams about more serious events immediately

== Disposal of logs ==

== Disposal of logs ==

−

Log data, and backups/copies/extractions, must not be destroyed before the duration of the required data retention period, and must not be kept beyond this time. Legal, regulatory and contractual obligations may impact on these periods.

+

Log data, temporary debug logs, and backups/copies/extractions, must not be destroyed before the duration of the required data retention period, and must not be kept beyond this time. Legal, regulatory and contractual obligations may impact on these periods.

Introduction

This article is focused on providing developers with concentrated guidance on building application logging mechanisms, especially related to security logging. Many systems enable network device, operating system, web server, mail server and database server logging, but often custom application event logging is missing, disabled or poorly configured. It provides much greater insight than infrastructure logging alone. Web application (e.g. web site or web service) logging is much more than having web server logs enabled (e.g. using Extended Log File Format).

Application logging should be consistent within the application, consistent across an organization's application portfolio and use industry standards where relevant, so the logged event data can be consumed, correlated, analyzed and managed by a wide variety of systems.

Purpose

Application logging should be always be included for security events. Application logs are invaluable data for:

Identifying security incidents

Monitoring policy violations

Establishing baselines

Providing information about problems and unusual conditions

Contributing additional application-specific data for incident investigation which is lacking in other log sources

Helping defend against vulnerability identification and exploitation through attack detection

Application logging might also be used to record other types of events too such as:

Process monitoring, audit and transaction logs/trails etc are usually collected for different purposes than security event logging, and this often means they should be kept separate. The types of events and details collected will tend to be different. For example a PCIDSS audit log will contain a chronological record of activities to provide an independently verifiable trail that permits reconstruction, review and examination to determine the original sequence of attributable transactions. It is important not to log too much, or too little. Use knowledge of the intended purposes to guide what, when and how much. The remainder of this cheat sheet primarily discusses security event logging.

Design, implementation and testing

Event data sources

The application itself has access to a wide range of information events that should be used to generate log entries. Thus, the primary event data source is the application code itself. The application has the most information about the user (e.g. identity, roles, permissions) and the context of the event (target, action, outcomes), and often this data is not available to either infrastructure devices, or even closely-related applications.

Other sources of information about application usage that could also be considered are:

The degree of confidence in the event information has to be considered when including event data from systems in a different trust zone. Data may be missing, modified, forged, replayed and could be malicious – it must always be treated as untrusted data. Consider how the source can be verified, and how integrity and non-repudiation can be enforced.

Where to record event data

Applications most commonly write event log data to the file system or a database (SQL or NoSQL). Applications installed on desktops and on mobile devices may use local storage and local databases. Your selected framework may limit the available choices. All types of applications may send event data to remote systems (instead of or as well as more local storage). This could be a centralized log collection and management system (e.g. SIEM or SEM) or another application elsewhere.

When using the file system, it is preferable to use a separate partition than those used by the operating system, other application files and user generated content

For file-based logs, apply strict permissions concerning which users can access the directories, and the permissions of files within the directories

In web applications, the logs should not be exposed in web-accessible locations, and if done so, should have restricted access and be configured with a plain text MIME type (not HTML)

When using a database, it is preferable to utilize a separate database account that is only used for writing log data and which has very restrictive database , table, function and command permissions

Use standard formats over secure protocols to record and send event data, or log files, to other systems e.g. Common Log File System (CLFS), Common Event Format (CEF) over syslog, possibly Common Event Expression (CEE) in future

Consider separate files/tables for extended event information such as error stack traces or a record of HTTP request and response headers and bodies.

Which events to log

The level and content of security monitoring, alerting and reporting needs to be set during the requirements and design stage of projects, and should be proportionate to the information security risks. This can then be used to define what should be logged. There is no one size fits all solution, and a blind checklist approach can lead to unnecessary "alarm fog" that means real problems go undetected. Where possible, always log:

Optionally consider if the following events can be logged and whether it is desirable information:

Sequencing failure

Excessive use

Data changes

Fraud and other criminal activities

Suspicious, unacceptable or unexpected behavior

Modifications to configuration

Application code file and/or memory changes

Event attributes

Each log entry needs to include sufficient information for the intended subsequent monitoring and analysis. The application logs must record "when, where, who and what" for each event. The properties for these will be different depending on the architecture, class of application and host system/device, but often include the following:

When

Log date and time (international format)

Event date and time - the event time stamp may be different to the time of logging e.g. server logging where the client application is hosted on remote device that is only periodically or intermittently online

For more information on these, see the "other" related articles listed at the end, especially the comprehensive article by Anton Chuvakin and Gunnar Peterson.

Note A: The "Interaction identifier" is a method of linking all (relevant) events for a single user interaction (e.g. desktop application form submission, web page request, mobile app button click, web service call). The application knows all these events relate to the same interaction, and this should be recorded instead of losing the information and forcing subsequent correlation techniques to re-construct the separate events. For example a single SOAP request may have multiple input validation failures and they may span a small range of times. As another example, an output validation failure may occur much later than the input submission for a long-running "saga request" submitted by the application to a database server.

Note B: Each organisation should ensure it has a consistent, and documented, approach to classification of events (type, confidence, severity), the syntax of descriptions, and field lengths & data types including the format used for dates/times.

Data to exclude

Never log data unless it is legally sanctioned. For example intercepting some communications, monitoring employees, and collecting some data without consent may all be illegal.

Never exclude any events from "known" users such as other internal systems, "trusted" third parties, search engine robots, uptime/process and other remote monitoring systems, pen testers, auditors. However, you may want to include a classification flag for each of these in the recorded data.

The following should not usually be recorded directly in the logs, but instead should be removed, masked, sanitized, hashed or encrypted:

In some systems, sanitization can be undertaken post log collection, and prior to log display.

Customizable logging

It may be desirable to be able to alter the level of logging (type of events based on severity or threat level, amount of detail recorded). If this is implemented, ensure that:

The default level must provide sufficient detail for business needs

It should not be possible to completely inactivate application logging or logging of events that are necessary for compliance requirements

Alterations to the level/extent of logging must be intrinsic to the application (e.g. undertaken automatically by the application based on an approved algorithm) or follow change management processes (e.g. changes to configuration data, modification of source code)

The logging level must be verified periodically

Event collection

If your development framework supports suitable logging mechanisms use, or build upon that. Otherwise, implement an application-wide log handler which can be called from other modules/components. Document the interface referencing the organisation-specific event classification and description syntax requirements. If possible create this log handler as a standard module that can is thoroughly tested, deployed in multiple application, and added to a list of approved & recommended modules.

Perform input validation on event data from other trust zones to ensure it is in the correct format (and consider alerting and not logging if there is an input validation failure)

If writing to databases, read, understand and apply the SQL injection cheat sheet

Ensure failures in the logging processes/systems do not prevent the application from otherwise running or allow information leakage

Synchronize time across all servers and devices [Note C]

Note C: This is not always possible where the application is running on a device under some other party's control (e.g. on an individual's mobile phone, on a remote customer's workstation which is on another corporate network). In these cases attempt to measure the time offset, or record a confidence level in the event time stamp.

Where possible record data in a standard format, or at least ensure it can be exported/broadcast using an industry-standard format.

In some cases, events may be relayed or collected together in intermediate points. In the latter some data may be aggregated or summarized before forwarding on to a central repository and analysis system.

Verification

Logging functionality and systems must be included in code review, application testing and security verification processes:

Ensure the logging is working correctly and as specified

Check events are being classified consistently and the field names, types and lengths are correctly defined to an agreed standard

Check the effect on the logging mechanisms when external network connectivity is lost (if this is usually required)

Ensure logging cannot be used to deplete system resources, for example by filling up disk space or exceeding database transaction log space, leading to denial of service

Test the effect on the application of logging failures such as simulated database connectivity loss, lack of file system space, missing write permissions to the file system, and runtime errors in the logging module itself

Verify access controls on the event log data

If log data is utilized in any action against users (e.g. blocking access, account lock-out), ensure this cannot be used to cause denial of service (DoS) of other users

Deployment and operation

Release

Brief the application/process owner about the application logging mechanisms

Ensure the outputs of the monitoring (see below) are integrated with incident response processes

Operation

Enable processes to detect whether logging has stopped, and to identify tampering or unauthorized access and deletion (see protection below).

Protection

The logging mechanisms and collected event data must be protected from mis-use such as tampering in transit, and unauthorized access, modification and deletion once stored. Logs may contain personal and other sensitive information, or the data may contain information regarding the application's code and logic. In addition, the collected information in the logs may itself have business value (to competitors, gossip-mongers, journalists and activists) such as allowing the estimate of revenues, or providing performance information about employees. This data may be held on end devices, at intermediate points, in centralized repositories and in archives and backups. Consider whether parts of the data may need to be excluded, masked, sanitized, hashed or encrypted during examination or extraction.

At rest:

Build in tamper detection so you know if a record has been modified or deleted

Store or copy log data to read-only media as soon as possible

All access to the logs must be recorded and monitored (and may need prior approval)

The privileges to read log data should be restricted and reviewed periodically

In transit:

If log data is sent over untrusted networks (e.g. for collection, for dispatch elsewhere, for analysis, for reporting), use a secure transmission protocol

Consider whether the origin of the event data needs to be verified

Perform due diligence checks (regulatory and security) before sending event data to third parties

See NIST SP 800-92 Guide to Computer Security Log Management for more guidance.

Monitoring of events

The logged event data needs to be available to review and there are processes in place for appropriate monitoring, alerting and reporting:

Enable alerting and signal the responsible teams about more serious events immediately

Disposal of logs

Log data, temporary debug logs, and backups/copies/extractions, must not be destroyed before the duration of the required data retention period, and must not be kept beyond this time. Legal, regulatory and contractual obligations may impact on these periods.