This topic describes the Data Retention tab for an Archiver. Administrators use this tab to define the criteria for log retention and storage.

On the Administration > Services > Config view > Data Retention tab of an Archiver, Administrators can define the criteria for log retention and storage. As an Administrator, you can configure hot, warm, and cold storage as well as multiple storage collections with different locations and criteria for retaining logs. For example, you can create a Compliance collection that stores logs for a specific time period as required by government regulations. You can create another collection that stores low value logs in hot storage with a much shorter retention period. The flexibility of these collections enables you to have significantly less overall storage requirements.

Total Hot Storage: Enables you to configure the total amount of Hot Tier storage available. You can select or add mount points (paths) for your Hot Tier storage locations. These mount points are attached to fast direct storage, such as Direct-Attached Capacity (DAC) storage and SAN.

Total Warm Storage: (Optional) Enables you configure the total amount of Warm Tier storage available. You can select or add mount points for your Warm Tier storage locations. These mount points are attached to secondary storage, such as NAS.

Total Cold Storage: (Optional) Enables you to configure the total amount of Cold Tier storage available. You can add a mount point for a Cold Tier storage location to back up your log files. This mount point is attached to offline storage, such as NAS, or temporary storage before archiving to tape. Security Analytics does not manage cold storage.

Collections: Enables you to define individual storage collections for different log types. You can specify the maximum size of the Hot and Warm Storage space, whether to use offline storage (Cold Storage), the number of days to retain the logs in the collection, the data compression, and whether to use a hash algorithm to ensure the data integrity of the files being saved.

Retention Rule: Enables you to define rules for each of your log storage collections. You must define at least one rule for each collection.

To access the Data Retention tab for an Archiver:

In the Security Analytics menu, select Administration > Services.

Select an Archiver service and >View > Config.

In the Services Config view for the service, click the Data Retention tab.The Data Retention tab for the Archiver is displayed.

Total Hot, Warm, and Cold Storage

The Total Hot Storage section shows the total amount of Hot storage available and the number of hot storage mount points. The Total Hot Storage section shows the total amount of Warm storage available and the number of warm storage mount points. The Total Cold Storage section shows the total amount of Cold storage and the remaining free space available in Cold storage.

Hot, Warm, and Cold Storage Mount Points Dialogs

In the Hot, Warm, and Cold Storage Mount Points dialogs, you can specify the mount points for your storage locations. You can specify portions of this storage to use for your log storage collections.

To access the Hot, Warm, and Cold Storage Mount Points dialogs, click the icon near the respective section.

The following table describes features of the Hot, Warm, and Cold Tier Storage dialogs.

Feature

Description

Adds a mount point.

Removes a mount point. You cannot delete a mount point that is in use unless you delete the associated collections.

Select the mount points that you want to include for the Total Hot, Warm, and Cold Storage. You can only select one mount point for Total Cold Storage.

Mount Point

Shows the path to the attached physical storage. For example: /var/netwitness/archiver/database0, which is the location of the hot storage DAC. Do not add collections or subdirectories to the mount points. Security Analytics will automatically create metadb, packetdb, sessiondb, and index directories for each collection defined on the Archiver: <storageLocation>/<CollectionName>/metadb <storageLocation>/<CollectionName>/packetdb <storageLocation>/<CollectionName>/sessiondb <storageLocation>/<CollectionName>/index

For example, if your hot storage mount point is /var/netwitness/archiver, then the following directories will be created for each of your collections: /var/netwitness/archiver/<CollectionName>/metadb /var/netwitness/archiver/<CollectionName>/packetdb /var/netwitness/archiver/<CollectionName>/sessiondb /var/netwitness/archiver/<CollectionName>/index

For Cold Storage, you must include the collection name format specifier %n somewhere in the cold storage mount point path name to avoid filename collisions between collections.

Storage Size

Shows the size of the attached storage. The Data Retention tab shows the total amount of storage for your reference.

Collections

The Collections section lists all of your storage collections along with Total Storage for Hot and Warm Storage.

The following table describes the features of the Collections section. You can hide some of the columns based on your requirements.

Selects a collection. For example, you can select a collection for editing or removal.

Collection

Shows the name of your collection, such as Default, Compliance, MediumValue, and LowValue. You can create multiple collections with different criteria for retaining logs. If you do not create any collections, the Default collection is used.

If a collection has errors, the collection name and the columns with errors appear in red text.

Usage / Hot Storage

Shows the current hot storage usage and the maximum hot storage for the collection. When the size of the logs reach the maximum hot storage amount, the logs are removed or they roll to the next available storage tier (warm or cold).

Usage / Warm Storage

Shows the current warm storage usage and the maximum warm storage for the collection. When the size of the logs reach the maximum warm storage amount, the logs are removed or they roll to available cold storage.

Shows the number of days that logs are retained before being removed or optionally moved to cold storage. No Limit indicates that log retention is not restricted by a specified number of days.For Hot and Warm Storage, size and retention period settings for a collection can override each other based on which criterion (size or time) is satisfied first.

Velocity (last hour)

Shows the number of logs captured over the last hour.

Oldest Date

Shows the date and time of the last log capture.

Duration

Shows how may days ago that the last log was captured. For example: 20 days.

Compression

Shows the compression type used for the meta and raw data in the collection.

Hash

Shows whether hash is enabled or disabled. When enabled, the hash algorithm is used to ensure the data integrity of the files being saved. By default, the only data being hashed is raw logs and the hash files are saved in the same directory as data.

# of Rules

Shows the number of rules applied to the collection. Define at least one rule for each collection. A collection without any associated rules shows a zero in red text as a warning: The collection name also appears in red text, which indicates an error in the collection.

Caution: If a collection does not have a rule, no logs will ever go into that collection.

Actions

Enables you to see the rules associated with a collection in the Retention Rule section when you select <actions button> > Select Rules. In the Retention Rule section, you can change the overall priority of the collection rules.

Total Storage

Shows the current total hot storage usage and the maximum total hot storage at the bottom of the Usage / Hot Storage column. It also shows the current total warm storage usage and the maximum total warm storage at the bottom of the Usage / Warm Storage column.

Any errors in the collection appear in red text. A dotted underline indicates that a tooltip is available with information about the error.

Collections that have editing disabled (grayed out) also have tooltips that provide information on the problem.

Retention Rules

The Retention Rules section lists all of the retention rules used for your storage collections listed in the order of rule execution.

The following table describes the features of the Retention Rule section.

Moves the selected retention rule up in the Retention Rule priority list. Retention Rule order is very important. Security Analytics evaluates the the retention rules for all of the collections in numerical order by the number listed in the Order column in the Retention Rule section.

You can also use drag and drop to reorder retention rules.

Move Down

Moves the selected retention rule down in the Retention Rule priority list. Retention Rule order is very important. Security Analytics executes the the retention rules for all of the collections in numerical order by the number listed in the Order column in the Retention Rule section.

Apply

Saves the rule order change.

Revert

Reverts the rule order change.

Selects or shows a selected retention rule.

Order

Shows the order of a rule in the overall list of retention rules.

Rule Name

Shows the name of rule, such as ComplianceDevices and GeneralWindowsLogs.

Condition

Shows the conditions for the rule. These conditions specify the type of logs to include in the collection. Rule and Query Guidelines presents the guidelines for all queries and rule conditions in Security Analytics Core services.

Collection

Shows Collection name and how many days that the collection is retained. For example: MediumValue (30 Days)