Finding Feature Information

Your software release may not support all the features documented in this module. For the latest caveats and feature information, see Bug Search Tool and the release notes for your platform and software release. To find information about the features documented in this module, and to see a list of the releases in which each feature is supported, see the feature information table at the end of this module.

Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/go/cfn. An account on Cisco.com is not required.

Cisco Data Collection Manager

Overview of Cisco Data Collection Manager

Cisco Data Collection Manager (DCM) is an efficient and reliable data collection agent that is embedded in managed devices, such as routers and switches. DCM works on a push model, which is based on a subscribe-and-notify data pattern, as opposed to the pull model, which is based on a request-and-response data pattern, in traditional SNMP-based network management.

Configuration and Deployment

The Bulkstat client application is implemented using the DCM core services to retrieve data and export it to the user. The Bulkstat client provides the only available user interface for DCM access. The client also provides CLI access through a new set of configuration commands and MIB access through the CISCO-DATA-COLLECTION-MIB. You can configure the data to be exported as a file or through NetFlow. Also, you can configure the data to be processed and the processed file to be collected by the NMS.

Data Collection

The Data Collection Manager (DCM) provides data subscription services for multiple data sources, such as the Simple Network Management Protocol (SNMP) MIB objects, the output of show commands, and Expression output. The DCM allows you to configure the data that needs to be collected. The DCM also allows you to specify when and at what interval the data should be collected.

Data Processing

The Data Collection Manager (DCM) supports advanced on-board data processing that includes baseline calculation, summary calculation, statistical distribution, and percentile computation. The DCM is integrated with CISCO-EXPRESSION-MIB to externally create and customize MIB objects for monitoring and to support the CLI to define expressions.

Data Export and Retrieval

The Data Collection Manager (DCM) provides data retrieval management to ensure that the data collection does not impact device resources. The DCM can export data in a file format using multiple export protocols such as FTP, TFTP, Secure copy protocol (SCP), and Secure File Transfer Protocol (SFTP). The DCM can also stream collected data as Internet Protocol Flow Information Export (IPFIX) records to facilitate management applications that already have the necessary infrastructure to receive data coming over NetFlow. The DCM provides a query mechanism with which data can be selectively exported based on the configured time interval and other selection criteria. The DCM application also provides data filtering services and exports the filtered data. You can also set primary and secondary destinations for exporting the collected data in a raw or processed format. Snapshots of the collected data can be stored for later retrieval.

Performance Management Solutions

The Data Collection Manager (DCM) can be used to manage various aspects of performance management. The following list provides a few scenarios.

Traffic Engineering

The primary goal of traffic engineering is to control traffic flow and provide Class of Service (CoS) and Quality of Service (QoS) to the end user, while using network resources optimally. It is the responsibility of the network management server (NMS) to provide low-level management through traffic conditioning, packet shaping, queue management, and other functions that regulate traffic flow through the network. The DCM can be used to collect data with a high granularity to help the NMS make dynamic traffic engineering decisions.

Capacity Planning and Trend Analysis

The key priorities for network operators today are acceleration in new service introduction, reduction in the complexity of deployment and management of services, and reduction in the capital or operational overhead expenditure. These services require unique capabilities that are specific to a particular network operator and the ability to provide specific service characteristics on a per-consumer basis. It is important to identify trends in the network traffic, forecast future traffic levels, assess whether the existing network capacity is sufficient to handle the projected load, and automatically redesign the network to support future traffic demands. The DCM can be used to collect resource variables that are important for effective capacity trend information, such as memory, queue depth, broadcast volume, buffer, Frame Relay congestion notification, and backplane utilization.

Diagnostics

The streaming function of the DCM can be used for real-time troubleshooting. Also, the DCM facilitates data retrieval for selective periods and aids in troubleshooting.

SLA Management

A service level agreement (SLA) includes a what-if analysis for network changes and application changes, baselining and trending for defined performance variables, exception management for defined capacity and performance variables, and QoS management. The DCM can be used to collect periodic data for reporting purposes.

Bulkstat

Two challenges that network providers usually face are data gathering and data analysis. Network providers need to gather large volumes of data to analyze the performance of the network and to have operational control over their network. Large service providers are strengthening their data gathering and analysis infrastructure. Traditionally, Simple Network Management Protocol (SNMP) agents are used to expose management data on managed systems. But, SNMP is not well suited for gathering large volumes of data, especially over short time intervals.

For example, service providers charge customers depending on the network usage. Also this data must be available on customer request. Accounting applications based on SNMP polling models consume significant network bandwidth because they poll large volumes of data frequently. The SNMP protocol data unit (PDU) is a complex data type specific to SNMP and is expensive to process because the SNMP objects and tables must be sorted in a lexicographic order. All the entries in SNMP MIB tables are lexicographically ordered by their object identifiers, because there is an implied ordering in the MIB based on the order of the object identifiers.

In such cases, the need to continuously poll large or bulk SNMP statistics can be avoided by using applications known as collectors to retrieve data.

The Bulkstat application is one such collector that uses the services of the Data Collection Manager (DCM) to provide the following functions:

Collecting SNMP MIB object values, the output of show commands, and the results of expression data.

Bulkstat Configuration Elements

Data Set

This section describes the data set elements that you can configure to collect Simple Network Management Protocol (SNMP) data, CLI data, and Expression MIB data. Only objects having the same index elements can be grouped in a single object list.

SNMP Data

The SNMP data set contains the following fields:

Table 1

SNMP Data Elements

Name

Description

Configuration Status

Objects

Specifies the object to be collected. Multiple objects can be configured to form a data set. The textual name of the object can be used for configuring an object. If the device does not recognize the textual name, the object identifier (OID) format can be used for configuring the name.

Mandatory

Object alias

Specifies the optional alias name that each object can have.

Optional

Note

Only objects having the same index elements can be grouped in a single data set. For example, the objects ifDescr and cieIfSpeedReceive belong to different tables, but they are indexed by ifIndex. Hence the two objects can be grouped in the same data set. ifDescr and entPhysicalDescr cannot be grouped in the same data set, because ifDescr is indexed by ifIndex and entPhysicalDescr is indexed by entPhysicalIndex.

Objects from tables having sparse dependency can also be grouped in the same data set. Similar SNMP data elements can be linked to multiple subscriptions. Each subscription can collect different entries for the objects based on the instance configuration and context configuration.

CLI Data

The CLI data set contains the following fields:

Table 2

CLI Data Elements

Name

Description

Configuration Status

CLI

Specifies the CLI command for which the show output needs to be collected. More than one CLI can be specified in the same data set.

Mandatory

Expression MIB Data

The Expression data set contains the following fields:

Table 3

Expression Data Elements

Name

Description

Configuration Status

Expression

Specifies the expression to be evaluated in the format specified by the EXPRESSION-MIB.

Mandatory

Object

Specifies the object configuration as specified by the EXPRESSION-MIB implementation.

Mandatory

Value

Specifies the type to which the resultant value should be typecasted.

Mandatory

Result process

Specifies whether the expression-computed result should be considered for processing. Result process also specifies whether the absolute value or the delta value should be used for processing.

Optional

Instance Set

This section specifies the instance set elements that you can configure to collect Simple Network Management Protocol (SNMP) data. More than one instance of the same type can be added to the set. Combinations of types of instance set elements are not supported.

The SNMP Instance set contains the following fields:

Table 4

SNMP Instance Elements

Name

Description

Configuration Status

Exact

Specifies the instance for which the data should be collected. More than one instance can be specified, but only fully qualified instances should be specified.

Optional

Wildcard

Specifies all instances for all objects under the object configured in the data set.

Optional

Range

Specifies the start and end instances. All instances within the range, including the start and end, are collected, but only fully qualified instances should be specified.

Optional

Repetition

Specifies the start of the repetition and the number of repetitions. All instances from the start until the number of repetitions within the subtree are collected.

Optional

Interface

Specifies the interface instead of the index. The ifIndex assigned to the interface will be used as an index. This can be used for MIB objects indexed by ifIndex.

Optional

Filter Set

This section describes the filter configuration per object.

The filter set elements that you can configure to collect Simple Network Management Protocol (SNMP) data are described here. More than one filter of the same type can be added to the set.

Table 5

Filter Elements

Name

Description

Status

Object match

Specifies the value to be used to match against the value retrieved for the object during collection. The value provided needs to match the type of the object. If there is an error in the type matching, the configuration is not accepted.

More than one value can be specified for an object, and more than one object can have matching values.

Specifies any one of the data types as defined in the topic "Data Set."

Mandatory

Instance

Specifies any one of the instance types as defined in the topic "Instance Set."

Optional, if not specified.

Default behavior of the instance set is wildcard.

Only applicable for SNMP.

Filter

Specifies any one of the filter types as defined in the topic "Filter Set."

Optional, if not specified.

Only applicable for SNMP.

Polling interval

Specifies the collection periodic interval in seconds. In case of recurring collection, the data is collected at the expiration of the collection interval until the collection is stopped.

Optional

Context

Specifies the management context from which to obtain data for this data group.

Optional

Process summary

Enables summary processing of the data marked to be processed in the corresponding data-set configuration.

Optional

Process distribution

Enables distribution processing of the data marked to be processed in the corresponding data-set configuration.

Optional

Process percentile

Enables percentile processing of the data marked to be processed in the corresponding data-set configuration.

Optional

Process auto-baseline

Enables auto-baselining processing of the data marked to be processed in the corresponding data-set configuration.

If auto-baseline process is enabled, the other processes, such as summary, distribution, and percentile configurations, if done previously, are removed because auto-baseline process uses these functionalities internally.

Note

Removing this configuration will not reinstate the other configurations that are removed.

Optional

Discard raw

Specifies whether to store raw data. If data is processed, the user can choose to store only process data by setting the option.

Optional

Data Profile

This section describes the data profile that is used to group multiple data groups. This is done to simplify the configuration and to aggregate data of similar nature. A data profile can have multiple data groups. A data group can have constraints in the data specified in the element. If two sets of data need to be written to the same file, the respective data groups should be linked as part of a single profile.

The data profile has the following fields:

Table 7

Data Profile Elements

Name

Description

Status

Data groups

Specifies the data group to be linked to this profile. Multiple data groups can be linked to a single profile.

Mandatory before activating a profile

Transfer interval

Specifies the transfer periodic interval in seconds. In case of recurring transfer, the data is transferred when the transfer interval expires.

Optional

Process interval

Specifies the process periodic interval in seconds. The data is processed during every collection interval as soon as it is collected. When the process interval expires, the processed data is written into a file and transferred.

Optional

Primary URL

Specifies the URL of the primary management station. The files containing the collected data are transferred to this URL when the transfer interval expires.

Mandatory

Secondary URL

Specifies the URL of the secondary management station to be used in case the transfer to the primary management station fails.

Optional

Schema

Specifies the file data format. The schemaASCII option is supported.

Optional

Retry

Specifies the number of times that the transfer is retried in case of transfer failures to both primary and secondary management stations. This command has an effect only if the retain command is configured in the profile. The retry interval is computed by dividing the retention time by the number of retries. For example, if the file is retained for 60 minutes and the retry is 6 times, the transfer is attempted every 10 minutes, until the transfer succeeds or the file is removed.

Optional

Flow exporter

Specifies the NetFlow exporter configured using the current NetFlow global configuration command flow exporter. Only those exporters that have opted for the Internet Protocol Flow Information Export (IPFIX) export protocol version are allowed.

Optional

Buffer-size

Specifies the maximum size to which the file containing the collected data can grow. When it reaches the limit, the file is closed and the transfer is attempted based on the transfer configuration associated with the data group or profile.

Optional

Retention memory

Specifies the time, in seconds, to retain the file in the memory.

Optional

Retention USB

Specifies the time, in seconds, to retain the file in the USB. This option is available only if the device supports the USB drive.

Optional

Calendar Scheduling

The Bulkstat application allows you to schedule each subscription for collection. A subscription can be scheduled for one-time collection or periodic collection. A periodic subscription can be repeated infinitely or for a specified number of repetitions. A timer is instantiated for every activated subscription.

Table 8

Calendar Scheduling Elements

Name

Description

Configuration Status

One shot

Specifies that the data is collected for a specified collection interval.

Optional

Recurring

Specifies that the data is collected regularly at the specified time, day, month, and for a specified collection interval.

Optional

SNMP Data Collection

The Data Collection Manager (DCM) is used to collect the Simple Network Management Protocol (SNMP) MIB Object data. This collection is supported through both Bulkstat CLI and CISCO-DATA-COLLECTION-MIB. This allows a network management server (NMS) application to configure a set of MIB objects and a set of instances whose values need to be collected on a periodic basis. You can configure the collected data to be periodically sent to the NMS through file export or NetFlow .

CLI Data Collection

You can configure the Data Collection Manager (DCM) module to collect show command output through the Bulkstat CLI. The DCM captures the raw show command output and periodically exports the data through the ASCII file format.

Expression Data Collection

The Data Collection Manager (DCM) can be configured to evaluate expressions as defined in CISCO-EXPRESSION-MIB and collect the results on a periodic basis. The DCM is also capable of periodically transferring the collected data in a file format.

Data Processing

Service providers rely on monitored SNMP data to alert network management systems (NMSs) of changing network conditions. By periodically monitoring the device data and comparing it against a set of thresholds, the network can automatically alert the operators, thereby allowing efficient operations.

Summary

You can enable summary processing on the collected object value and calculate minimum, maximum, and average values. A summary is calculated for only those objects that are marked as process capable in the data group and uses the absolute or delta value as per the object configuration.

Distribution

You can enable distribution processing on the collected object value by specifying the object type, minimum value, maximum value, and the number of buckets to distribute the value. Based on the configuration, counters are maintained per bucket and are incremented whenever the data falls into a bucket range.

Percentile

You can enable percentile processing on the collected object value. A percentile is calculated on every process interval expiry. Distribution configuration is mandatory to enable percentile processing. Percentile computation is done assuming that the distribution is normal.

Auto-baseline

You can enable baseline processing on the collected object value. The baseline internally uses all summary, distribution, and percentile calculations to provide baseline values. You can configure either baseline processing or other forms of processing, such as summary, distribution, and percentile calculations.

The auto-baseline feature in DCM calculates the baseline values for variables of interest on the device and allows network management applications or network operators to retrieve the baseline values. The baseline values can be displayed in terms of percentiles or a median with standard deviation.

File Data Export

The file data export feature on the Data Collection Manager (DCM) exports the collected data based on the transfer configurations. Data can be exported in various formats, and Bulkstat files are one such format to collect data. The format in which the data is inserted into the file conforms to the schemaAscii format described in CISCO-DATA-COLLECTION-MIB and CISCO-BULK-FILE-MIB. The data sequence in which the data is stored is determined based on the sequence in which the data is received.

The Cisco File Transfer module is responsible for transferring the files as per the transfer configuration. This module interfaces with the Cisco IOS IFS module to transfer the file to the specified URL. A file can be retained in the device whether the transfer was a success or a failure.

File names are created using the following format:

Raw data file name: <profile-name>_<host-name>_raw_<time when the file is created in "%y%m%d_%H%M%S%k">

Processed data file name: <profile-name>_<host-name>_proc_<time when the file is created in "%y%m%d_%H%M%S%k">

NetFlow Export

The NetFlow protocol is used extensively in network and application performance management to collect IP traffic information. Many enterprises today rely on NetFlow along with traditional Simple Network Management Protocol (SNMP) data. The benefits of having the Data Collection Manager (DCM) collect and export the SNMP data over NetFlow are as follows:

NetFlow export allows management vendors that are NetFlow centric to use the functionalities of the SNMP collection MIB without using the SNMP stack. This allows vendors to provide better visibility into customer networks without huge development efforts.

NetFlow export provides an efficient output. NetFlow export can eliminate the need to store data in the device for a long period of time. Also, continuous NetFlow export allows for real-time troubleshooting, where a network management system (NMS) can turn on high-granularity monitoring without a storage concern.

Configuring an SNMP Bulkstat Data Set

The first step in configuring the Simple Network Management Protocol (SNMP) periodic data collection and transfer mechanism is to configure one or more data sets. A data set is used to group objects of similar types, based on the data source. The data set is defined outside of the data group. This external definition gives the user the flexibility to use the same data set across multiple data groups and to collect the output for different instances and different contexts.

Note

All objects in an SNMP data set must be indexed by the same MIB index. However, the objects in the data set must not belong to the same MIB or the MIB table.

Perform this task to configure the SNMP Bulkstat data set.

SUMMARY STEPS

1.enable

2.configureterminal

3.bulkstatdatadata-set-nametypesnmp

4.objectoid[aliasalias-name]

5.end

DETAILED STEPS

Command or Action

Purpose

Step 1

enable

Example:

Device> enable

Enables privileged EXEC mode.

Enter your password if prompted.

Step 2

configureterminal

Example:

Device# configure terminal

Enters global configuration mode.

Step 3

bulkstatdatadata-set-nametypesnmp

Example:

Device(config)# bulkstat data interface-stats type snmp

Defines an SNMP Bulkstat data set and enters SNMP bulk statistics data set configuration mode. The creation of an SNMP Bulkstat data set creates a row in the cdcDGBaseObjectEntry table in the SNMP MIB.

Configuring an SNMP BulkStat Instance Set

The Simple Network Management Protocol (SNMP) instance set specifies the instances for which the data should be collected. Each subscription can collect different entries for specified objects based on the instance configuration. While more than one instance of the same type can be added to the instance set, a combination of different types is not supported.

Perform this task to configure the SNMP Bulkstat instance set.

SUMMARY STEPS

1.enable

2.configureterminal

3.bulkstatinstanceinstance-set-nametypesnmp

4.exactoidoid

5.exactinterfaceinterface-id

6.wildcard

7.wildcardoidoid

8.wildcardinterfaceinterface-id

9.repetitionoidoidmaxvalue

10.rangestartoidendoid

11.end

DETAILED STEPS

Command or Action

Purpose

Step 1

enable

Example:

Device> enable

Enables privileged EXEC mode.

Enter your password if prompted.

Step 2

configureterminal

Example:

Device# configure terminal

Enters global configuration mode.

Step 3

bulkstatinstanceinstance-set-nametypesnmp

Example:

Device(config)# bulkstat instance exact type snmp

Defines an SNMP Bulkstat instance set and enters SNMP Bulkstat instance set configuration mode. The creation of an SNMP Bulkstat instance set creates a row in the cdcDGInstanceEntry table in the SNMP MIB.

Note

An instance created using this command can be linked to more than one data group.

Step 4

exactoidoid

Example:

Device(config-bs-is-snmp)# exact oid 1.2.3

Example:

Device(config-bs-is-snmp)# exact oid 1

(Optional) Indicates that the specified instance, when appended to the object list, is the complete OID.

Step 5

exactinterfaceinterface-id

Example:

Device(config-bs-is-snmp)# exact interface Ethernet0/0 sub-if

(Optional) Specifies an interface name and number, for example interface Ethernet 0, instead of specifying the ifIndex OID for the interface.

Step 6

wildcard

Example:

Device(config-bs-is-snmp)# wildcard

(Optional) Specifies whether an object used for evaluating an expression is to be wildcarded during an event configuration.

Step 7

wildcardoidoid

Example:

Device(config-bs-is-snmp)# wildcard oid 1.2.3

Example:

Device(config-bs-is-snmp)# wildcard oid 1

(Optional) Indicates that all subindices of the specified OID belong to this schema.

Step 8

wildcardinterfaceinterface-id

Example:

Device(config-bs-is-snmp)# wildcard interface Ethernet0/0 sub-if

(Optional) Specifies an interface name and number, for example interface Ethernet 0, instead of specifying the ifIndex OID for the interface.

Step 9

repetitionoidoidmaxvalue

Example:

Device(config-bs-is-snmp)# repetition oid 1.2.3.4 max 2000

(Optional) Configures data collection to repeat get-next for the maximum number of instances starting from the specified oid instance.

Step 10

rangestartoidendoid

Example:

Device(config-bs-is-snmp)# range start 1.2.3.4 end 1.2.3.6

(Optional) Configures a range of instances for which the data is collected.

The expressions are in ANSI C syntax. However, the variables in an expression are defined as a combination of the dollar sign ($) and an integer that corresponds to the object number of the object used in evaluating the expression.

(Optional) Configures discontinuity properties for the object if the object sampling type is set to delta or changed. The discontinuity-object-id supports normal checking for a discontinuity in a counter.

The type keyword sets the value for objects with discontinuity properties.

Configuring a Bulkstat Profile

The profile element is used to group multiple data groups. This grouping simplifies the configuration and aggregates data of a similar nature. If two sets of data need to be written to the same file, the respective data groups should be linked as part of a single profile.

Perform this task to configure the Bulkstat profile:

SUMMARY STEPS

1.enable

2.configureterminal

3.bulkstat profile profile-name

4.data-group data-group-name

5.interval transfer {process | raw }{seconds}}

6.file format schemaASCII

7.file retain {disk url | memory seconds}

8.file size bytes

9.file transfer {retry number | url {primary url | secondary url }}

10.flow exporter fnf-exporter-name

11.enable

12.end

DETAILED STEPS

Command or Action

Purpose

Step 1

enable

Example:

Device > enable

Enables privileged EXEC mode.

Enter your password if prompted.

Step 2

configureterminal

Example:

Device# configure terminal

Enters global configuration mode.

Step 3

bulkstat profile profile-name

Example:

Device(config)# bulkstat profile if-stats

Creates a profile with the given name and enters Bulkstat profile configuration mode. If the profile is already created, this command sets the context for the existing profile.

Step 4

data-group data-group-name

Example:

Device(config-bs-profile)# data-group if-dg

Specifies the data group to be linked to this profile. Multiple data groups can be linked to a single profile.

Step 5

interval transfer {process | raw }{seconds}}

Example:

Device(config-bs-profile)# interval transfer raw 100

Example:

Device(config-bs-profile)# interval transfer process 4000

Specifies the transfer periodic interval in seconds. In case of recurring transfer, the data is transferred at the expiration of the transfer interval until the transfer is stopped.

The process keyword specifies the process periodic interval in seconds. The data is processed during every collection interval as soon as it is collected. When the process interval expires, the processed data is written to a file and transferred.

Step 6

file format schemaASCII

Example:

Device(config-bs-profile)# file format schemaASCII

Configures the file-related parameter for a profile.

Specifies the file data format in schemaASCII.

Step 7

file retain {disk url | memory seconds}

Example:

Device(config-bs-profile)# file retain memory 2000

Configures the file-related parameter retainfor a profile.

disk--Retains the file in the specified location in the disk for a specified amount of time in seconds.

memory--Retains the file in the memory for a specified amount of time in seconds.

Step 8

file size bytes

Example:

Device(config-bs-profile)# file size 2048

Configures the file-related parameter for a profile.

size--Specifies the maximum buffer size in bytes.

When the limit is reached, the file is closed and transfer is attempted based on the transfer configuration associated with the data group or the profile.

Configuration Examples for Cisco Data Collection Manager

Example: Collecting Sorted CPU Processes

The following is sample output that shows the configuration tasks that you need to perform to collect data about the processes in the cpu in the device, such as a router or a switch in a sorted order. The polling interval is set for 300 seconds, while the interval at which the raw data is transferred is set at 1500 seconds.

Example: Collecting SNMP Interface Statistics

The following shows the configuration steps that you need to perform to collect SNMP interface statistics for the specified interface, Ethernet. The polling interval is set at 30 seconds, while the interval at which the raw data is transferred is set at 60 seconds.

Technical Assistance

Description

Link

The Cisco Support and Documentation website provides online resources to download documentation, software, and tools. Use these resources to install and configure the software and to troubleshoot and resolve technical issues with Cisco products and technologies. Access to most tools on the Cisco Support and Documentation website requires a Cisco.com user ID and password.

Feature Information for Cisco Data Collection Manager

The following table provides release information about the feature or features described in this module. This table lists only the software release that introduced support for a given feature in a given software release train. Unless noted otherwise, subsequent releases of that software release train also support that feature.

Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/go/cfn. An account on Cisco.com is not required.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.