Netflow Data Export

This section contains troubleshooting information for NDE.

Web Application

Explanation When you are on the Monitor > Hosts, Monitor > Apps, or Monitor > Conversations page, the data shows only every other or more auto-refresh cycles. This problem is caused by the implementation operation of the NDE source device. Entries in the NetFlow cache are expired after being inactive for a time, when the end of a connection is detected, or when the expiration time has been reached. The expired flow is exported to the destination. If the aging time is longer than the NAM refresh interval, there will be no expired flows and NetFlow packets flow in one refresh interval of the NAM.

Recommended Action To solve the problem, either increase the auto refresh interval on the Setup > Preferences menu, or change the aging time of the NetFlow entries. Before you make any change to the aging time at the NDE source device, refer to the NDE usage guidelines for performance issues.

Cisco IOS Software

For the MSFC or routers, use the following command to specify the aging time:

Router(config)# ip flow-cache timeout"active"||"inactive" seconds

Router(config)# mls aging "fast time" | "long" | "normal" seconds

Catalyst Operating System Software

For the PFC, use the following commands to specify the aging time:

Console(enable) set mls agingtime [long-duration | fast | ip]

To set the aging time for flows that are long active, use the long-duration keyword.

Toset the aging time for flows that do not exceed the packet threshold, use the fast keyword.

To set the aging time for IP flows, use the ip keyword.

Explanation The Monitor > Hosts and Monitor > Conversations page does not contain the data of an active flow. This problem could be caused if the active flow is not expired yet, if the device has an NDE filter, or if a full cache is preventing insertion of new entries. The active flow is not in the NetFlow packets that are exporting to the NAM.

Recommended Action Check the filter long duration aging time or dropped flow packets as follows:

Verify the long duration aging time with these commands:

Console(enable) show ip cache flow

or

Console(enable) show mls netflow aging

or

Console(enable) show mls

Active flows that have their active time below the long duration aging time are not expired yet, and they have not been exported to the NAM. You can set the aging time to a lower value. Refer to the NDE usage guidelines for the device.

Verify the dropped flow packets with these commands:

Console(enable) show ip cache flow

or

Console(enable) show mls netflow aging

or

Console(enable) show mls

Flows could drop because they are not entered into the caches allowing their export to the NAM when they are expired. The NetFlow cache might be full because of busy networks. To correct the problem, you could increase the cache size, or adjust NDE export with the NDE flow mask or version 8 aggregation cache. Refer to the NDE usage guidelines for the device.

Explanation There is no data for the default NetFlow data source of the device.

Recommended Action In the GUI, go to the Setup > Data Sources > NetFlow > Listening Mode page and click on Start. Wait for a few auto refresh cycles. If the device is not displayed in the table, the NAM is not receiving any NetFlow packets from the device. This condition could be a network problem, or the device may not be configured correctly.

To verify that a NetFlow device is configured to send NetFlow packets to UDP port 3000 of the NAM, use the following commands:

Console> show ip flow export

or

Console> show mls nde

Displayed information should show whether or not NetFlow export is enabled or disabled and show the IP address and port to which the NetFlow packets are being exported. If the information is not correct, refer to the configuration section in the User Guide for the Network Analysis Module Traffic Analyzer, 4.1.

Explanation There is no data for NetFlow data sources that are configured for specific interfaces, but the default NetFlow data source for the device has data.

Recommended Action This problem could occur because a NetFlow record that contains information about the specified interfaces does not exist. To find out which interfaces the NetFlow records have, follow these steps:

Step 3 Wait until the row for the device has more than three NDE packets counted.

Step 4 Select the device.

Step 5 Click Details. A window appears displaying a list of interfaces that the NAM has seen in the NDE packets.

Step 6 Make sure that the interfaces selected for the NetFlow devices are included in the list. If the interfaces are not included in the list, configure the NetFlow source devices using the following commands:

For the IP routed cache, use these commands:

Console(config) interfacetype slot/port

Console(config-if) ip route cache flow

For the MLS cache, use these commands for Cisco IOS software:

Console(config)# mls nde interface

For the MLS cache, use these commands for the Catalyst operating system software:

Console>(enable) set mls nde destination-ifindex enable

or

Console(enable) set mls nde source-ifindex enable

Make sure that the flow mask is set to full, interface-destination-source, or interface-full.

If the information is not correct, refer to the configuration section in the User Guide for the Network Analysis Module Traffic Analyzer Release 4.1.

Explanation When creating a NetFlow data source from the Setup > Data Sources > NetFlow > Custom Data Sources screen, only the local device's address appears in the drop-down box.

Recommended Action A device is created in the Setup > Data Sources > NetFlow > Devices screen. After adding a device from this screen, a default NetFlow data source for the device appears in the Setup > Data Sources > NetFlow > Custom Data Sources screen. Now, the drop-down box displays the device address included in the list.

Explanation When creating a NetFlow data source, no available interfaces list is displayed.

To make sure that the community string is correct, follow these steps:

Step 1 Go to the Setup > Data Sources > NetFlow > Devices window.

Step 2 Click on the radio button of the device to display information about the interfaces.

Step 3 Click Test.

A popup window appears displaying the status of the device. If there is an error in this window, the community string may not be correct. Correct the community string by selecting the device, click Edit, and provide the correct community string. Also, ensure that the remote device accepts SNMP connections.

Explanation The Monitor > Conversations page has the source column as 0.0.0.0 for all entries. This problem occurs when the NDE device flow mask is set to destination.

Cisco IOS Software

If using Cisco IOS software to set the flow mask to full, interface-destination-source, or interface-full, enter this command:

NDE Flow Records Interfaces

Explanation An NDE packet has multiple NDE flow records. Each flow record has fields of flow input SNMP if-index and flow output SNMP if-index. The information may not be available due to an unsupported NDE feature of the Cisco IOS or Catalyst operating system version or misconfiguration of the NDE flow masks.

Recommended Action In most cases, turning on NetFlow on an interface will populate the NetFlow cache in a switch or router with flows that are in the input direction of the interface. As a result, the input SNMP if-index field in the flow record will have the if-index of the interface that has NetFlow turned on.

Interface Special (0)

Explanation NDE packets sometimes have NetFlow records reporting either or both input if-index and output if-index fields as 0. This problem may be due to one or more of the following reasons:

–Flows that are terminated at the device.

–Configurations of the device.

–Unsupported NetFlow feature of the platform at the device.

Recommended Action Remove flows that terminate at the device, check the device configuration, and make sure that there are no unsupported features on this platform at the device.

NDE Flow Mask and Version 8 Aggregation Cache

This section describes how some of the flow masks and NDE version 8 aggregation flows affect the data collection screens in the NAM. Table 5-3 lists the effects on the data collection screens. Due to a lack of information, some collections may display "Others" only in the Monitor > Apps, 0.0.0.0 in Monitor > Hosts and Monitor > Conversation pages.

Table 5-3 Effects on Data Collection Screens

Flow

Effect

Full flow-mask is supported

Highly recommended. Refer to the NDE usage guidelines for the device to apply full flow masks.

Note Although the NAM supports NDE aggregation, the information that you received for a specified aggregation type is limited to that aggregation and other NDE details are not available. To receive more information about your NDE configuration, use the full flow mode.

Destination only flow-mask

•Monitor > Apps displays "Others" only.

•Monitor > Apps detail popup window has no data.

•Monitor > Hosts has 0.0.0.0. Detail popup window has no data.

•Monitor > Conversations has 0.0.0.0 to some hosts. Detail popup window has no data.

•Support NetFlow custom data sources that are set up for specific interfaces.

Destination-Source flow-mask

•Monitor > Apps displays "Others" only.

•Monitor > Apps detail popup window has no data.

•Monitor > Hosts has data. Detail popup window has no data.

•Monitor > Conversations has data. Detail popup window has no data.

•Support NetFlow custom data sources that are set up for specific interfaces.

NDE version 8-Protocol-Port-Aggregation

•Monitor > Apps displays data.

•Monitor > Apps detail popup window displays only 0.0.0.0.

•Monitor > Host displays only 0.0.0.0.

•Monitor > Conversation displays only 0.0.0.0 to 0.0.0.0.

•No data for custom NetFlow data sources that are set up for some specific interfaces.

•In Catalyst operating system software, to boot to the application image, use the reset 9 hdd:1 command.

Symptom You receive a verification failed message when installing a patch on the NAM.

Possible Cause The cause could be that the time and date on the NAM are not correct, the patch is not the same as an official Cisco patch, the patch might be from a previous release of the NAM, the FTP process may have failed, or the FTP image being pointed to is not a patch (it may be a full application image).

Recommended Action Make sure that the signature verification is used to ensure that the patch is an authentic Cisco patch and that the patch is for the correct NAM release. For example, a patch for the NAM 2.2 release cannot be applied to a NAM running the NAM 4.1 software. Make sure that the date and time on the NAM is set to synchronize with the switch or with the Network Time Protocol (NTP). Make sure that the URL location is valid for the patch (verify the username)

Symptom You are unable to log into the maintenance image with the same password for the NAM application image.

Note This message is applicable only for the WS-SVC-NAM-1 and the WS-SVC-NAM-2 modules.

Possible Cause The NAM application image and the maintenance image have different password databases for the root and guest accounts. The default passwords for root and guest differ between the maintenance image and the NAM application image. Any password change performed in the NAM application image does not change the maintenance image password and vice versa.

Recommended Action Use the maintenance image password.

Symptom You lost your password for the maintenance image and want to recover it.

Possible Cause The maintenance image does not support resetting passwords from the switch. Upgrading the maintenance image sets the password for root and guest to default in the maintenance image.

Symptom When attempting to load the new NAM 4.1 image on the NAM, the following message displays:

Incompatible image! Upgrade aborted.

Possible Cause This image is not supported on the specified NAM. Two NAM 4.1 images are available: One each for the WS-SVC-NAM-1 and WS-SVC-NAM-2. This symptom occurs only if an incompatible image is used.

Recommended Action The newer NAM shares a common format and the same image filename for upgrades can be used.

Symptom When attempting to load the wrong image on a WS-SVC-NAM-1 or WS-SVC-NAM-2, the following message displays:

ERROR: /tmp/upgrade:No space left on device

Possible Cause This image is not supported on the specified NAM. Two NAM 4.1 images are available: One each for the WS-SVC-NAM-1 and WS-SVC-NAM-2. This symptom occurs only if an incompatible image is used.

Recommended Action The application and maintenance file image formats are different between the previous NAM releases and the newer WS-SVC-NAM-1 and WS-SVC-NAM-2. The newer NAM shares a common format, and the same image filename for upgrades can be used between these newer modules.

Symptom A SPAN session does not show up in the Traffic Analyzer Active SPAN window.

Possible Cause In Catalyst operating system software, a SPAN session becomes inactive if the module containing the destination port is removed from the switch chassis. The NAM is not seen by the SPAN session because the SPAN configuration is removed from the SNMP agent by the supervisor engine.

You can also use the hw-module module module_number mem-test-full command as follows:

Router(config)# hw-module module 5 mem-test-full

For the Catalyst operating system software, you can enable a full memory test when you use the set boot device bootseq mod#mem-test-full command. This option is disabled by default. This example shows how to enable a full memory test:

Console (enable) set boot devicecf:1 4 mem-test-full

Device BOOT variable = cf:1

Memory-test set to FULL

Warning:Device list is not verified but still set in the boot string.

Console> (enable) show boot device 4

Device BOOT variable = cf:1

Memory-test set to FULL

This example shows how to reset the partial memory test:

Console> (enable) set boot device cf:1 4

Device BOOT variable = cf:1

Memory-test set to PARTIAL

Warning:Device list is not verified but still set in the boot string.

Console> (enable)

Console> (enable) show boot device 4

Device BOOT variable = cf:1

Memory-test set to PARTIAL

Symptom When you click the Test button in the Setup > Chassis Parameters menu window, the popup window indicates that both the SNMP read and write to the switch failed.

Possible Cause Verify that the SNMP read-write community string is the same as the SNMP read-write community string defined for the switch.

Note The password is case sensitive.

Recommended Action If the community string is correct and the test still fails, check that the switch has enabled the IP permit list as follows:

Step 1 Log in to the switch in enable mode.

Step 2 Enter the show IP permit command.

If the IP permit list is enabled, make sure that the NAM internal address is added to the IP permit list. The NAM address is 127.0.0.X, where X is the NAM module number multiplied by 10 plus 1. For example, if the NAM is at module 4, then its address should be 127.0.0.41.

Symptom When a NAM is running in a switch with Catalyst operating system software, the NAM may be shown as unreachable when you use the ping command or the NAM Traffic Analyzer application.

Possible Cause The NAM IP address and the IP address of the switch (interface sc0) are not in the same subnet. This problem can occur if you change the switch IP address and the NAM VLAN assignment. The NAM automatically synchronizes its VLAN assignment to the same VLAN in which the switch (interface sc0) resides. When this occurs, the NAM IP address resides on a different subnet from the VLAN assigned to the NAM. The router then drops any packet destined to the NAM IP address. You cannot add a static route to the router because of route overlap caused by improper VLAN assignments and subnetting.

Recommended Action Make sure that the NAM IP address and the switch are in the same subnet and in the same VLAN.

Symptom You cannot connect to the NAM.

Possible Cause The initial configuration is incorrect or not configured.

Symptom After configuration, the TACACS+ authentication and authorization fails.

Possible Cause There are three possible causes: The name and password do not match the login configuration in the TACACS+ server, the TACACS+ secret key configured in the NAM does not match the secret key configured in the server, or the wrong TACACS+ server IP address is configured in the NAM.

Recommended Action To determine the cause of the problem, follow these steps:

Step 1 Log in as a local user.

Step 2 Choose Admin > Diagnostics > TechSupport.

Step 3 Scroll down to view the /var/log/messages area.

Step 4 Look for the following messages near the end of the log and take the recommended actions:

...PAM-tacplus[612]:auth failed:Login incorrect

Possible Cause The name and password do not match the login configuration in the TACACS+ server.

Recommended Action Log in to the TACACS+ server and configure the authentication and authorization for the NAM user. (See the TACACS+ documentation for information on login configuration.)

Web Username and Password Guidelines

Observe the following web username and password guidelines:

•You cannot use the CLI username (root or guest) and password to log into the NAM Traffic Analyzer application because they are administered separately. You also cannot use your NAM Traffic Analyzer username and password to log into the NAM CLI.

You can create web users with a local database or using TACACS+. You can create a web user with the same username and password as used on the CLI. However, you must still make password changes in both places.

•You can use TACACS+ in addition to a local database or instead of a local database. (The local database is always checked first.) To use only TACACS+, eliminate the local database users by either of these methods:

–Use the NAM CLI rmwebusers command to remove only local users, not TACACS+ users, because they are administered separately on the TACACS+ server.

–From the Admin tab, click Users, and then delete all local database users individually.

Caution Do not delete all local database web users until you have verified that you can log into NAM Traffic Analyzer as a TACACS+ user.

•You can recover the password in situations where you have forgotten the local web admin user password, or when another user with account permission logged in and changed the local web admin user password.

To recover the passwords, follow these steps:

Step 1 Access the NAM CLI.

Step 2 Enter these commands:

web-user

user name name

exit

Step 3 At the prompt, enter the new password.

Step 4 Enter Y to confirm the new password.

When the NAM TACACS+ setting is misconfigured and a local database user account is not available to fix this problem from the web interface, you may be able to fix the TACACS+ configuration by using the CLI interface.

To recover the passwords, follow these steps:

Step 1 Access the NAM CLI.

Step 2 Enter this command:

ip http tacacs+ enable tacacs+ server

Step 3 Follow the commands to enter the TACACS+ secret key.

Supported MIB Objects

Table 5-4 lists the RMON and RMON2 MIB objects supported by the supervisor engine and the NAM. The supervisor engine implements some objects from the RMON MIBs as specified in Table 5-4. The supervisor engine RMON implementation is completely independent of the NAM implementation, and no MIB objects are shared.

To collect etherStats from a physical interface on the switch, configure the etherStatTable on the supervisor engine instead of on the NAM. The etherStats are collected accurately on multiple physical interfaces simultaneously.

If you are interested in the etherStats for a specific VLAN, configure the etherStatsTable on the NAM. For the data source, use the ifIndex corresponding to that VLAN.

Any alarmVariable configured on the supervisor engine must reference a MIB object on the supervisor engine. An alarmVariable configured on the NAM must reference a MIB object on the NAM.

Note You cannot configure an alarmVariable on the NAM that references a MIB object on the supervisor engine or configure an alarmVariable on the supervisor engine that references a MIB object on the NAM.

Local Interfaces in the NAM ifTable

This section explains the local interfaces in the NAM-1 and NAM-2 devices. The four versions of the Network Analysis Module (NAM) are as follows:

•WS-SVC-NAM-1

•WS-SVC-NAM-2

•WS-SVC-NAM-1-250S

•WS-SVC-NAM-2-250S

The WS-SVC-NAM-1 appears in the supervisor engine CLI (in the Catalyst operating system) and ifTable as three ports. The first port is unused. The second port is the management port. The third port is the data port (for receiving SPAN traffic). The supervisor engine CLI (in Cisco IOS software) parses the ports to ("analysis module . . ."). On the NAM ifTable, the management port appears as the first port (ifIndex.1) and the data port appears as the second (ifIndex.2).

The WS-SVC-NAM-2 appears in the supervisor engine CLI (in the Catalyst operating system) and ifTable as eight ports. Ports 1, 3, 4, 5, and 6 are unused. Port 2 is the management port (the same as on WS-SVC-NAM-1). Ports 7 and 8 are both data ports and can be SPAN targets. The supervisor engine CLI (in the Cisco IOS software) parses the ports to ("analysis module . . ."). On the NAM's ifTable, the interfaces are as follows:

•ifIndex.1: Is designated the management port.

•ifIndex.2: Represents the traffic from both data ports (also known as "All SPAN").

•ifIndex.3: Represents the traffic from the first data port (named "data port 1")

•fIndex.4: Represents the traffic from the second data port (named "data port 2")

Note The data ports for NAM-1 and NAM-2 devices are IEEE 802.1Q trunk ports. Packets are received with an 802.1Q header (except for packets with the ports native VLAN ID), affecting offsets (for example, the filters on the IP headers) in the packets.