This document describes techniques to troubleshoot problems that you might experience with Cisco Unity audio quality. Audio quality for the Cisco architecture for voice, video and integrated data (AVVID) solution is a perceptual measure of how good audio sounds as it reaches the intended recipient. With any subjective evaluation, it is most important to realize the measurable areas of audio quality and identify how these areas affect the reported audio quality problem.

For example, if a reported problem is “loud Unity system prompts,” volume can be described as the problem area that impacts audio. If you isolate the investigation to volume levels, you can take the steps in this document to further investigate what part of the environment might change volume levels.

After you categorize the type of audio quality issue, you can determine the source of the audio distortion thorough a process of elimination. This can be a complex process, depending on the number of devices in any AVVID deployment that originate, control, or deliver audio streams to Cisco Unity. Keep in mind the complexity of the environment and the various paths the audio stream take, to more quickly identify the source of audio distortions. Each of the known areas of audio quality issues in this document follow this logic as troubleshooting progresses.

For example, when the problem area that affects audio quality is known to be volume, there are several ways to change volume in the AVVID environment. The Volume Levels and Gain Control section first addresses any potential volume changes Cisco Unity can make to an audio stream; it then covers other known sources of volume modification that can impact audio received by Cisco Unity. If none of the topics in that section correct the audio distortion, check the other listed symptoms for a similar sounding audio issue.

The information in this document is based on these software and hardware versions:

Cisco Unity version 2.4.6.161, 3.x versions through 3.1(5), and 4.x versions through 4.0(2).

Netmon Network Analysis Tool from the Windows 2000 Resource Kit

Sniffer Pro version 4.5 from Network Associates, Inc. (NAI).

Cisco Unity utility AudioStat, available in Cisco Unity 4.0(1) and later versions; and Capripper utility, available in Cisco Unity 4.0(2) and later

Note: Not all are required for every type of audio quality issue described.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Note: The links in this document contain the most current information known on the type of audio distortion possible in the environment. Review the symptoms of each before you attempt troubleshooting steps.

The symptom of this problem area relates to unexpected changes in the volume level of a voice mail message or volume levels from Cisco Unity measured as “too loud” or “too quiet.”

Volume level and gain control issues are most often reported when audio is too quiet in a message for the user to understand the content. There are also possible fluctuations of volume that could be reported as creating a “garbled” message, where the volume goes up or down regardless of the volume of a speaker’s voice (for example: Audio Volume Fluctuation on Phone Load).

Cisco Unity versions 3.1(2) and later also contain an automatic gain control (AGC) feature, which might contribute to fluctuations in volume if incorrect settings are introduced in the configuration. This is most often noted by large increases in volume at the end of a message (for example: Audio Volume Fluctuation from Incorrect AGC Settings).

Finally, adjustments to recording or playback gain where AGC is used can modify the incoming audio stream before it reaches the AGC operation. This can cause extreme volume levels in the message (for example: Audio Volume Extremes from Incorrect TSP Settings).

Audio volume distortion can occur at any point in a voice stream, from an originating cellular phone on an external call through to the IP phone that receives a message from that caller. Volume gain might be distorted with an increase or decrease in volume through these devices:

PSTN devices (unknown, and usually not programmatically altering volume, although “noise-reducing” microphones can adjust input gain)

Cisco Unity versions later than 3.1(2) with AGC enabled have a target nominal volume level of –26 decibels by default. When other devices in the deployment might also control gain levels, it can be easiest to isolate a volume issue by first disabling these settings on gateways that deliver audio to Cisco Unity.

Note: When the gain levels vary on different gateways, the settings should be noted so that you can return the levels to the original values after you troubleshoot (if needed).

Note: Ensure that you do not increase the volume of speakers or headphones while you are troubleshooting this type of issue.

Use these steps to troubleshoot volume level and gain control issues:

Identify which audio streams are affected by the volume problems.

Are all audio streams from Cisco Unity at a low volume? If only a particular part of an audio stream is affected (for example, only Cisco Unity system prompts or only messages), proceed to Step 2.

When all audio from Cisco Unity is at a low volume and phone-to-phone volumes are low, first check gain settings on gateways in the environment.

From a single source (a local IP phone, for example), leave messages for Cisco Unity with similar volume levels. Applying a steady speaking voice to say the same 10 to 15 second phrases.

An internal call should serve as the “control” point for volume on the IP network. Calls that are made out through the PSTN and back into Cisco Unity show the volume levels that come through any number of analog gateways.

After you complete all of the test calls, stop the UDT trace and gather the diagnostic for the time period that the calls were made.

Find the individual calls by their timestamp in the diagnostic, and view the Power and Gain adjustment settings for each call.

If calls consistently have a Gain adjustment of ±5db or greater, either the gain on the gateways need adjustment or the Cisco Unity TSP registry settings should be checked (proceed to Step 2).

When only audio from Cisco Unity is at a low volume, check the TSP playback and record volume levels from the registry key on the Unity server:

Run Regedit and verify the keys in HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems, Inc.\Cisco TSP.

The WaveDBGainPlayback and WaveDBGainRecord keys should both have the value of 0.

A clear example of when this setting can affect audio is when a gateway has a positive gain applied and the WaveDBGainRecord has a negative value. The volume level is incorrectly being adjusted at the gateway and Cisco Unity. A volume increase applied at the gateway followed by a volume decrease at the TSP could result in the same lower volume level going into Cisco Unity as that which entered the gateway.

In the same way, when a gateway has negative gain applied and WaveDBGainRecord has a negative value, any value on the WaveDBGainPlayback could be useless to adjust low volume messages played from Unity.

Finally, a negative WaveDBGainRecord value with a negative WaveDBGainPlayback value will always decrease the volume of any audio played out of Unity. Older TSP versions (pre 3.0) might retain a WaveDBGainRecord value of 5, which causes distorted volume levels on recorded messages.

Note: If the Cisco Unity version is 3.1(2) or later and AGC is active, the WaveDBGainPlayback and WaveDBGainRecord values should always be 0. As with the multiple points of volume adjustment described in the previous paragraph, if the audio volume level is already manipulated by AGC, then it is not recommended that you adjust either recording or playback volume levels in the TSP.

Check if the registry setting values for AGC are set incorrectly. The bad values are usually:

AGCsamplesize is 4e20 hex (20000 decimal) and should be 1f40 hex (8000 decimal)

AGCgainthreshold is 28 hex (40 decimal) and should be 5 hex (5 decimal)

If so, change the registry settings to the correct values, as described in the AGC Registry Keys table that can be found in the Audio Quality document.

Note: If the quality of the prompts are unacceptable when you use the G.729 codec, you can change the codec to G.711 for better audio quality.

Identify which part of the audio stream is affected by the volume change: Cisco Unity system prompt, messages left for subscribers from outside the local network deployment (analog or digital), or messages left for subscribers from other subscribers.

When only messages from callers outside of the VoIP deployment have volume levels above or below messages left by subscribers or Cisco Unity system prompts, verify gain levels on gateways that deliver this audio:

See the Connectivity, Jitter, Packet Delay, and the ‘Garbled Message’ section of this document, if the volume level is fluctuating quickly and consistently within any message. Even though the volume is actually zero in many parts of these types of distorted messages, it can be perceived as a volume fluctuation issue.

“Garbled message” is most often reported when a message left for a Cisco Unity subscriber is either missing enough audio content to be considered unintelligible or when the packets are interleaved such that the content sounds out of order. Rather than longer specific periods of silence in the message, which might be related to volume levels (as previously noted), the garbled message might have other distortions present.

This example is notable because of the distortion that is inserted for brief intervals at various points in the message. Not only are missing packets the cause of silence insertions, but additional signals from other audio streams—which the NIC is delivering incorrectly—are heard as “pops” throughout the message.

Audio connectivity distortions can also occur at several points in a network from a gateway receiving an external call through to the IP phone that receives a message from that caller. Use these steps to troubleshoot audio connectivity distortion issues:

Identify which audio streams are affected by the garbled audio.

Are all audio streams from Cisco Unity garbled?

When only Cisco Unity system prompts are garbled, verify the fix for garbled prompts playing from Unity through a Catalyst 6000, refer to DDTs issue Cisco bug ID CSCdx36894 (registered customers only) . The distortion associated with this defect is heard in this example: Unity Garbled Prompts Distortion.

Where all audio from Cisco Unity is garbled and phone-to-phone audio is also garbled, verify the duplex settings on the Unity NIC configuration and on the switch port to which Unity is connected. These should not be set to auto-negotiate, but rather to hard-coded values (usually 10/100 full duplex).

This eliminates the possibility that auto-negotiation of packet delivery is a cause or contributor to distortion of the audio streams.

If the distortion continues after taking the step above, check the network topology and available bandwidth to Cisco Unity. Check relevant gateways for alignment errors.

If alignment errors are found on any switch ports, and the Cisco Unity server is an HPQ DL380G2 (MCS7837, MCS7847), then the distortion could be associated with an incorrect NIC driver. These errors are typically reported as alignment errors, runt packets, or Frame Check Sequence (FCS) errors by switches. The NC-Series NIC statistics might report the errors as either Transmit Underruns or Receive Overruns. Check the N100NT.SYS driver on the Cisco Unity server. If it is version 5.30.74.001, the solution is to install an older version of the driver from: http://welcome.hp.com/country/us/en/support_task.html.

If the distortion continues or the NIC driver downgrade does not apply to the hardware, the audio stream that is entering the Cisco Unity server must be validated for any distortion per the instructions in the next step (Step 2).

Identify whether the source of distortion is internal or external to Cisco Unity.

This is a simplified description of the audio stream path from the Cisco Unity NIC to a wave file:

UnityAvWav converts the stream to PCM, applies AGC, and converts the audio to the destination codec format —> temporary file created

Temporary file completed at end of recording —> wave file created and passed to the mailstore, after leading or trailing silence is trimmed

When you identify the inconsistencies in the audio stream based on the different paths it takes before it becomes a wave file delivered through email, you can isolate the source of distortion to one of these reasons:

An external source of audio distortion

A defective NIC card

A bandwidth-related distortion that impacts the wave driver logic while collecting the incoming audio stream

An objective comparison of the audio streams as they pass through the components is necessary to confirm any of these sources.

Caution: You should not use the methods in Steps 3 and 4 as general troubleshooting methods, if other steps have not been attempted. To capture audio traffic without discretion is not recommended, and it might affect the AVVID network.

Caution: You should use a maintenance window when you do Steps 3 and 4, as the actions could impact service.

Use a test phone and test subscriber to obtain the best objective measure of audio quality, unless a specified endpoint has been identified as the source of distortion.

Collect the incoming audio stream from the network to the NIC, for a comparison of external audio to audio written to the mailstore.

Span the Unity port at the network device to which Cisco Unity is connected, and collect the packets through a Sniffer. You can also use Netmon (from the Microsoft Windows 2000 Resource Kit) from any Windows 2000 server that is external to the Unity server, to collect the data going to the Unity server.

Note: It is recommended that you only run Netmon on the Cisco Unity server itself after the audio stream has been collected external to the Unity server NIC.

For packet captures, use a capture filter to take UDP packets from any source with a destination of the Cisco Unity server IP address. Save the captures from the testing time period, to compare them to the data that you will collect in the next procedure.

In order to collect the incoming audio to the Cisco Unity wave driver—Use Netmon on the Cisco Unity server to collect the data incoming UDP audio streams. After Netmon is installed, complete these steps to configure Netmon:

Copy the RtpParser.dll file to …\WINNT\system32\NETMONFull\PARSERS. This file is available in the Audio directory of the Cisco Unity \Commserver\Utilities directory.

Select the appropriate interface and capture only UDP traffic to Unity.

In order to collect the audio stream the wave file created (written by the Cisco Unity wave driver)—Most often, problem audio is reported from a user directly as they receive the voice mail(s) that contain distortion. If so, request these messages to help isolate the issue. For further troubleshooting, it is best to create a test subscriber account that you can access from a mail client, so that you can save the voice mail wave files to disk. If this is not possible, have the messages forwarded from this test mailbox to an outside address, to assist in the collection of the wave files. Finally, if neither of these are possibilities (where a voice-mail only system is in place, for example), this procedure allows for direct access to the wave files before they are delivered:

Stop the Microsoft Exchange Information Store service from the Exchange mailstore with which Cisco Unity is homed.

This places Cisco Unity into the Unity Message Repository (UMR) mode, where all messages sent to subscribers go first to the …\Commserver\UnityMTA directory. For more information, refer to How to Start Cisco Unity in UMR Mode.

Collect the resulting wave files from this directory after test messages are sent.

Caution: No messages are delivered on the Exchange mailstore during this time. This procedure impacts any incoming call—use only during a maintenance window.

Restart the Microsoft Exchange Information Store service after the tests are completed.

In order to compare the collected data, use the Capripper utility to convert the packet capture files into wave files (also found in the Audio directory of the \Commserver\Utilities directory in Cisco Unity version 3.1(5) and later):

Copy the capripper.exe file and the cap file to be decoded into a new directory.

Drag the .cap file containing the test audio streams onto the capripper.exe to create a series of .wav files named for the endpoints in each call, as shown in these screens:

Listen to the test calls from the packet capture and from the mailstore. If there is any difference between the stream collected from the network to the NIC and the stream collected from the NIC to the Unity wave driver, there is likely a hardware issue with the NIC.

For instance, this example—Packet Capture Encoded to a Wave File—collected from traffic to the NIC, compared with this example—Unity Message Wave File—collected from the Cisco Unity system, shows an issue with a NIC hardware failure. In this case, if the server has a dual-NIC configuration, disable the currently used NIC and enable the second NIC. If no more reports of distortion occur, the NIC is the source of the audio distortion. If the server has only one NIC available, install a PCI-NIC where possible and switch all traffic to run through this interface. If no further distortion is heard, contact Cisco Systems Technical Support to report the defective equipment.

When the previous steps show that audio that enters the Unity server and passes through the NIC is of acceptable quality, but distortions still exist in the wave file written to the drive, you must collect the audio stream with Netmon. Determine the arrival time of the packets for the audio stream. If there is more than 20ms of delay between packets, the wave driver might insert silence into the wave file as a normal operation. If packet captures show delay as the cause of multiple silence insertions, DDTs issue Cisco bug ID CSCdx41866 (registered customers only) , addressed in Cisco Unity version 3.1(5), might broaden the window for packets to be received by the wave driver. Although you can set a greater packet delay with this registry key, this indicates unacceptable delay in the network that you should investigate.

In order to investigate the delay, run the AudioStat utility (found in Cisco Unity versions 4.x and later) from the Audio directory of the \Commserver\Utilities directory. This tool displays average delays and silence insertions for each recording along with the source IP address of the device that is sending audio to Cisco Unity.

These screens show the AudioStat utility:

If any single device shows increased latency, investigate that link in the network. The AudioStat utility helps to isolate where in the topology latency is introduced.

Another frequent cause of jitter and choppy voice messages is when Voice Activity Detection (VAD) is enabled. You can overcome this when you turn off VAD if bandwidth is not an issue.

DTMF recognition and voice-mail response issues that affect audio quality are most often reported as one-way audio during a call to Cisco Unity or a lack of response from the voice mail system after digit entry by the caller. The voice mail responsiveness issues can sometimes be correlated to the integration factors listed later in this section, but they most often might depend on Exchange deployment configuration as described in Cisco Unity: Delays in the Subscriber Conversation.

Silence suppression, voice activity detection (VAD), and comfort noise might also contribute to message clipping or cut off (where the system is recognizing silence as a call termination event). This section contains a list of documents to assist with configuration and troubleshooting.

Refer to these troubleshooting documents, if the Cisco Unity integration involves a legacy PBX:

Echo is included here as a general audio quality issue that affects AVVID deployments, which should be familiar to anyone that troubleshoots audio quality issues. It has yet to be the type of issue that could ever impact a message left on a Cisco Unity system. Echo can occur during an IP call when the distortion is generated through digital or analog gateways.