Tag Archives: cucm

With the release of CE8 code for Cisco video endpoints (like the SX10 (8.1), SX20 (also MX200-G2 and MX300-G2) and SX80-based endpoints like the SX80, MX700 and MX800), and the appropriate infrastructure components, multistream video is a possibility. Multistream video allows an endpoint to send multiple resolution video streams and have the bridge pass the most appropriate streams to the far-end video units. The far end video unit would receive a full resolution stream of the active speakers, and then low quality streams of the other participants. The most useful feature of multistream video is the ability to use both screens of a dual-screen video unit to see remote participants (when doing single-stream transcoded mode, you can only do single screen video, and secondary screen content.) Multistream also allows for ActiveControl layout, which allows the endpoint to choose the video layout vs. the video bridge determining the layout of the participants (which has rudimentary DTMF layout control).

Components used in my lab configuration:

SX10 (CE8.1)

MX800 (CE8.0.1)

(2) 8845 video phones (used to inject more video streams — these endpoints do not support multistream, they do single stream and receive their layout from the bridge)

Conductor XC4.1

vTS 4.2(4.23)

CUCM 11.0(1)21900-11 (Latest and greatest version is a requirement) -or- VCS X8.7.1

This guide assumes you’ve already setup a Rendezvous (aka MeetMe) number/URI that is routed to Conductor/vTS and you’re able to to normal conference calls. We’ll modify settings to enable multistream.

The relevant portion of this configuration is to make sure your SIP trunk to conductor is in a Location that supports full quality video. I sent the inter-region bandwidth to UNLIMITED in my test system. Cisco recommends a minimum of 1mpbs per screen, otherwise the vTS bridge may kick that video unit down to single-stream transcoded mode.

Configure the endpoint to support multistream

In CUCM the setting is in the device specific settings, Multistream Mode needs to be set to Auto. Despite some of the documentation reading otherwise, Auto will attempt to do multistream, there is not actually an On setting.

Configure CUCM

Configure the SIP Profile used by the SIP trunk to Conductor to include the following settings:

On the Conductor server, under the Conference Template you’re using for your conference, select advanced template parameters and add:

Enable iX protocol – True and the box checked

Multiscreen layout – ActivePresence and the box checked

No settings on vTS need to be changed, it will automatically do multistream if the endpoints meet the requirements, and CUCM (or VCS) and Conductor are properly configured.

When you join with a multi-stream endpoint you will see the following on vTS Conferences page:

You’ll notice the endpoints that support multistream show Multistream, and the 8845 phone named “Mike White” is Standard because it only supports a single stream.

If we look at the statistics for 5580 (the SX80) you’ll see multiple video streams being sent and received:

Lastly if we look at the call statistics from the video endpoint itself, we see the same information:

The touchpanel now shows more details in the layout. You can see each participant in the conference and the active speaker.

While you can select from several canned layout modes (same typical layouts are you’re used to), this version doesn’t yet support complete drag and drop layout of individual participants where you want them. If you select a particular participant, you can see information about any of the participants and boot them if you are meeting organizer:

Overall its very cool, and sets the groundwork for much more flexibility in the future with layout control.

I took some time to help a customer upgrade their system from CSR (Cisco System Release) 9.1 to 10.5 recently. As any good upgrade goes, it wasn’t without some significant drama….

[In particular, they were upgrading from CUCM 9.1(2) to CUCM 10.5(2); CUCM IM&P 9.1 to 10.5(2); CUC 9.1(2) to 10.5(2) and UCCX 9.0(2) to 10.6(1); Expressway x8.1 to x8.5]

Expressway-C and E was a textbook upgrade using the .gz upgrade files.

Unity Connection (aka CUC) was the first core component that we chose to bite off because it doesn’t have any version dependencies with the other components. It was a textbook upgrade without issues.

The minor snag was CUC going unlicensed immediately because it was pointing to ELM on CUCM 9.1 and CUCM didn’t have 10.x licenses installed. So watch out for that. It was rather odd that CUC didn’t give us a 60-day grace period. I moved it to it’s own PLM with appropriate licensing installed there.

We next chose to upgrade UCCX as 10.6(1) is compatible with CUCM 9.1(2). [http://docwiki.cisco.com/wiki/Unified_CCX_Software_Compatibility_Matrix_for_10.6%281%29]

This is a refresh upgrade so you must install a refresh upgrade COP file on UCCX before installing 10.6(1). Because it is a refresh upgrade the system upgrades the underlying VoS (RHEL-based Voice Operating System) and CCX is down while the system reboots and upgrades the OS. I selected to have CCX stay on 9.0(2) after the upgrade, so that 10.6(1) is the inactive version and will just need a version switch reboot to go live.

The switch version reboot turned into a bit of a mess. I issued it but the server didn’t seem to actually do anything. I issued a CLI utils system reboot command about 2 minutes later and it screamed back that I shouldn’t do that as the system was in a version switch and the database could be corrupted. I let it sit about 30 minutes and tried again. This time it rebooted without complaining and came up on 10.6(1).

I had to run the typical process of updating the CAD client (this customer will move to Finesse in the next phase of the upgrade), using the Client Configuration tools you download and install from CCX.

Next was the CUCM publisher. This ended up being a multi-hour affair. I’d already heard about the very common problem of the Common partition being full, so I’d taken the liberty to use RTMT to clear out old logfiles. You basically go to Trace and Log Central in RTMT, select Collect Files, choose a time period (I chose the previous 5 month period) did not ZIP the files (for time sake) and most importantly checked the box to Delete the files from the server.

I had the Common partition down to 50% utilized before the upgrade.

It failed citing the good old bugid CSCuc63312:

There is not enough disk space in the common partition to perform the upgrade. For steps to resolve this condition please refer to the Cisco Unified Communications Manager 9.1(1) Release Notes or view defect CSCuc63312 in Bug Toolkit on cisco.com.

So second-guessing myself, I decided to use the sledge hammer known as ciscocm.free_common_space_v1.1.cop.sgn COP file to clean out the common partition (this COP file script nukes the currently installed inactive version). I gave that a run and rebooted the server.

The next attempt at install also failed with the same CSCuc63312 error!

Knowing that the Common partition wasn’t the issue (which you can see in the install logs that you can copy and paste from the GUI), I started doing some digging around. It turns out that if the main partition doesn’t have enough room (which you can see in the log files by searching for the word “needed”), it will throw the CSCuc63312 error erroneously.

I found a couple of TAC cases where the next step to clean up room on the main partition is to clean up the TFTP directory. This customers TFTP directory was over 5GB in size from multiple versions of large firmware for endpoints like the DX650 and Telepresence codec firmware for endpoints like the C40, SX20, SX10, etc.

Even after cleaning up the TFTP folder I was still hit with the same stupid error message: Not enough space in the Common partition. Even though I knew Common had plenty of room and I was fighting a main partition space issue.

Some enlightenment

I remembered that from CUCM 10.0 on, the OVA template had increased the disk size from 80GB to 110GB. It turns out that if you increase the size of the VM’s disk in 10.0 or greater, CUCM will automatically see the space and take advantage of it. CUCM 9.x doesn’t do this automatically.

The special sauce to get 9.1 to expand the disk is to install the ciscocm.vmware-disk-size-reallocation-1.0.cop.sgnCOP file, shut the VM down, resize the VM HDD in ESXi to 110GB and then boot it back up. CUCM will reboot a couple of times during the process and then come all the way up.

After doing this CUCM 10.5(2) was able to install successfully. Keep in mind that this is a refresh upgrade so the system will be down for an hour or so while VoS is updated, and the server will go through a couple reboots.

CUCM IM&P (CUP)

I initially tried to install the upgrade ISO and received an invalid checksum error. Thinking it was the version of IM&P I’d downloaded from CCO (it’s been updated twice in the past week) I re-downloaded and hit the same error. Note the current version of CUP is 10.5(2a) as of this writing. If I’d paid attention to the documentation I’d have realized that you need to install the ciscocm.version3-keys.cop.sgn COP File which has the new keys that the 10.5(x) software images are signed with. After installing this, the upgrade would recognize the ISO and upgrade.

Collab Edge is now supported. The official Mobile-Remote-Access-via-Expressway-Deployment-Guide is located here.

I’m updating this document to reflect changes made in Expressway-C/E 8.1 that make importing the certificates MUCH easier.

Introduction

This document explains how to deploy Collaboration Edge with on-prem presence (IM&P/CUP) on a non-redundant set of Expressway-E and C VMs. Deploying with WebEx Messenger is not covered here, but the bulk of the configuration is the same as far as the Expressway piece.

The biggest challenge in the initial deployment was finding all of the necessary documentation! Things you need to know like certificate chaining, or OpenSSL are in various docs. I’ve linked all of the documents that I used and tried to summarize things to make it quicker to deploy.

Updated jabber-config.xml on CUCM with RemoteAccess turned on (This is no longer required for Jabber 9.6+)

Two certificates (one for VCSe another for VCSc) – either signed by your own CA (OpenSSL or similar) or publically signed certs like GoDaddy, Verisign, etc.

A few notes about nomenclature:

Collaboration Edgeis the architecture umbrella term for the VCS/Expressway edge proxy for CUCM-registered clients (Jabber and TC7.0 TP units). It’s commonly used to refer to the Jabber piece of it, but will support endpoints too. (The DX650 will support Collaboration Edge in a future release of firmware. Traditional IP phones will not be supported, they will use VPN Phone or CUBE lineside proxy.)

Mobile and Remote Access (MRA) is the term used in VCS/Expressway documentation for the VPN-less Jabber (and CUCM-registered TC7.0 endpoint) proxy feature.

Cisco Expressway-Edge is the same software as VCS-Expressway, just packaged for CUCM registered endpoints. Expressway-Edge is a VCS-Expressway that is deployed as a Mobile and Remote Access proxy or for traversal calls for CUCM registered endpoints. There is a license file actually changes the title to say “Expressway-E” when it is loaded. In the rest of the document, I will refer to VCS-Expressway, VCS-E, Expressway-Edge, Expressway-E as Expressway-E since we are primarily talking about MRA.

Expressway-Core is the same story. It is VCS-Control software deployed as an MRA proxy only with the Expressway-C license loaded. We call it VCS-Control when it is licensed for device registration, non-traversal calls, FindMe, and other features like Lync interop. For purposes of this document I’ll call it the Expressway-C below.

Customers with a valid UCSS contract for UCL-Enhanced, CUWL-STD, or CUWL-PRO are entitled to Expressway-Edge and Expressway-Core for free (for MRA) and their license will reflect the Expressway names. Licenses are charged for the other VCS features mentioned above. Licenses are required and have a cost for B2B/B2C (Jabber Guest) calls through Expressway-C/E Each box requires one media session license to get a session through.

VCS-C and VCS-E can have the MRA features turned on and run on a pair and do both functions. We are still awaiting clarification as to when you must break these apart and run a separate set of VCS (for B2B, interop) and Expressway (for MRA) servers. (Update: VCS is supported for limited sized deployments.)

Expressway licenses should be orderable via PUT, or you can use your existing VCS severs by upgrading to X8.1. (Update: I posted a later post that discusses what to order.)

2) Decide if you want to deploy valid security certificates on CUCM, IM&P and CUC. You will likely want to do this independent of Collaboration Edge as all of the Jabber clients are no longer trusting self-signed certificates. By providing a publicly trusted cert, Jabber won’t throw Invalid Certificate errors as you log in. Granted they are only shown once during the very first login if the user accepts them on each client. If you do put certificates on those components I’d suggest getting them for a 5-year term so you aren’t dealing with it in a year when the certificates expire.

Directory Lookup Considerations

MRA only supports UDS as the directory lookup service. If you are inside you can use LDAP (EDI/BDI), outside UDS.

Jabber-config.xml Update for 9.6 clients

Update: This section is no longer required as current versions of the clients (Win 9.7, iOS 9.6.1, Android 9.6, OS X 9.6) to do MRA by default, negating the requirement to pull the jabber-config.xml file first (expect in the case of split internal/external domains).

CUCM UC Service Profiles

Make sure that you’ve configure CUCM UC Service Profiles (this should have been done as part of your initial IM&P/CUP deployment and won’t be covered here) and assigned them to the end users.

Recall there is a single OVA that does both Expressway-E/C and VCS-E/C that you need to deploy — it’s just a matter of how you configure and license (request via PUT as mentioned above) it as to what it is called. Download the OVA from Cisco here

You’ll deploy the Expressway-C on your internal network (presumably on the same VLAN as CUCM and other UC components) e.g. 10.10.1.30

You’ll deploy Expressway-E one of two ways. Either on a stick in your DMZ (perhaps 10.99.99.30), or two-legged with the external interface in the DMZ network (e.g. 10.99.99.30 – or on your public address space), and the internal interface on the internal network (presumably on the same VLAN as Expressway-C and other UC components – e.g. 10.10.1.31).

You’ll need to trunk your DMZ to your ESXi host if you haven’t, or figure out how to deal with getting the Expressway-E external (LAN2) interface in the DMZ network.

Once the VM’s are deployed edit the Expressway-E VM settings to put LAN2 in the DMZ network (alternately you can put LAN2 on the inside and LAN1 in the DMZ). If you don’t see options for the second NIC, or options for NAT, you are missing the Advanced Networking license. You need this in order to have it two legged, or do NAT.

Firewall Configuration

I don’t know how long I wasted on my first install because I forgot to modify the ACL to include the additonal ports that MRA requires. I just assumed that beause VCS was working for B2B that it would work for MRA. Not the case!

You’ll need to configure NAT on your firewall from a public IP address outside your network to the DMZ address of Expressway-E, or do 1:1 public to your DMZ if you’ve deployed it with a public address.

You will need two DNS servers for MRA to function properly. Jabber decides if it is inside the network or outside the network depending on what SRV records it can resolve. Depending on what records it resolves it will either try to use MRA or it will directly connect to CUCM/IM&P.

Internal DNS Server

Create two A records:

sjc-expressway-edge-01.domain.com A – (make this name whatever you want) Pointing to the INSIDE interface of Expressway-E for two-legged deployments, or pointing to the DMZ address if it’s on a stick. The record is used by Expressway-C to lookup and validate the certificate against. You will use this hostname anywhere you are asked for the expressway server’s name whe configuring the C server.

sjc-expressway-core-01.domain.com A – (any name you want) Pointing to Expressway-C.

Follow this chapter of the Expressway Admin Guide – Mobile and Remote Access (feature preview) beginning at p.52 but stop half-way down p.56 (before the beginning of the Certificates section).

A couple notes: I did not enable TLS verify mode on my CUCM and IM&P server definitions because just wanted to get it up and running. I’m suggesting putting real certs on CUCM, IM&P, and CUC, and turning TLS verify on, but this can be done later.

Valid CA-signed certificates are required to setup the traversal zone for MRA. You can either get public ones, or sign your own with your own CA. I’ve done it both ways. The major reason for a valid trusted CA-signed certificate is to stop Jabber from throwing a certificate warning on the initial MRA login to Expressway-E itself. I highly recommend deploying a publicly trusted CA signed certificate.

I used Chrome on Windows to export the three Go Daddy certificates individually to Base 64 .PEM and then loaded them into the Expressway-E/C trusted CA list. This worked perfectly for me after loading and rebooting the servers. The UC traversal zones came right up.

Sign your own using OpenSSL if you’d like

If you want to use OpenSSL to create your own CA cert and sign your CSR, it is actually easier than you’d think.

You’ll follow the procedure twice. Once for Expressway-E and once for Expressway-C. Take the CA root cert that you generated and import it into the trusted list on both boxes, and then import your signed server cert on the appropriate box.

Traversal Zone Configuration

Resume the configuration tasks in the Admin guide on p.58 making sure to put the proper settings for both Expressway-E and Expressway-C.

If your certificates are good, you will see the traversal zone go active on both servers under Status | Unified Communications. If not, double-check your configuration settings, and double-check your certificates.

Troubleshooting Zone Configuration

If the zone won’t go active and you think it looks good, check the logs to see what is happening. My initial attempt where the certificates were not chained properly showed a continuous loop of TLS failures. When I had my Expressway-C pointing to the external public address instead of the inside interface of Expressway-E, TLS looked good and even the SSH tunnel showed “up” but traffic wasn’t actually flowing.

The best place I found to troubleshoot this stuff was by putting the Expressway-C and E in “Devel mode” to enable the Experimental menu. (Instructions for this are found on p.207 of the admin guide.) The reason for this is because the CollabEdge/MRA feature is still considered experimental. You need to look at the Developer Logs. You can enable them for debug level as well as collect a tcpdump.

HTTP Whitelisting

Make sure to add your Unity Connection, and any other servers that Jabber needs access to. Unity Connection requires it for Visual Voicemail to work.

Launch Jabber 9.6 internally

Update: No longer required unless you are doing separate internal and external domains. (I’ll detail this in a later post.)

Launch Jabber on your client device on the inside network (so that it has direct access to CUCM/IM&P). When you enter your email address Jabber should automatically discover your servers (using the before setup internal DNS SRV records). If Jabber does not auto-discover, troubleshoot your SRV records. The easiest method is to use dig or nslookup.

Enter set type=SRV, then type _cisco-uds._tcp.domain.com. This should resolve to the hostname of CUCM, or the IP address of it.

If using the hostname, exit nslookup and try to ping the hostname.

Once you enter your credentials you will likely be presented with several invalid certs to accept, and your client should connect and have IM, Presence, CUCM, CUC, and be able to IM and do voice/video calls.

Sign out and close Jabber

Launch Jabber 9.6 externally

Disconnect from your internal network and make sure your device is outside your network where a) it cannot resolve the internal SRV records, and b) it can resolve the external _collab-edge SRV record and access your Expressway-E from the outside.

Launch Jabber on your device. Jabber will attempt to resolve _cisco-uds._tcp.domain.com and will fail to do so. It will also attempt to resolve _cuplogin._tcp.domain.com and will fail. It will then attempt to resolve _collab-edge._tls.domain.com and get pointed to the public IP address of Expressway-E.

It will then connect to Expressway-E, and a if everything is configured properly it will login and you’ll show connected to IM, CUCM and CUC!

Notes about iOS

On iOS the timeout for attempts to login is MINUTES long. Be very patient for it to either succeed or fail. It can take a significant amount of time to login successfully on the 9.6 build. 9.6.1 is supposed to be much faster.

If your login fails, click the Send Error Report and email it to yourself. Open the ZIP file and look through (going from bottom to top) to see where the errors are. The logs will include more than just the current login attempt, so note the time when you are attempting to login and look at the timestamps in the log. This is critical so that you aren’t troubleshooting an old login that isn’t relevant to your current problem.

From my experience:

When I had firewall issues, I was seeing CONNECTION_TIMEOUT errors when trying to login via MRA, but not when I was inside.

When I had neglected to enable RemoteAccess in jabber-config.xml I was seeing RemoteAccess Policy errors.

Summary

I’m impressed with the ability to finally be able to do voice/video calls from anywhere! It’s about time. Collab Edge is still considered a Feature Preview by Cisco and isn’t TAC supported yet. Please send me questions that you have as you attempt to deploy it.

Have you fought the fight to setup ASA phone-proxy (back in the day) or VPN Phone on the ASA? While VPN Phone was a solid solution, it could be a bear to setup and troubleshoot.

So, file this in the how did I miss this folder, but CUBE 10.0 (IOS 15.3(3)M) on the ISRG2 and ASR supports line-side SIP proxy for all of the current generations of phones (69xx, 79xx, 89xx and 99xx). So CUBE can do the dirty work for you.

I’ll set it up soon and document my experience with it once I get my CollabEdge guide out, but here is more information:

We don’t yet have every app supporting 5.5 (or even 5.1). All apps support 5.0. Here are the ones who are supporting 5.5 (for which versions, check their page on http://www.cisco.com/go/uc-virtualized)VMW-VS5-HYP-K9 and R-VMW-UC-FND continue to ship a “5.x license” and 2 media files (5.1 and 5.0). PUT for 5.5 not complete yet (ETA TBD). Plan for 5.5 TBD.

5.5 breaks backwards compatibility with older native OS versions relative to 5.1. E.g. 4.0 thru 5.1 can support UCM 8.0.2 thru 10.0, but 5.5 can only support UCM 9.x and 10.0. Don’t assume every version will work with 5.5. I am emphasizing this because for ESXi 4.0 thru 5.1, every release of UCM supported every ESXi release. For first time that isn’t true so watch out.

5.5 increases the minimum physical RAM required. 4.0 thru 5.1 only required 2 GB, but 5.5 requires 4 GB. All of our TRC builds should still be ok but if you are doing Specs-based remember this. Sizing Guidelines page and upcoming UC on UCS mini-SRND call this out.

For UCM 10.0 with ESXi 5.5, read the OVA readme for some migration considerations due to the native OS changes we made in 10.0.

9.4(1) Firmware was just released for the 8941/8945. This release adds several features that the platform has been needing for sometime. The two major features that customers have been asking me about are Electronic hookswitch capability, and Firmware file sharing.

Please note, the firmware is the first build in the 9.4 train, and shows a significant number of open caveats.

Major Features

Electronic Hookswitch has been sorely needed. The addition allows for certain Platronics wireless models to answer the phone from the headset without a lifter. – From Plantronics: Announcing New Cisco EHS Cables
APC-42 EHS for Cisco 6945 phone will be available at the end of February 2014. APC-82 EHS cable for Cisco 8945 phones, samples and customer shipments were available as of January 2014.

Peer Firmware Sharing allows for an 8945 at a remote site to cache and share its firmware with other phones at that site. This is a huge feature for customer upgrading firmware at sites that have multiple phones over slow WAN links. This will save a significant amount of bandwidth and time in upgrading remote sites. (This feature requires the latest Devpack be installed on CUCM to enable CUCM to take advantage of that feature on the phone firmware.)

Gateway Recording for SIP –One major feature in CUCM 10.0 is the ability for CUCM to fork calls to the recording server from teh gateway rather than the BiB in the phone. This makes sense in cases where there are remote phones, or phone models that don’t have a BiB. The voice gateway will need to be converted from h.323 or MGCP to SIP to take advantage of this feature, and all phones will need to be upgraded to a firmware level that supports it.

Upgrading – Use Caution

I followed the typical – OS Admin install cop file, restart TFTP, and reboot a test phone, then reboot all of the phones process.

Unfortunately I had several phones completely hang and get stuck in a loop upgrading. The fix required physical intervention. Several phones, after initiating the reboot, got stuck at the “Upgrade in Progress” screen. The phones in this state quit doing CDP and were no longer in the CAM table of the switch they were plugged into, so I had no way to figure out which ports they were plugged into remotely to shut them off. They had to be physically bounced. The problem is that they would go right back to the “Upgrade in Progress” hung state. Infinite loop…