Behind the Scenes at the Ciscolive! Network Operations Center

I have been privileged to be part of a very dedicated team of networking subject matter experts for the past 5 years in prepping and running the Ciscolive! event network. Before 2008 the network was installed and monitored by a commercial events networking company. However, it became clear with the number of Technical Marketing Engineers, Services Engineers and product developers on-site for Ciscolive! that we were well equipped to take on the responsibility in addition to our speaking responsibilities.

Planning

The preparation for Ciscolive! starts many months before the event with weekly, collaborative Webex sessions to discuss design criteria, venue particulars and product/feature configurations. Generally the teams align into core Route/Switch, Data Center, Security, Wireless, and Network Management/Operations, however we work collaboratively to design and implement the best show network possible. We have individuals that are part of our Cisco Remote Management Services (RMS) team, Video Surveillance and partner support from our CenturyLink Internet Service Provider and NetApp storage vendor.

A small part of the NOC staff will travel to the venue to survey the wireless, power and Internet service capabilities several months beforehand.

The event network is typically staged in San Jose. Much of the team engages remotely, some internationally, however the San Jose personnel are able to physically visit the equipment. We configure the distribution and access-layer devices, along with wireless LAN controllers and APs with as much practical staging configuration as possible. However, the event network can be dynamic, so we have to be prepared to make mass changes.

A month out Joe Clarke, a peer Distinguished Services Engineer, and I started to install and configure the various network management and support service applications. With an abundance of UCS compute and NetApp storage, we tend to run many Cisco management applications, including beta releases. However, some of the primary applications we deployed were:

Other NOC SMEs in Security, Video Surveillance and Unified Collaboration also install their own monitoring and management applications in our UCS/NetApp environment.

The core network is staged and stays up until shipment. The wireless access points and access switches get staged, but then are shut down so we don’t have hundreds of devices crowding a lab. Typically 10 days from the start of Ciscolive! the equipment is shut down and loaded onto trucks for transport to the venue.

Onsite

Although the show was to run June 23 – 27, we needed to be “show-ready” by 3pm on Saturday (6/22) when registration opened. I arrived Tuesday (6/18) morning with the plan to start working in earnest on Wednesday. However, our 6-rack core network was already staged in a conference room and I thought you might as well jump in!

The conference room would later be reconfigured as a training breakout room. With ‘behind-the-scenes’ access I’m often impressed to see how the venues can bring in large, mobile power panels to serve the requirements of our equipment. We’re not talking simple 3-pin NEMA 5–15 outlets here!

Much of the wireless APs, antennas and pole mounts were still in boxes and needed to be broken down. We had nearly 2 Million square feet of floor-space in the Orange County Convention Center. This venue was 30% larger than the previous year’s San Diego Convention Center. There would be a LOT of walking to deploy and fix misconfigured equipment. We relied heavily on the Cisco Academy ‘Dream Team’ of around 40 college students to help us with configuration, device placement and troubleshooting. It was very rewarding to spend time mentoring students that are just getting started in their careers. Thanks Cisco Network Academy team!

For the next several days we worked long hours to ensure the network was configured to the topology seen below.

The main network was in the Orlando County Convention Center (OCCC), however we also had a network extension to the Peabody Hotel next-door for executive meetings. This was the first year we extended the wireless network into the World of Solutions and keynote halls.

Some breakout rooms had to be visited to troubleshoot why some CTS systems were not registering. Oftentimes the problems were related to port configuration changes, physical patching or incorrect trunk configuration. We also identified and remediated a potential spanning-tree loop before it became service-impacting to the show.

As mentioned earlier, it’s interesting to get ‘behind-the-scenes’ of the show. The main key-note area has some incredible audio/visual technology behind the main stage to serve the venue audio, ultra-wide screens, image magnification and demos.

Sometimes you have to go to lengths to ensure network connectivity. We had 8 wireless access points being served PoE and network connectivity by two Catalyst 3560CGs in the 2,600 seat Chapin Theatre. The APs and switches were on a catwalk allowing access to the stage lighting, 60 feet up. Anyone afraid of heights AND the dark need not apply.

Ken and I hit the catwalks to fix some switch connectivity.

Extreme console connectivity

The 6 racks of equipment that was temporarily staged in a breakout room was our core network and compute/storage for the show network and services. Thursday evening we shut it all down and moved it into the World of Solutions in Hall B. Onsite attendees may recall this equipment was located in a miniature, enclosed data-center in the rear of the World of Solutions at the Network Operations Center. OCCC Hall B did not get air-conditioning until late Sunday in preparation for the show. During this time the loading bay doors were open to allow fork-lifts and other machinery to bring equipment in and out.

The racks were put into the ‘plexiglass data-center’ with air handlers because it was more economical to create a mini cooling eco-system, rather than turn on A/C for the whole World of Solutions with the loading bays open.

The last couple days before the show started was spent updating port configurations and ensuring our data collection was started. In events such as Ciscolive! the room configurations can be dynamic and require us to do switchport changes to accommodate different needs. We may need to add access points, digital signage or accommodate a router for VPN purposes. This year we used our Cisco Service Portal (CSP) and Cisco Process Orchestrator (CPO) tools from our Intelligent Automation portfolio. Rather than train up the Cisco Academy volunteers on Cisco Prime LMS/Infrastructure we optimized port configuration changes by developing a simplified and customized portal in CSP that linked with Cisco Process Orchestrator to automate the changes using Cisco Prime LMS.

The result was a system that extracted a list of switches we were managing, presenting them in a drop-down selector. The NOC staffer would then see a dynamically updated list of switch ports, based on the port count of a Catalyst 3560CG or 3750-X. They could select one or more ports, then a port profile, such as ‘Wired Access, Wireless AP, Digital Signage, IP Phone, IP Camera, or Trunk’. A list of VLANs was presented and a data-retrieval rule showed the application of the VLAN with name, service, location and IP subnet it served. Once the ‘order’ was submitted CSP would transfer the selections to CPO. CPO would build the ‘job’ that could be consumed by Cisco Prime LMS, then monitor the status of the job, finally sending an email to the NOS staff.

We deployed a couple hundred port changes throughout the event.

Circle back in a couple weeks when I’ll have another blog about what happened during and after Ciscolive!.

Apologies for that, Erik! There was a finite amount of screen real-estate to work with. :)
I updated the blog post. If you look at the image now, right below is a hyperlink to a PDF of the network topology. Hope that helps!

Hello Anonymous,
ISE would have been used if we were providing differentiated services to users by their 'classification' - eg. Event Staff, Attendee, Speaker, World of Solutions Vendor, etc.
However, we did not break down traffic and access (policy) by user. That would have required device registration and assignment into policy groups (and AD groups). We were running the network pretty open as far as 'registration' went. You'll note there was no captive portal/BBSM-like service in place. This was intentional.
We did secure the network by VLAN/subnet. So if you were on the wireless 'ciscolive2013' network (and the associated VLAN), you were restricted from the show/NOC service network by the ASA. Other networks were only allowed to the Internet and no where else.
CiscoSecure ACS was used to provide the AAA/TACACS+ logins to the routers/switches and centralized login authentication to the various management applications that could source AAA/TACACS+.

Some of the individuals posting to this site, including the moderators, work for Cisco Systems. Opinions expressed here and in any corresponding comments are the personal opinions of the original authors, not of Cisco. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Cisco or any other party. This site is available to the public. No information you consider confidential should be posted to this site. By posting you agree to be solely responsible for the content of all information you contribute, link to, or otherwise upload to the Website and release Cisco from any liability related to your use of the Website. You also grant to Cisco a worldwide, perpetual, irrevocable, royalty-free and fully-paid, transferable (including rights to sublicense) right to exercise all copyright, publicity, and moral rights with respect to any original content you provide. The comments are moderated. Comments will appear as soon as they are approved by the moderator.