Sunday, January 23, 2011

How many times have you had to fill out a change control document to upgrade code on your network devices where you've detailed the redundancy, portions of the networks impacted, application owners notified only to have it rejected due to "impact"? Prior to my current job at Cisco, this was a common theme. I wished I had a device that would let me roll code without impacting traffic. Fast forward a few years and my wishes have come true with In Service Software Upgrade (ISSU) within NX-OS.

A brief history lesson - Storage switches have had this capability for a long time in the higher end platforms that are considered director class. It makes sense to have ISSU functionality on fibre channel switches because fibre channel as a protocol relies on the network to guarantee delivery of frames. Dropping frames means bad things for storage traffic. Moving the capability for ISSU to Ethernet/IP networks makes sense in a modern data center where high density virtualization and the "always on" mindset prevail. Networking teams have been clamoring for ISSU for a long time. Let's face it, rolling code isn't one of the more exciting things to do on a network, but it's a necessary function, good news is that we now have it.

We'll focus on ISSU on the Nexus series of devices though know that other products in Cisco's portfolio support it. To provide a hitless upgrade capability the device and software require an intrinsic separation of the control plane and data plane. This allows changes to be made in the control plane, like software version, without affecting the data plane, through which the packets and frames that traverse the device pass. NX-OS has been engineered from day one to have this separation of planes. Coupling it with years of experience in ISSU on the Cisco MDS and one of my most favorite features of NX-OS is born.

So enough talk, let's get into the action. To start an ISSU we use the install all command as shown below where we specify the kickstart image and system image to use.

Once that is completed, the install routine also shows the type of upgrade per module, reflecting a rolling upgrade for line cards and reset for the supervisors. Rolling upgrades are non-disruptive as the modules have been engineered to provide this functionality and not drop link to ports or disrupt switching.

Compatibility check is done:

Module bootable Impact Install-type Reason

------ -------- -------------- ------------ ------

2 yes non-disruptive rolling

5 yes non-disruptive reset

6 yes non-disruptive reset

9 yes non-disruptive rolling

Finally, a nice table is presented showing the details of the upgrade and waits for the green light to continue.

At this point, the supervisor that was the secondary (module 6 in my example) has reload and come up with the new code. This triggers the primary to initiate a Stateful Switch Over (SSO) to the new code running in the control plane. Meanwhile, data is still traversing the switch with no impact. J

Since our telnet session was disconnected during the SSO (telnet isn't SSO aware), we need to re-establish the session and issue a command to continue monitoring the upgrade.

rfuller@cmhlab-tools:~$ telnet cmhlab-dc2-sw2-otv1

Trying 10.2.0.4...

Connected to cmhlab-dc2-sw2-otv1.csc.dublin.cisco.com.

Escape character is '^]'.

User Access Verificationlogin: adminPassword:Cisco Nexus Operating System (NX-OS) SoftwareTAC support: http://www.cisco.com/tacCopyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained in this software areowned by other third parties and used and distributed underlicense. Certain components of this software are licensed underthe GNU General Public License (GPL) version 2.0 or the GNULesser General Public License (LGPL) Version 2.1. A copy of eachsuch license is available athttp://www.opensource.org/licenses/gpl-2.0.php andhttp://www.opensource.org/licenses/lgpl-2.1.php

cmhlab-dc2-sw2-otv1# show install all statusThere is an on-going installation...Enter Ctrl-C to go back to the prompt.Continuing with installation, please wait

Module 2: Non-disruptive upgrading.-- SUCCESSModule 9: Non-disruptive upgrading.-- SUCCESSInstall has been successful.With that, we've upgraded our NX-OS, had the system automatically copy the files to the right locations, modify the boot values and didn't drop a frame. How's that for hot?

cmhlab-dc2-sw2-otv1# show ver i uptime

Kernel uptime is 0 day(s), 0 hour(s), 26 minute(s), 50 second(s)

*NOTE* The Kernel has been up for just a while but we'll see that the overall system has been up much longer

Tuesday, January 18, 2011

I finally decided I needed to do some blogging, so here we go. Before we get into the fun stuff, let's talk a bit about who I am. This will help you decide if you are in the right place or not.

My name is Ron Fuller and I work as a Technology Solutions Architect with Cisco in Dublin, Ohio. I work with our Enterprise customers on data center architecture, which means I'm not a product guy per se. Architectures can be enabled by a product or suite of products though I happen to think some enable it better than others. ;) I am a dual CCIE #5851 (Routing and Switching and Storage Networking) and have held a myriad of certification from other vendors including Novell - where I started my certification track and was a Master CNE, VMware, SNIA, Microsoft, HP, Okidata, IBM, ISC2, CompTIA and more. Certifications have been a focal point for me early in my career and certainly opened doors that would have otherwise remained closed in tough times.

I have had the opportunity to be published a few times and my latest effort was a collaboration with two great guys who I am lucky to call friends as well, David Jansen and Kevin Corbin. We created NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures with CiscoPress. The book was released last June and we're already working on a 2nd Edition because of the many changes and innovations NX-OS has brought to market in the last few months and those coming! I have a passion for NX-OS and if you've been following me on Twitter (@ccie5851) you might have picked up on it. ;) I have a sticker on my laptop that says it all.

On a personal front, my wife and I have four awesome, smart, creative, cute....you get the picture...kids. We live north of Columbus OH and love to travel- WITH the kids - especially if there is a F1 race involved. We've become very adept at long haul travel with kids and have taken them with us to Japan, England, France, Germany, Australia and our last big adventure, China. I may blog about the science of traveling with little ones in the future. We think we've got a good system but may be biased.

As I mentioned earlier, F1 is a great excuse to travel and for that matter, I'm a fan of most autosports though F1 holds a special place in my heart. It is the perfect integration of technology (I'm a geek after all!) and speed, exotic locations and competition. I do watch Indycar and it's probably best to say I monitor NASCAR. NASCAR has so many races and they are so long that it becomes quite the commitment to actually WATCH every race. I still miss the days of Dale and Rusty beating and banging on each other, but as with all things, change happens.

I'm sure more of my idiosyncrasies will emerge as I write, but know that I plan to discuss NX-OS and Nexus switching, some UCS action, MDS and whatever else comes up. Its an exciting time in the Data Center space and I couldn't be happier to be hip-deep in the action!

Newest LiveLesson!

vExpert 2017

About Me

Field Engineer at VMware focused on NSX though blog posts are all my own. Husband, father, F1 fanatic and geek.
Ron Fuller is a Staff Engineer in the Network and Security Business Unit (NSBU) focused on NSX for VMware. He has 22 years of experience in the industry and has held certifications from VMware, Novell, HP, Microsoft, ISC2, SNIA, and Cisco including two CCIEs No. 5851 (Routing and Switching/Storage Networking). His focus is working with customers to address their challenges with comprehensive end-to-end Data Center architectures and how they can best utilize VMware technology to their advantage. He is the co-author of the VMware Press NSX Fundamentals LiveLesson video series. This adds to his existing body of work with CiscoPress. He has had the opportunity to speak in Europe, Australia and the United States on multiple networking and security topics. He lives in Ohio with his wife and four wonderful children and enjoys travel and auto racing. He can be found on Twitter @ccie5851.