Days in the life of a professional packet shepherd.

Cisco CSR1000V on VMWare Fusion

The release of the Cisco CSR1000V is definitely garnering some buzz around the Internets. The platform is intended for deployment in a VMWare ESXi environment, but it seems to work just fine using VMWare’s desktop virtualization products as well. Join me as I boot up a couple of cloudy routers on my Mac Mini! The CSR platform was announced at Cisco Live 2012 in San Diego. It’s finally been released and already tutorials and information are popping up on the ‘net. If you have a Cisco.com account, you can download the software here. You can also check out some tutorials such as this blog post from INE on Installing the Cloud Services Router in VMWare ESXi, or this work by Daniel Dib on integrating the CSR1000V with Dynamips. As the CSR is distributed in both an OVA format for import into ESXi, and also in an ISO format, I figured I’d see if the ISO would install under VMWare Player on my work laptop. It did, and seemed to run fine. So this evening, I’m going to try it on VMWare Fusion 5 running on my Mac Mini (quad-core i7 and 16GB RAM, so we should have plenty of resources to run a couple of them).

An Easier Install than Turning Rack Screws

After downloading the image from Cisco’s site, we need to create a VM. I go through the normal VMWare Fusion new machine setup, with the exception of customizing a few settings at the end. Cisco dictates the VM for the CSR must have 4GB of RAM and 4 cores. I will oblige. First, select the ISO image for installation: Next, we select the OS type. It’s well known that the CSR (and all IOS XE) runs on a Linux base. I’m assuming it’s a 2.6 kernel. It seems to work. After clicking “Continue” above and seeing the machine summary, I edit the virtual machine settings to set the CPU and memory per Cisco’s specs: By the way, when they say 4GB of RAM, they mean it. Three of these things ate up a solid 10GB of RAM on my test machine. The VMX processes were using about 2.9 GB of RAM at idle. Next, we set up the NICs. The Gig0 interface on the CSR is automatically put in to a management VRF called “Mgmt-intf” and as is typical for management interfaces on things like the ASR it seems impossible to remove it from that VRF. So that Gi0 NIC will always be an OOB management interface. For all of my example devices, I bridged this onto my Ethernet adapter and just let it pick up DHCP. I then hit the “Add Device…” button and added another network adapter. This one, I attached to a “host-only” network that I created in Fusion called vmnet3. This will act as one of the links to another CSR in my lab. By the way, a graphical means for creating new VMnet segments is finally available in Fusion 5 via the application preferences.

Fire It Up!

Once the VM is configured, hit the “Power On” button. We see that the CSR posts and then loads GRUB:

Booting these things seems to take a while. Particularly the first boot, as it’s actually installing from the CD image. After a while it will spit out some console output:

Following the install process, the VM automatically ejects the disc image, reboots, and then chugs a bit more. After some patience, we are rewarded with a familiar screen:

You can see above that the Gi0 management interface actually got DHCP from my server already. Lo and behold, that most basic of tests seems to work:

So there we go, a virtual IOS-XE router running in VMWare Fusion, sending pings to the universal ping target.

Rinse, and Repeat

One router is fun to poke around in, but it takes more than one router to make a network. At this point, I shut my router off, and in Finder simply made a few copies by holding Option while dragging the icon to make a new one. I added -1, -2, and -3 identifiers to them. Note that the complete VM is about 650MB.

I went back into VMWare Fusion, and went to File > Open… to re-import the new VMs. I went into each and added the -1, -2, and -3 to the names. I also reassigned some NICs so my topology went like this:

I booted all of the routers. You can see in this screenshot that the UDI of router 2 was changed. The UDI is basically the identifying element that licenses are based on (much like a serial number), so if you run off copies of a router they must be licensed individually. More on licensing at the end of this post.

Remember what I was saying about memory? Here is my machine with all 3 CSRs booted:

Ouch. Anyway, once all the routers were up, I put some very basic config on each (like an enable and line password), then used a normal terminal to telnet into each on their Gi0 management interfaces. Success!

Back to Text Mode

Now that you’ve seen all the routers get built and booted, and we’re into telnet on each of them, I will stop with the screenshots and just show normal terminal output. By the way, if you don’t want to use Gi0 for management, you can add a serial port to the VM and run the command “platform console serial,” and the console output will go to the serial port instead of the VMWare virtual console. I didn’t mess with that in my lab. Apparently in the fancy version of ESXi vSphere you can easily provide console server functionality in which case this might be a good feature. Once I got the routers up, I noticed that CDP is disabled by default. I’m assuming maybe they disabled it to minimize control-plane activity that would chew up virtual CPU, especially since the CSR is kind of intended for an environment that it may not really have a lot of neighbors to be sharing link info with. Once I enabled CDP globally and on each interface (yes, both were disabled by default), we got CDP info:

Closing Thoughts (and maybe a bit more)

I’ll be perfectly honest: I have a relatively hard time wrapping my brain around the supposedly myriad uses for the CSR platform. I can understand a use case for a public cloud type thing where you have an instance acting as an MPLS CE node for WAN connectivity or a VPN connection back to a main data center. Maybe that’s pretty much what it’s for in most cases. Extending MPLS switching into the cloud data center and LISP and stuff like that is a little outside of my grasp to see how helpful it would be.

I have a couple of customers looking into public cloud “stuff” where perhaps the CSR could offer a more attractive option for connecting the cloud servers back to the customer’s network than either a provider-based solution (like the Amazon EC2 VPN) or some other Linux-based firewall solution like PFSense being used to generate a VPN tunnel (both of which I’ve recently come across in my day-job).

Honestly, I’m much more interested in the CSR as a flexible lab platform. For years now, network engineers have had only a few options for Cisco labbing:

A physical lab – Real world behavior, but expensive to build, power hungry, noisy.. Still has potential issues with Cisco licensing, and with the new ISR G2 license model, real-gear labbing with advanced features could become quite difficult.

GNS3/Dynamips – Actually can work very well and the non-Cisco extensions for running other types of hosts using QEMU and VirtualBox are an excellent edition to GNS3. Unfortunately, Dynamips is reaching the end of the road with IOS 15.1, as it only emulates MIPS platforms like the 2600, 3600, 3700, and earlier 7200 NPEs.

IOU/IOL – I’m not going to go into great detail about Cisco’s internal builds of IOS for Linux and Unix platforms (Google can help you if you’d like more info) but the difficulty with these is obtaining them, since they’re internal Cisco tools not made available for public access, and in addition they tend to be buggy on certain features. The only benefit of IOU is that there is a mostly-working layer 2 implementation that can act like a switch. IOU is what is used in the CCIE R&S troubleshooting section (and rumored to be used in the config section of the upcoming version 5), but it’s not a very good option for every day lab work.

None of these options combines convenience, flexibility, reasonable cost, and the ability to use them without being on the wrong side of Cisco’s licensing or even copyright terms.

The CSR potentially does offer this. According to the release notes, the basic install is enabled for 2.5 Mb/s of throughput, apparently with a 60-day full feature trial license. It’s not entirely clear to me what happens after 60-days, whether it reverts to a base license that forgoes some of the advanced features like VPN and QOS, or if it stops functioning entirely until a license subscription is purchased, or if it keeps working full featured, but at the limited throughput level. In any case, having to reinstall a new instance after 60 days to reset the demo timeline is just fine with me from a lab perspective.

I sincerely hope that people won’t abuse this option by trying to run these things in production perpetually using only the demo license, as doing so will quickly force Cisco to drop this option. The 2.5 Mb/s throughput limit would probably also be a big hindrance in real-world usage as most enterprises that would be actually deploying public-cloud infrastructure would probably want more than a couple T1’s worth of throughput, but for lab use that limitation is also really not a problem.

Once an Amazon AMI version is officially released (or if someone were to convert the VMWare image to AMI), you could run a 9 or 10-router lab (using EC2 Medium nodes) for about $1/hr which is much less expensive than most rack rental options. A home lab would probably need to be an ESX box with 32GB of RAM — not a super-thrifty option, but still probably less expensive than building a physical rack of 10 routers. Especially current-generation, IOS XE routers!

I’m looking forward to playing more with the CSR1000V and reading through the configuration guide in greater detail. There are a number of new commands present to handle the virtual nature of this platform and I’d like to understand those. This will also intersect well with my desire to start getting more into VMWare by building a decent home ESXi lab.

The eval license is providing 50 Mbps, at least my version is able to push those 50 Mbps. And it will fall back to 2.5 after the license expires. There are 100 Mbps and 250 Mbps licenses as well (in CCW I can see them but not order them yet) and only with the standard feature set (no encryption, no MPLS, no LISP). With the help of LISP providers could “trade” unused IPv4 space by anycasting certain prefixes out of their own networks and picking host routes or small blocks from that IP pool. This way customers would be able to multihome smaller prefixes than /24.

I really like the CSR1000v and I hope Cisco will minimize its footprint. Four vCPUs and 4G RAM may be too expensive for some customers compared to vShield Edges or Linux-based solutions.

In terms of being suitability for LAB use, I’d really like to see the possibility of using memory overcommitment and optimization techniques such as what GNS3 enables today, such as “ghostios” and “sparemem”. This is one area where GNS3/Dynamips excels, and is how I can develop a good model of a network with 30-35 routers on a decent laptop (Quad-Core/8-16GB RAM), and perhaps a lot more with real server hardware as you mention.

I was excited to hear about this, and planned on using it in a customer’s vCloud lab environment… until I saw the specifications needed and noticed that it downgrades to only 2.5mb after license expiration.
I am still on the fence about use in many vCloud deployments. It is great for an org edge with site vpn, but trying to find a true VSE replacement vm I’m not sure it has the performance needed for the price.
I’d love a pure VSE replacement on the vApp level. Currently I have have vApp reset and start times upwards of 15 minutes per vapp due to the time needed to deploy and boot the VSE appliances. Vyatta is my current goto for a feature rich, low resource, router VM for a since vCloud org.
When you need to have Org VDC network totally emulate a corporate worldwide environment, it get’s a bit convoluted trying to route between vApps, so I do direct/isolated org networks the vApps attach directly to, routed entirely with a single vyatta.

Even with 100Mbps and 250Mbps, I am concerned about performance issues when there should be line-rate throughput.

I just keep hoping the CSR1000V isn’t going to be as bad as the ASA vm is.

I think it’s unrealistic to think that Cisco is going to completely give the thing away. It’s licensable at several throughput levels in the various feature sets. The 2.5 Mb license is intended for PoC testing, and hopefully they will keep that enabled for labbing/test purposes.

Cisco styles themselves a software company (even though they don’t always act like a not-hardware-company) so their position would be that the software that runs in the CSR is the IP they must protect and charge for.

Is Vyatta cheaper? Of course. Is Asterisk cheaper than an Avaya PBX? Of course. Is pfSense cheaper than a Juniper firewall? Of course. I’m using “cheaper” to mean initial capital investment to acquire the product and obviously open source products beat commercial products every time in that single metric. Is the OPEX cheaper over the long run? I’m of the camp that says “probably not” but that’s a religious debate so I’m going to steer clear of that one…

If you can do what you need to with Vyatta, and the vendor support is sufficient for your needs, and you don’t have to worry about corporate standards or sourcing requirements that would prevent a non-Cisco or an open-source solution, then it sounds like you have a good fit.

For orgs that have very large, homogeneous (Cisco) deployments, the CSR fills a gap in that you can configure and manage it exactly like any other router in your environment rather than asking a support team to learn a different product with a different interface, a different (or no) support structure, etc. Is that a good enough reason to shell out for the CSR over Vyatta? For some orgs, yes. For some, probably not.

As I mentioned in the article, while I can see some use cases for the CSR I’m actually more interested in the option for a legitimate virtualized lab environment that doesn’t require emulating old hardware with a glitchy emu or stealing Cisco’s internal tools.

[…] year ago when the Cisco CSR1000V was publicly released, I quickly tested the notion of running the Cloud Services Router in VMWare Fusion on the Mac, rather than on a full vSphere server. Since then, I occasionally see that some readers […]

I am using CSR1000v with IOUs (IOLs actually) and XRv… When ospf is set, everything ping saway except for the CSR1000v…nothing shows up on the ospf nei list either.
checking ospf events i saw that the router is showing “Bad pkt rcvd” which I am still unable to fix..
please help 🙂

I haven’t tried that combination of devices, but I’d certainly troubleshoot it like any other networking issue. Do you get ARP resolution and other evidence of basic layer 2 connectivity? If not, I’d look at encapsulations, VLANs, MTUs, and things like that. You can easily get things like MTU tangled up with multiple virtual networking layers. If you have ARP (which implies layer 2 and IP broadcast), check unicast, and IP multicast. If you are failing one of those, just chase the piece that’s broken. Good luck!