Yet another blog about networking

Menu

Getting Used to Cumulus Linux – Linux as a Network OS

A few weeks ago I started at Cumulus Networks doing pre-sales and consulting. My background is almost entirely Cisco based with only a generic user-level knowledge of Linux. I wanted to start a series describing some of the things that have struck me or I’ve struggled with in learning my way around a switch running Cumulus.

In this first post I wanted to describe just what is Cumulus Linux?

The Operating System

Before Cumulus Linux makes sense you have to look at where network operating systems (NOS) came from. If we look at NX-OS, the NOS that runs on Nexus platforms, it (very) roughly looks like this

In NX-OS you have an operating system based on Linux, however you interact with with a management layer that hides Linux and the guts of the system from you. Because of this you don’t have many options for changing things. If you want a feature or application and it’s not already in NX-OS, then there isn’t anything you can do about it. For example, if you really love Ruby and want to write scripts on the box in Ruby you can’t. The Nexus 9k comes with Python, but not Ruby.

Cumulus Linux on the other hand isn’t based on Linux, it is Linux.

This doesn’t look much different than the NX-OS image above, but what is different is that Cumulus acts as a 48 port server. You log into the bash shell and have Linux. Need a software update? apt-get upgrade. Want to start writing automation scripts in PHP? apt-get install php5

These are some trivial examples, but where it gets really interesting is that any tool that exists for Linux server management can now be applied to the network. The entire Linux ecosystem is now supported on your switch. Any tool, package or software is yours to use. What I personally think is interesting are not the examples I can come up with here, but that users aren’t limited with what they want to do on their switches.

The Hardware

Cumulus Linux is just a piece of software. That’s it. If we could get away with servers running our datacenter networks end-to-end it would have been done years ago. Unlike Arista or Cisco, Cumulus doesn’t make hardware. Cumulus relies on hardware partners like Dell to build the switches that run our software. Just like servers. RedHat doesn’t make servers, they make software that runs on any server you want.

Inside the partner switches it’s the same Broadcom chipsets that run the Nexus 3000, 9000 and Arista platforms. Just like RedHat doesn’t care if it’s an HP or Dell server as long as it’s x86, Cumulus doesn’t care if you bought the switch from EdgeCore or Penguin Computing, as long as it’s Broadcom it can run Cumulus Linux.

What Cumulus does that other Linux distros don’t do is the magic to take routing and switching information from the kernel and push it into the Broadcom chipset. Without this piece you wouldn’t have hardware acceleration allowing for multiple ports of 10 or 40 gig.

Summary

Cumulus is just another version of Linux that has been modified to power Broadcom ASICs. Now the switch looks (and more importantly, is managed) just like your servers. Unlike traditional Network Operating Systems you don’t have a walled garden where it’s built on top of Linux but you don’t really have control. As Strech from Packet Life put it, it’s “unapologetically Linux”.

5 thoughts on “Getting Used to Cumulus Linux – Linux as a Network OS”

Nice post, Pete. Is this only for switching or do they do routing as well? The Linux suite of routing is a bit lagging behind commercial options (imo) especially since it seems to be lacking MPLS support.

Quagga is packaged with the switch, so you have BGP and OSPF routing functionality as well. Cumulus has made a ton of changes/fixes to Quagga to make it more robust and scale that we are trying to upstream, but we are moving a little faster than Quagga mainline at this point.

MPLS is definitely a place where support is lacking, but it’s something that’s on the radar as a customer request (and the hardware can support it, so it’s only a software issue). Same thing for multicast.