Having fought with Cisco TAC to prove that there is actually an issue they need to deal with, I can say that a transparent, open-source network stack would be vastly preferred. The more eyes on the code, the better.

Ahhhh... Cisco and Cisco TAC. Glad I don't have to deal with either one these days.

I've been on a teleconference with low-level Cisco people where they insulted - severely - a multi-billion (yes that's a B) dollar joint customer of mine and theirs. Cleaning up that mess was my job for a few quarters, of course, Cisco didn't even realize there was a problem.

I now consult for another customer entirely that uses Cisco gear and their customers tell me tales of woes with Cisco nearly every time I talk to one.

The arrogance problem at Cisco is deep and wide. So, while I'm absolutely no fan of Facebook, I think this will be good all the way around.

"We should be able to treat a switch like a server in the rack," Frankovsky said. "We should be able to load a Linux-based operating system, and that server just happens to have a lot of I/O ports on it."

This paragraph sums up what I've been thinking about for a while. If a company is not satisfied with off-the-shelf routers, they can easily build their own running Linux, BSD, etc, but this was never an option for switches. At worst, switches are black boxes implemented entirely in hardware, and at best, you get managed switches that have some knobs you can tweak (which might or might not be the knobs you're interested in).

There's a ton of potential here. Cisco actually charges $10,000 for a switch, comparable to a $173.26 "TP-Link" (TL-SG2424) switch you can buy from Wal-Mart... Of course the latter has crap firmware (software) with a crippled and foreign user interface.

Foundry (now Brocade) has long done very well by making managed switches several times cheaper than Cisco's, which are almost command-compatible with Cisco's (IOS) UI. I see much more Foundry/Brocade branded network hardware in data centers racks, than Cisco gear. But prices could be much lower, still, if no-name Chinese manufacturers could make the dirt-cheap hardware, and a standard firmware could be installed to give them all common functionality and user-interface. On the low-end we have this with APs, thanks to DD-WRT, Tomato, OpenWRT, etc. If Facebook and the OpenCompute project can give a little push, to get this happening with managed switches, the networking world will look very different in just a couple years.

There's a ton of potential here. Cisco actually charges $10,000 for a switch, comparable to a $173.26 "TP-Link" (TL-SG2424) switch you can buy from Wal-Mart... Of course the latter has crap firmware (software) with a crippled and foreign user interface.

What switch are you talking about that Cisco charges $10,000 for that is comparable to a 24-port TP-Link switch? I see the 3560X-24T-L that is comparable. Except, it's $2,500, has nearly double the forwarding capacity, supports Layer 3 roles, 50% larger MAC Table(if doing only Layer 2 stuff).

The fact is the bigger problem is the fact that IOS is a giant kluge that supports doing anything and everything rather decently. But that is a problem for some companies, where they want higher performance in certain ways than others, which they actually are headed towards with the 3560X/3750X and the switch template setups they are doing now.

Of course, none of this even mentions routing, which DCs don't really do much of sides at the core, this is just about the Top of the Rack Switches which DCs are headed towards SDN for, which to me, still seems kind of dangerous and could be a giant risk. So of course a DC switch to people who believe SDN is the next best thing in networking are going to think that stuff needs to be cut from IOS, because for them, it does. Routing needs to go away, Multicast? Who needs that!

There's a ton of potential here. Cisco actually charges $10,000 for a switch, comparable to a $173.26 "TP-Link" (TL-SG2424) switch you can buy from Wal-Mart... Of course the latter has crap firmware (software) with a crippled and foreign user interface.

What switch are you talking about that Cisco charges $10,000 for that is comparable to a 24-port TP-Link switch?

Technically; there's not much difference anymore, routing isn't really a high-resource task considering the power of even basic modern computing hardware. You can perform basic routing in a lot of switches these days; and if you're loading up a custom Linux-based switch as the article suggests, you could run a very feature-rich router right on the switch.

That price omits the uplink module which will add at least another $400, while the TP-Link already includes 4 SFPs.

Besides, do you really think you're making Cisco look good?

"No! Cisco is only 20X more expensive!!! Not 40X!"

Quote:

The fact is the bigger problem is the fact that IOS is a giant kluge

No, I can assure you, the absolutely astronomical prices are problem #1.

The constantly rotating firmware bugs are a significant #2 as well... Gee, do I go with the firmware that mis-reports billions of packet discards, or the one that leaks memory so quickly the switch needs to be restarted every month? Thanks Cisco!

The concept definitely has merit and of course the likes of Cisco and Juniper are looking to leverage SDN to respond to the more virtualized network of today. Interestingly enough, CIsco and Juniper (and I'm sure others in the ICT space) define themselves as 'software companies.' That software being IOS and JunOS respectively. The downside to creating this specialized type of software is that you need hardware to run it on. While I'm sure everyone can come up with a horror story about any network infrastructure vendor's hardware and software, the two tend to work best when they are developed together. Cisco tried licensing IOS in the past (DEC being an example) and found that, not only does your software suffer (hi Microsoft) if the hardware is bad, but there is a TON of revenue to be had in making the hardware yourself. They will fight, successfully, to keep it that way. Even if it means dropping their margins to do so.

There is also something to be said for support. Red Hat built it's business on software support. The underlying software being given away. Enterprises like support, it brings security, and in many cases is required to remain compliant with SOX, HIPPA, etc. Open Source Infrastructure won't solve that problem. TAC (whoever's it is) is here to stay.

Technically; there's not much difference anymore, routing isn't really a high-resource task considering the power of even basic modern computing hardware. You can perform basic routing in a lot of switches these days; and if you're loading up a custom Linux-based switch as the article suggests, you could run a very feature-rich router right on the switch.

You wouldn't do that in a data centre, you won't get near the required throughput levels over l3 interfaces.Small deployments then this would be feasible but it's not clear what level they are targeting with this.

Technically; there's not much difference anymore, routing isn't really a high-resource task considering the power of even basic modern computing hardware. You can perform basic routing in a lot of switches these days; and if you're loading up a custom Linux-based switch as the article suggests, you could run a very feature-rich router right on the switch.

You wouldn't do that in a data centre, you won't get near the required throughput levels over l3 interfaces.Small deployments then this would be feasible but it's not clear what level they are targeting with this.

In a modern switch architecture you don't do routing in the OS anyway. It's taken care of once you program the switch's TCAM appropriately. The general-purpose CPU does things like presenting a management interface and running the control plane protocols necessary for you to know what to program into the TCAM.

Technically; there's not much difference anymore, routing isn't really a high-resource task considering the power of even basic modern computing hardware. You can perform basic routing in a lot of switches these days; and if you're loading up a custom Linux-based switch as the article suggests, you could run a very feature-rich router right on the switch.

You wouldn't do that in a data centre, you won't get near the required throughput levels over l3 interfaces.Small deployments then this would be feasible but it's not clear what level they are targeting with this.

In a modern switch architecture you don't do routing in the OS anyway. It's taken care of once you program the switch's TCAM appropriately. The general-purpose CPU does things like presenting a management interface and running the control plane protocols necessary for you to know what to program into the TCAM.

True enough, think you're talking the CEF stuff which should take the pressure of the CPU but just the packetisation delays out of the interface would slow things down enough. That said, it's not clear if they're only talking ToR switches for now which I don't think you'd do this kind of thing on. Would guess that they are focussing on straight layer 2 switching but I'd be interested in how they get port density and cooling off server hardware (NICs consume more power than I'd thought).

Technically; there's not much difference anymore, routing isn't really a high-resource task considering the power of even basic modern computing hardware. You can perform basic routing in a lot of switches these days; and if you're loading up a custom Linux-based switch as the article suggests, you could run a very feature-rich router right on the switch.

You wouldn't do that in a data centre, you won't get near the required throughput levels over l3 interfaces.Small deployments then this would be feasible but it's not clear what level they are targeting with this.

In a modern switch architecture you don't do routing in the OS anyway. It's taken care of once you program the switch's TCAM appropriately. The general-purpose CPU does things like presenting a management interface and running the control plane protocols necessary for you to know what to program into the TCAM.

True enough, think you're talking the CEF stuff which should take the pressure of the CPU but just the packetisation delays out of the interface would slow things down enough. That said, it's not clear if they're only talking ToR switches for now which I don't think you'd do this kind of thing on. Would guess that they are focussing on straight layer 2 switching but I'd be interested in how they get port density and cooling off server hardware (NICs consume more power than I'd thought).

What I'm talking about is actual hardware-based forwarding using merchant silicon switch chips. That goes to your next question regarding the actual hardware they intend for this project, which is not just a server with a bunch of NICs. It's dedicated forwarding hardware, almost assuredly based on one of the usual suspects (Broadcom or Fulcrum/Intel). Look at any number of ToR switches from several vendors and you'll see a common configuration: 48x10GbE and 4x40GbE. These are almost all based on a Broadcom Trident+ chipset (with some exceptions).

You wouldn't do that in a data centre, you won't get near the required throughput levels over l3 interfaces.

You've never seen a Cisco Nexus 7000, have you? Shame...

I have and I've seen the cost of the M series modules required to do that. I'd still say you should still be avoiding any L3 routing in the core, if possible, for architecture reasons and there's still more latency on routing compared to forwarding.

Can't see where this initiative sits against the Nexus product range. Could see them targeting ToR 3xx0 Catalysts where they might be over specced for what's asked of them but there's more to the Nexus stuff than I would think could be accomplished with off the shelf kit. Has there been any announcement on what they are intending on finally releasing (or what the spec published should be capable of)?

Will it stream to your wall post? Or maybe sniffer your corporate data to post it openly on Facebook?

Yea thanks. I will see many companies adapting this ridiculous propositions. By the way what makes Facebook think they can better design racks than most companies? They can´t. That may work in particular only for their needs, but it does not work for the rest. Rack design are made by tons of hardware companies and people, they run pretty much successful in tons of datacenters and I don't think they would apply ridiculous designs changes.

Any moron can connect things the other way around. Facebook is just a huge company with tons of morons and money to spend. If that picture in the article is actually from Facebook then im 100% on track with this, both with "runned by morons" and how idiotic that rack looks. This is pure marketing, the flashy blue connections (facebook color) just proves my point, marketing.

You wouldn't do that in a data centre, you won't get near the required throughput levels over l3 interfaces.

You've never seen a Cisco Nexus 7000, have you? Shame...

I have and I've seen the cost of the M series modules required to do that. I'd still say you should still be avoiding any L3 routing in the core, if possible, for architecture reasons and there's still more latency on routing compared to forwarding.

Can't see where this initiative sits against the Nexus product range. Could see them targeting ToR 3xx0 Catalysts where they might be over specced for what's asked of them but there's more to the Nexus stuff than I would think could be accomplished with off the shelf kit. Has there been any announcement on what they are intending on finally releasing (or what the spec published should be capable of)?

Technically; there's not much difference anymore, routing isn't really a high-resource task considering the power of even basic modern computing hardware. You can perform basic routing in a lot of switches these days; and if you're loading up a custom Linux-based switch as the article suggests, you could run a very feature-rich router right on the switch.

Yes, most enterprise switches do have basic to moderate L3 capability and are effectively high port-density routers if you configure them to be that way. However, the converse is not really true at all. Higher end routers (such as the one referenced), offer a number of advanced capabilities that can be executed over multiple types of physical media (copper, fiber, optical) with minimal processing delay.. think deep packet inspection, policy based routing, DMVPN, MPLS, etc. There are still major differences between routers and switches based on today's capabilities, but to your point, switches today can certainly do most of what routers did 10 years ago.

"We should be able to treat a switch like a server in the rack," Frankovsky said. "We should be able to load a Linux-based operating system, and that server just happens to have a lot of I/O ports on it."

This paragraph sums up what I've been thinking about for a while. If a company is not satisfied with off-the-shelf routers, they can easily build their own running Linux, BSD, etc, but this was never an option for switches. At worst, switches are black boxes implemented entirely in hardware, and at best, you get managed switches that have some knobs you can tweak (which might or might not be the knobs you're interested in).

Good point. We recently asked our IT department to set up a 10 GB ethernet switch for our department so we can edit videos in place from our 40TB server in another building. They got bids from a company and when the time came for them to deliver, they say, "We can't get that for you, but here's one for $20k more."

I'm totally on board with this. Even more so than the open compute stuff since there *is* robust competition in the OEM server space and building your own is already kinda-sorta doable. But the networking hardware has far more vendor lock in because of the proprietary interfaces and it's also *way* too expensive.

There's a ton of potential here. Cisco actually charges $10,000 for a switch, comparable to a $173.26 "TP-Link" (TL-SG2424) switch you can buy from Wal-Mart... Of course the latter has crap firmware (software) with a crippled and foreign user interface.

Foundry (now Brocade) has long done very well by making managed switches several times cheaper than Cisco's, which are almost command-compatible with Cisco's (IOS) UI. I see much more Foundry/Brocade branded network hardware in data centers racks, than Cisco gear. But prices could be much lower, still, if no-name Chinese manufacturers could make the dirt-cheap hardware, and a standard firmware could be installed to give them all common functionality and user-interface. On the low-end we have this with APs, thanks to DD-WRT, Tomato, OpenWRT, etc. If Facebook and the OpenCompute project can give a little push, to get this happening with managed switches, the networking world will look very different in just a couple years.

Ugh, will these be like DD-WRT? The thing that hasn't been updated in... 6 years? No thanks.

Will it stream to your wall post? Or maybe sniffer your corporate data to post it openly on Facebook?

Yea thanks. I will see many companies adapting this ridiculous propositions. By the way what makes Facebook think they can better design racks than most companies? They can´t. That may work in particular only for their needs, but it does not work for the rest. Rack design are made by tons of hardware companies and people, they run pretty much successful in tons of datacenters and I don't think they would apply ridiculous designs changes.

Any moron can connect things the other way around. Facebook is just a huge company with tons of morons and money to spend. If that picture in the article is actually from Facebook then im 100% on track with this, both with "runned by morons" and how idiotic that rack looks. This is pure marketing, the flashy blue connections (facebook color) just proves my point, marketing.

Intel supports SDN/open network as it sees x86 everywhere.Some of the atoms and atom based sbc(single board computers) do have integrated gigabit ethernet for industrial coms

Would would be nice to have a baseboard like a cubieboard or marsboard or similar with pluggable expansion boards with the network connectors and network processors.Might not be the most efficient (processing or powerwise) but should be able to keep it cheap and easily expandable.

Could be possible to turn it into either a switch or NAS or router or whatever combo network device you needand keep it reasonably cheap.I'd happily through in for a kickstarter project for this!!!Unfortunately no spare time to contribute or get involved

Another way would be base board then use an fpga for the switch controller - not fully open source as you would have to use proprietary tools to produce the fpga firmware. Though you could build ontop of the netfpga projectwhich has done quite a lot of workhttp://netfpga.org/http://netfpga.org/10G_specs.htmlThough using an fpga would blowout the cost :-(but still should be cheaper than most commercial 10G switches

The features Facebook is taking out. Do they exist because of "let's make more money", or do they actually serve a better purpose, just not Facebook's needs?

Some of that is spelled-out, like the decorative front bezel which ruins airflow...

Mostly, Facebook (and Google, and Amazon, and others) are stripping down servers to only just the minimum they need. Single NIC port, no video, simple rack-mount case, single power supply, single non-RAID SATA (not SAS/SCSI) controller, etc.

Servers from Dell, HP, IBM, etc., throw in everything but the kitchen sink, because they want to always be compatible, lowest-common-denominator, etc. There's nothing wrong with that, particularly when you need a few, highly reliable and manufacturer supported servers. But when you have a cluster, the economics completely changes, and a larger number of slightly less robust servers is far, far better.

Will the power button be a thumbs-up button that lights up? Seriously, FB and networking hardware? Stick to social. And who will these switches cater to? Enterprises will not want to have these "engineering" edition switches.

Will the power button be a thumbs-up button that lights up? Seriously, FB and networking hardware? Stick to social. And who will these switches cater to? Enterprises will not want to have these "engineering" edition switches.

Maybe you haven't actually read the origins of this, but the bigger Internet companies basically suffer from a bunch of unique issues that only apply when you have several 100k of servers. Google, Microsoft, Apple, FaceBook, Amazon and others DO face these issues and in many cases rather than just buying HP or Dell computers, they are designing their own servers and cutting out the middle men and having the East Asian ODMs make them for them. Though in some cases they have used special services or special companies here in the US to act as a broker. Dell did some of that for Google and HP did some of that for Microsoft.

FaceBook doesn't really want to make switches, though they probably do want to facilitate getting servers and switches that meet their needs for less. They also have people in certain places who really DO believe in Open Source Software and Hardware and are thereby contributing.

Google built its own switches using the same merchant network processors the networking gear companies use internally. The difference here is that FB is open-sourcing it, which allows smaller companies like mine that can't afford to pay a few engineers to build the networking O/S to piggyback on FB's R&D efforts. It's mostly a PR and recruiting exercise for FB - it doesn't cost them very much, it helps them recruit top-notch talent interested in being able to share what they do with a larger audience than simply their employer.

I hope they also include L4 load-balancing/failover capability in their design. While modern load balancers (sorry, the term is now Application Delivery Controllers) do additional stuff like SSL acceleration, you still pay a significant premium for L4 load-balancing functionality over basic L2/L3 switching, even though it doesn't tax the underlying hardware significantly more.