Sunday, September 13, 2009

Under certain conditions, the LAN controller (i.e. NIC) on an Asus P6T Deluxe v2 motherboard may start sending strange packets after the computer is shut down. This constant stream of packets from the NIC may cause all traffic through your switch to halt. The cause of the strange packets appears to be triggered by a bug in the 'sky2' network driver in Linux. See the information below to learn more about what conditions trigger this behavior and possible workarounds that can be used until the bug is fixed.

NOTE: This problem does not apply to Windows users.

Background

I recently assembled a computer using the Asus P6T Deluxe v2 motherboard and ran into an issue with the dual gigabit LAN controllers built-in to the motherboard. While I was at work last week, I remotely connected to my new system through SSH and then shut it down after I had retrieved some information I needed (I usually like to shut down my systems while I'm not at home just to conserve energy). I was also connected to another sytstem at home and about fifteen minutes later, after shutting down the Asus P6T-based system, I lost all network connectivity to the systems on my home network. Without going into too many details, I eventually found out that the LAN controllers on the P6T motherboard can sometimes go into a weird state (after the machine is shut down) where they repeatedly send packets that look like the following (output is from 'tcpdump'):

The P6T LAN controller sends about 50 of these packets per second and after a while this causes all network traffic going through my switch to cease. As you might imagine, it is especially annoying if this happens while I am at work and no longer have a way of accessing any of my systems until I get home.

At the time I discovered this, I called Asus technical support to see if this was a known issue or if I could possibly have a faulty motherboard. Calling them was a mistake. All I got was a condescending tone from the other side of the line and a person telling me that it was impossible for the NIC to send packets while the system was shut down. He wouldn't believe what I was telling him. That conversation ended pretty quickly once I realized I wouldn't be getting any support from Asus on this issue. Before I hung up though they did mention that someone with a Maximus II Formula motherboard had called in with a similar issue a week earlier. I later found out that the Maximus motherboard uses the exact same LAN controller as the P6T motherboard, a Marvell 88E8056 chip. I would suspect that any motherboard that uses this Marvell controller would experience the same network problems.

After the lack of support from Asus, I decided to dig into the issue myself and after a lot of testing I found out how to reproduce the problem 100% of the time. Before I explain how to reproduce the behavior/bug, here is a summary of the hardware and software I have tested with:

Motherboard: Asus P6T Deluxe v2

Dual gigabit NICs using the Marvell 88E8056 chip.

Other motherboards using the Marvell LAN controller may also be affected.

Switch: Netgear GS108T Gigabit Switch

I'm not sure if network activity halts when other gigabit switches are used.

Kernel Version - I tested with the standard installed kernel on all distributions except Ubuntu 9.04 where I decided to test with a vanilla non-patched kernel from kernel.org in addition to the standard Ubuntu kernel.

Switch Port Configuration - I used a managed switch and performed tests where the switch port connected to the P6T NIC was configured for either 10, 100, or 1000 Mb/s (and it did make a difference as you'll see in the table).

WOL enabled before shutdown? - I recorded whether or not wake-on-lan was configured before the system was shut down (e.g on linux wake-on-lan can be turned on with a command similar to 'ethtool -s eth0 wol g').

The last four columns record my observations:

Ethernet Link Reset During Shutdown?

During some tests, the led link lights on the switch would indicate that the link was momentarily shut down and then reestablished at the time the system was powered down.

On other tests, the led link lights on the switch would indicate that the link was never reset during shutdown (as if the link was still operating in the same mode as it was when the system was turned on).

In other tests, power to the NIC was shut off and the link was never reestablished (as expected if WOL was not enabled).

Ethernet speed after shutdown - During the tests where the ethernet link to the switch was reset after the sytem was shut down, I recorded the speed of the reestablished ethernet link.

WOL works? - Shows whether or not wake-on-LAN worked after the system was shut down. For some tests this column is not applicable because WOL was never enabled/configured.

Strange packet bug encountered? - Shows whether or not I was able to see the strange packet sending behavior.

NOTE: In cases where the packet sending bug was observed, I first had to send around eight broadcast packets on the network to trigger the problem where the P6T NIC started sending the strange packets. I could send any type of broadcast packet. For example, the problem could be triggered by sending either broadcast pings (e.g 'ping -b 192.168.1.255') or wake-on-LAN magic packets (they didn't have to be specifically directed to the P6T NIC MAC address).

NOTE: In each case where I encountered the packet sending bug, traffic immediately resumed through the switch after after unplugging the cable from the offending network port.

Evironment / Settings

Observations

OS/Distro

Kernel
Version

Switch Port
Configuration
(Mb/s)

WOL
enabled
before
shutdown?

Ethernet Link
Reset During
Shutdown?

Ethernet
speed after
shutdown

WOL
works?

Strange
packet bug
encountered?

Windows 7 RTM

n/a

10

yes

yes

10

yes

no

Windows 7 RTM

n/a

100

yes

yes

10

yes

no

Windows 7 RTM

n/a

1000

yes

yes

10

yes

no

Express Gate

Unknown

10

no

no

10

n/a
(WOL not
enabled)

no

Express Gate

Unknown

100

no

no

100

n/a
(WOL not
enabled)

no

Express Gate

Unknown

1000

no

no

1000

n/a
(WOL not
enabled)

yes

Ubuntu 9.04

2.6.28,
2.6.31

10

yes

yes

10

no

no

Ubuntu 9.04

2.6.28,
2.6.31

100

yes

yes

100

no

no

Ubuntu 9.04

2.6.28,
2.6.31

1000

yes

yes

100

no

no

Ubuntu 9.04

2.6.28,
2.6.31

10

no

no

n/a
(NIC gets
powered
down)

n/a
(WOL not
enabled)

no

Ubuntu 9.04

2.6.28,
2.6.31

100

no

no

n/a
(NIC gets
powered
down)

n/a
(WOL not
enabled)

no

Ubuntu 9.04

2.6.28,
2.6.31

1000

no

no

n/a
(NIC gets
powered
down)

n/a
(WOL not
enabled)

no

Ubuntu 9.10

2.6.31

10

yes

yes

10

no

no

Ubuntu 9.10

2.6.31

100

yes

yes

10

no

no

Ubuntu 9.10

2.6.31

1000

yes

yes

100

no

no

Ubuntu 9.10

2.6.31

10

no

no

10

n/a
(WOL not
enabled)

no

Ubuntu 9.10

2.6.31

100

no

no

100

n/a
(WOL not
enabled)

no

Ubuntu 9.10

2.6.31

1000

no

no

1000

n/a
(WOL not
enabled)

yes

A couple of things to note from the table:

On Windows, where WOL worked and the bug was never encountered, the ethernet link was always reestablished at 10 Mb/s when the system was shut down no matter what speed of the link was when the system was powered on.

On Linux, if WOL was enabled, the ethernet link would be reestablished at either 10 or 100 Mb/s when the sytem was shut down (depending on the speed of the link while the system was running).

On Linux, when WOL was not enabled and the system was affected by the bug, the ethernet link was never reset when the system was shut down and it remained operating at the same speed as when the system was running. It was as if the NIC was still operating in the same mode as when the sytem was turned on instead of going into a special mode that listened for WOL packets.

Summary

The strange packet sending bug was encountered if all of the following conditions were met:

The system was powered down from one of the following distros

Ubuntu 9.10

Express Gate (i.e. Splashtop) Linux built-in on the motherboard

Status of other distros is unknown. I suspect that this problem may not be kernel-version specific but due to a patch that some distros apply to the sky2 driver...but I am not certain.

A Gigabit switch was used AND the ethernet link between the switch and LAN controller was established at 1000 Mb/s before the system was shut down

The sky2 network driver was not configured for wake-on-lan before the system was shut down

Possible Workarounds:

Enable wake-on-lan before the system is shut down (wake-on-lan will not work, but at least the NIC won't start sending the strange packets)

Use a Linux distro or kernel that does not exhibit this problem

Anyhow, hopefully this helps somebody that has been experiencing the same network problems and wasn't sure why it was happening (I was worried for a little bit that it might be a faulty motherboard or a failing switch). If this problem gets fixed in the sky2 driver on Linux, I'll update this post to reflect the status.

No problem. What's terrible is that I took all the time to put this information together on the blog but I still haven't contacted anybody on the Linux kernel mailing lists and pointed them to this information so they can hopefully fix the problem. I still need to do that... it is still broken as of Ubuntu 10.04.

I have a Windows Vista x64 system, and can confirm that the same thing is happening on my network setup using a P6T. We have an RV082 Router, hooked up to two gigabit switches (about 20 or so computers on the network), and once every blue moon, the network controller on the P6T grinds the network to a halt by sending out hundreds of packets per second. The only way to stop this, is to cut the power to the MB and reboot.