I'm assisting a former employer in managing some servers from out of state.

In a series of lessons in what NOT to do, I remotely power cycled a server (lesson 1) in order to check a permissions on a directory that was causing mail to crash. They rebooted the server a few days before, so I thought it would be fine to reboot it two days later (lesson 2). Unfortunately, I missed the udev update (lesson 3) that had been installed, so when the server came back online, it failed to start. I was able to get someone on site to swap to a backup drive, rebuilt the kernel with the DEVTMPFS flags enabled, and we were able to the machine to boot. However, now we have a eth0 issue:

It seems the hardware address for eth0 has changed. According to ifconfig, eth0 is:

However, according to the DHCP server, when ever this server comes online, it gets queried with

Code:

DHCPDISCOVER from 78:2b:cb:18:77:d3 via eth0: unknown client

I have NO IDEA where this Mac address is being configured. And it disappears when the machine boots into the backup drive.

We've tried modifying (with the mac address querying DHCP) or deleting /etc/udev/rules.d/70-persistent-net.rules - neither of which has any effect.

When I allow the MAC address querying the DHCP server to take an IP address - I can then ping that IP, but I can't ssh, or do anything else (and no network services come online with that IP). All nmap revleals is that it is a DELL box, and no ports are open.

We've tried modifying, deleting, or leaving a commented version of /etc/udev/rules.d/80-net-name-slot.rules

Code:

udevadm test-builtin net_id /sys/class/net/ifname 2> /dev/null

The contents of /sys/class/net

Code:

eth0 eth1 lo sit0

Maybe I need to modify the address in the eth0/address file of /sys/class/net ?

I've tried building the ethernet card into the kernel, but that doesn't work (Broadcom NetextremeII - my understanding is the Broadcom NetextremeII works only as a module).

I'm just totally stumped. I do have a fairly recent backup into which I am able to boot (and no data was lost, fortunately). so I can access the various partitions on the problem drive. And I've studied this thread fairly extensively, but nothing discussed seems to work (it's possible I missed something).

And as a second question - aside from rebuilding the kernel with DEVTMPFS, how do I prevent this from happening on the other gentoo servers at their site? Other than the version of udev, is there something else I need to check?

Anyway - any assistance would be GREATLY appreciated (and any mocking will also be understood).

Last edited by CM73 on Tue Feb 05, 2013 11:08 pm; edited 1 time in total

When you boot into the configuration that receives an address, but offers no service, what is the output of ip addr ; ip route ; ss -an? You can have an unprivileged user in the data center run these and either relay their contents to you, or write them to persistent storage and then reboot into the backup to access the saved data.

Thanks - I didn't catch that. I'll post the results when he installs the firmware.

Out of curiosity, though... why would that *suddenly* be required when the server was working perfectly fine - reboot and all - on Wednesday and suddenly need the firmware after the world update on Friday.. The machine wasn't that out of date - maybe a two weeks or so - so I'm a bit surprised that, whatever update was done, hosed the machine.

Thanks - I didn't catch that. I'll post the results when he installs the firmware.

Out of curiosity, though... why would that *suddenly* be required when the server was working perfectly fine - reboot and all - on Wednesday and suddenly need the firmware after the world update on Friday.. The machine wasn't that out of date - maybe a two weeks or so - so I'm a bit surprised that, whatever update was done, hosed the machine.

I have some IBM System X servers at work with the exact same card and I can't remember them ever working without the firmware. I suppose some mistake was done, perhaps --depclean removed linux-firmware.

So I chrooted (ala the Gentoo installation guide) into the *non-bootable* drive, and installed the firmware:

Code:

>>> Installing (1 of 1) sys-kernel/linux-firmware-20120924
* This package will overwrite one or more files that may belong to other
* packages (see list below). You can use a command such as `portageq
* owners / <filename>` to identify the installed package that owns a
* file. If portageq reports that only one package owns a file then do
* NOT file a bug report. A bug report is only useful if it identifies at
* least two or more packages that are known to install the same file(s).
* If a collision occurs and you can not explain where the file came from
* then you should simply ignore the collision since there is not enough
* information to determine if a real problem exists. Please do NOT file
* a bug report at http://bugs.gentoo.org unless you report exactly which
* two packages install the same file(s). Once again, please do NOT file
* a bug report unless you have completely understood the above message.
*
* Detected file collision(s):
*
* /lib/firmware/tigon/tg3.bin
* /lib/firmware/tigon/tg3_tso5.bin
* /lib/firmware/tigon/tg3_tso.bin
* /lib/firmware/bnx2/bnx2-mips-09-6.2.1.fw
* /lib/firmware/bnx2/bnx2-rv2p-09ax-6.0.17.fw
* /lib/firmware/bnx2/bnx2-rv2p-09-6.0.17.fw
* /lib/firmware/bnx2/bnx2-rv2p-06-6.0.15.fw
* /lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw
* /lib/firmware/bnx2/bnx2-mips-09-6.2.1a.fw
* /lib/firmware/bnx2x/bnx2x-e1h-6.2.9.0.fw
* /lib/firmware/bnx2x/bnx2x-e2-6.2.9.0.fw
* /lib/firmware/bnx2x/bnx2x-e1-6.2.9.0.fw
*
* Searching all installed packages for file collisions...
*
* Press Ctrl-C to Stop
*
* None of the installed packages claim the file(s).
*
* Package 'sys-kernel/linux-firmware-20120924' merged despite file
* collisions. If necessary, refer to your elog messages for the whole
* content of the above message.
* If you are only interested in particular firmware files, edit the saved
* configfile and remove those that you do not want.

>>> Recording sys-kernel/linux-firmware in "world" favorites file...

* Messages for package sys-kernel/linux-firmware-20120924:

* Your configuration for sys-kernel/linux-firmware-20120924 has been saved in
* /etc/portage/savedconfig/sys-kernel/linux-firmware-20120924 for your editing pleasure.
* You can edit these files by hand and remerge this package with
* USE=savedconfig to customise the configuration.
* You can rename this file/directory to one of the following for
* its configuration to apply to multiple versions:
* ${PORTAGE_CONFIGROOT}/etc/portage/savedconfig/
* [${CTARGET}|${CHOST}|""]/${CATEGORY}/[${PF}|${P}|${PN}]
* This package will overwrite one or more files that may belong to other
* packages (see list below). You can use a command such as `portageq
* owners / <filename>` to identify the installed package that owns a
* file. If portageq reports that only one package owns a file then do
* NOT file a bug report. A bug report is only useful if it identifies at
* least two or more packages that are known to install the same file(s).
* If a collision occurs and you can not explain where the file came from
* then you should simply ignore the collision since there is not enough
* information to determine if a real problem exists. Please do NOT file
* a bug report at http://bugs.gentoo.org unless you report exactly which
* two packages install the same file(s). Once again, please do NOT file
* a bug report unless you have completely understood the above message.
*
* Detected file collision(s):
*
* /lib/firmware/tigon/tg3.bin
* /lib/firmware/tigon/tg3_tso5.bin
* /lib/firmware/tigon/tg3_tso.bin
* /lib/firmware/bnx2/bnx2-mips-09-6.2.1.fw
* /lib/firmware/bnx2/bnx2-rv2p-09ax-6.0.17.fw
* /lib/firmware/bnx2/bnx2-rv2p-09-6.0.17.fw
* /lib/firmware/bnx2/bnx2-rv2p-06-6.0.15.fw
* /lib/firmware/bnx2/bnx2-mips-06-6.2.1.fw
* /lib/firmware/bnx2/bnx2-mips-09-6.2.1a.fw
* /lib/firmware/bnx2x/bnx2x-e1h-6.2.9.0.fw
* /lib/firmware/bnx2x/bnx2x-e2-6.2.9.0.fw
* /lib/firmware/bnx2x/bnx2x-e1-6.2.9.0.fw
*
* Searching all installed packages for file collisions...
*
* Press Ctrl-C to Stop
*
* None of the installed packages claim the file(s).
*
* Package 'sys-kernel/linux-firmware-20120924' merged despite file
* collisions. If necessary, refer to your elog messages for the whole
* content of the above message.
* If you are only interested in particular firmware files, edit the saved
* configfile and remove those that you do not want.
>>> Auto-cleaning packages...