CentOS 6 - cannot log in/shell quits

Have a VM, running CentOS 6, it's a development machine and it was on and off for a few weeks (devs had something more important to fix); following an update I've noticed that I cannot login from the network, only via VM console (running Hyper-V on a 2008 R2 cluster) but I figured it's the usual shitty MS linux IC issue (happens sometimes) so didn't bother...Devs are back, trying to log in but now I couldn't log in even via VM console... did a single user root passwd reset, still no dice. Back to single mode, added new user (with su - in mind, also to see if root got somehow corrupted), reboot, still nothing.

- Check login shells. Like maybe /bin/bash was corrupted, or changed.- Check the /var/log/secure and see if anything weird is there.- Make sure some jackass didn't put "logout" as your first command in .bash_profile, .bashrc, and so on. I used to do that way way way way long ago in a university far far away when the lab had a strict policy of not leaving an unattended terminal logged in. Mwah ha ha...- Also check that the PAM is set correctly: http://www.centos.org/docs/5/html/Deplo ... h-pam.html - a package upgrade may have swapped these around, fucking up the order of authentication, or put in a PAM module that is required for login, but is not working (this happened once when someone messed up a winbind server).

Oh, god dammit, this burned me more ways than one, and several just recently when some chowderhead made a bunch of DB VMs 7.0gb. Yes. Yes, check this. I cannot tell you how many symptoms a full disk will emulate and I waste my time Googling the error.

Make sure it is not a discrepancy in your known_hosts file for SSH. To be sure remove any entry for the hostname/IP in the known_hosts file on the workstation you are trying to connect with. A conflict in host key entries can cause this type of issue.

Make sure it is not a discrepancy in your known_hosts file for SSH. To be sure remove any entry for the hostname/IP in the known_hosts file on the workstation you are trying to connect with. A conflict in host key entries can cause this type of issue.

I don't have any eth up (or even present? Cannot remember: how do you list your eth NICs?), I cannot even log in on local console (VM), only in single mode.

There you go. Look in /etc/pam.d/ and remove that file until you figure out why the install went bad. If it's not there, look in /etc/pam.conf, which usually just loads /etc/pam.d/* by default these days.

There you go. Look in /etc/pam.d/ and remove that file until you figure out why the install went bad. If it's not there, look in /etc/pam.conf, which usually just loads /etc/pam.d/* by default these days.

Yes, that's what I did except as I said earlier there's nothing like that in my /etc/pam.d/login - but I checked my /etc/pam.d/system-auth and lo and behold mkhomedir.so was required... commented out, rebooted and I am able to log in. =) THanks for everyone especially for you.

Next up: where di my eth interfaces go...? I cannot really use yum without ethernet... temporarily I'll add a legacy network card and see what happens...

They should show up with "netstat -i". If they don't you might wanna sift through "lspci" to see if the system can even see any of them. If you do find them "ethtool" can show most any information about the physical link status, run tests, etc.

Oh and that PAM module, is the file name actually "pam-mkhomedir.so"? Because there shouldn't be a dash in there, it should be an underscore.

Next up: where di my eth interfaces go...? I cannot really use yum without ethernet... temporarily I'll add a legacy network card and see what happens...

Ah... a VMWare running CentOS 6. Well, welcome to a special level of hell. Okay, that's a dramatic exaggeration to make me look important. But I suspect probably TWO things have gone wrong, and I am so used to changing them now it's not even funny. Long story short: when VMWare auto-changes the NIC (like from a clone or template deployment), it auto-changes the MAC address, which makes CentOS detect it as a new interface and then barfs all over it. I want you to check TWO files:

Code:

/etc/udev/rules.d/70-persistent-net.rules

You're going to find some lines in there with multiple NICs. In vSphere, check the MAC assigned to the NIC and remove in that udev file any line with that is not that MAC address. Or comment it out if it makes you nervous; I won't judge. Then make sure uncommented lines have the correct MAC (next to ATTR{address}==)and device "NAME" is eth0 whatever you have). If you have more than just one NIC, do the same for that second NIC.

Next, go into

Code:

/etc/sysconfig/network-scripts/ifcfg-eth0

And comment out "HWADDR" and "UUID." Then do the same if you have an eth1, eth2, or whatever. If you don't, just remove any "ifcfg-eth[xxx]" you DON'T have. Sometimes they get auto-created and cause a mess.

Yes, it certainly is a good idea to test whether the box comes up cleanly, but if you're cleaning up multiple issues its sometimes worthwhile to save time by cleaning up all of them at once and then reboot the box, instead of rebooting it after every step.

This is probably not that important for most VMs, but for physical machines which can take several minutes to boot ('lo HP DL380 G7 and later, especially if the box has several 10GE cards or FC HBAs) this can add up to a significant amount of time.

This is probably not that important for most VMs, but for physical machines which can take several minutes to boot ('lo HP DL380 G7 and later, especially if the box has several 10GE cards or FC HBAs) this can add up to a significant amount of time.

True; back when I worked with the DLs in my last job, I kept wondering why modern machines took so long to boot. I mean, the DL380s took almost 2-3 minutes just to get to the OS. Criminy, that was nervewracking on an RDP.

If you're running Linux under Hyper-V, make sure that all the network adapters are set to a static MAC. Otherwise every time the VM is powercycled or migrated(I believe) it'll change the MAC to something else. As Punk Walrus's post covers, most modern Linux distributions(or rather, most modern versions of udev) use the MAC address to create static ethX entries(so the order of the ethX interfaces don't change on reboots/kernel upgrades as they sometimes did). This obviously doesn't work overly well if the hypervisor likes to change the MACs on a regular basis though.

Next up: where di my eth interfaces go...? I cannot really use yum without ethernet... temporarily I'll add a legacy network card and see what happens...

Ah... a VMWare running CentOS 6. Well, welcome to a special level of hell. Okay, that's a dramatic exaggeration to make me look important. But I suspect probably TWO things have gone wrong, and I am so used to changing them now it's not even funny. Long story short: when VMWare auto-changes the NIC (like from a clone or template deployment), it auto-changes the MAC address, which makes CentOS detect it as a new interface and then barfs all over it. I want you to check TWO files:

Code:

/etc/udev/rules.d/70-persistent-net.rules

You're going to find some lines in there with multiple NICs. In vSphere, check the MAC assigned to the NIC and remove in that udev file any line with that is not that MAC address. Or comment it out if it makes you nervous; I won't judge. Then make sure uncommented lines have the correct MAC (next to ATTR{address}==)and device "NAME" is eth0 whatever you have). If you have more than just one NIC, do the same for that second NIC.

Next, go into

Code:

/etc/sysconfig/network-scripts/ifcfg-eth0

And comment out "HWADDR" and "UUID." Then do the same if you have an eth1, eth2, or whatever. If you don't, just remove any "ifcfg-eth[xxx]" you DON'T have. Sometimes they get auto-created and cause a mess.

Reboot.

You should be good to go.

Just as an addendum to this, if you have only one NIC there's no harm in simply deleting the persistent-net.rules file, it'll be regenerated the next time you reboot, or you can run udevadm to regenerate it right away. Before we had a more proper deployment platform around here I'd load our RHEL template with a script that did this on first boot.

Just as an addendum to this, if you have only one NIC there's no harm in simply deleting the persistent-net.rules file, it'll be regenerated the next time you reboot, or you can run udevadm to regenerate it right away. Before we had a more proper deployment platform around here I'd load our RHEL template with a script that did this on first boot.

Most people only have one NIC, like myself, and I tried searching for what you answered, "udevadm," so I am going to check this out when I get to work today. Thanks, Sunner!