The point was, /var is fine without a init thingy, right now. It may change soon tho since /var has been discussed and I see the errors on my machine already. I have all but /boot and / on LVM.

same here, so if you see errors, than something is not good._________________Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
ProjectFootball

The errors are not init related tho. The errors is because /var is on a separate partition. If I built a init thingy from scratch and only mounted /usr then did the switch_root, I would get the same errors. I know this because I got the errors even when /usr was on /.

The reason I mentioned it is about the question related to /var. Right now it works without a init thingy and /var on a separate partition. In the future that may change. Given the errors, it may be sooner than some think. The reason I run into this is because /var is also on LVM now. LVM is complaining, not the init thingy or udev.

The errors are not init related tho. The errors is because /var is on a separate partition... The reason I run into this is because /var is also on LVM now. LVM is complaining, not the init thingy or udev.

Sounds like the lvm2 changes TrogDog and I ran into as well (second half of post.)

genstorm wrote:

Are you 100% sure? I haven't read about this anywhere else yet, all the talks is about /usr only.

Yes, it was discussed on gentoo-dev a while back, in the context of alsa initscript saving state in /var, and alsa started up by udev in response to finding a sound-card. While I'm not sure if alsa still does that, we were told that there is no way to stop third-party device-startup scripts requiring a file effectively anywhere (the second url gives other examples.)

Lennart Poettering wrote:

you are mistaken, we support /var split off just fine, without initrd. Same for /home. These splits make a lot of sense and are not cumbersome to implement, hence we support them.

It's funny that he still can't get past /usr split "not making sense" when the RedHat move of everything to /usr now touts its benefits.

It's u-turns like that (another one is how dbus requires xml config files: I mean, really? and having finally learnt it's not a good idea, systemd doesn't use them) that make one doubt the experience of those involved. Another recent example is their insane usage of binary logfiles.

Linus wrote:

this is just pure and utter shit.. It's pure crap.. the kind of thinking and code that just makes me go "No way, not today, not ever". It's stupid.

The thing is, they can't get those ideas to fly in kernel-space, as core devs have veto over their work. There is no such control in user-space: most kernel devs don't really care what happens in user-space; in fact they plan for it to do crazy things, or their OS is not robust. The only strict rule is not to break an ABI exported to user-space. These changes are far more invasive (in terms of their consequences for other people's work) and throw away 50 years of design principles.

Linus wrote:

You made the statement that you want to get away from 30 years of history, but the fact is, "30 years of history" is a damn good argument for "that thing worked".

I'm not saying they haven't done anything worthwhile, nor learnt anything in the last few years. They just really don't get Unix principles, imo, and see them as needless restrictions rather than useful discipline. I know the nature of voluntary development (those who write the code, decide what it does) but still find it amazing that what I see as a bunch of students can get away with such far-reaching, ill thought-out changes across a *nix ecosystem, especially given the amount of money in Linux-based corporations.

I also believe that there is a culture of "not worrying about it" which becomes complacency. In the case of Linux development, the core GNU make tool does expansion of library dependencies before any command sees them, which means if a lib happens to exist on the host system, or more relevantly in /usr, it'll get used irrespective of any -L parameters to ld. When I read about that, I suddenly understood why you get "random" linkage. That clearly leads developers not to worry about it, since they can't figure out where it comes from nor how to change it (it's GNU-specific and not very well-known), and it's a distro or installer problem, assuming any sort of portability. So we get to packagers deciding "it's easier just to assume that /usr is available" which leads to "split /usr is cumbersome."

IMO if there hadn't been such a long history of unwanted linkage to /usr, there never would have been this culture of just assuming /usr in distros, nor the desire to get away from "worrying about it."

The errors are not init related tho. The errors is because /var is on a separate partition... The reason I run into this is because /var is also on LVM now. LVM is complaining, not the init thingy or udev.

Sounds like the lvm2 changes TrogDog and I ran into as well (second half of post.)

This I think fixes it according to a post on the mailing list:

Quote:

The secret, as Neil reminded us, is to change /etc/lvm/lvm.conf to read:

Code:

locking_dir = "/run/lock/lvm"

I have not tested it on my rig but others posted that it fixed their problem. I need to try that on mine but my problem is that I rarely reboot. Gotta love Linux.

All that is for folks that use dracut to build the init thingy. May work for others tho.

Basically I think we could set the 3 udev-related ones to 0 (although I imagine the first is auto-detected during the scan, since udev is not running.) The last one looks the most promising in terms of our lost /dev/vg/lv links though we still have /dev/mapper/vg-lv (which is what makes me think it detects udev is not running.)

Since I don't run mdadm, I might as well set md_component_detection to 0 as well, though in general it's safest to leave it on.

Quote:

All that is for folks that use dracut to build the init thingy. May work for others tho.

For me, it's about more than one way to skin a cat. Some find it useful to kill the cat first. Others like to get scratched a little bit first. We learn either way.

If one fix doesn't work, try another. Eventually one or a combination of two or maybe all three will fix it. Then again, I just put my sledge hammer beside my case and things tend to at least try harder. I did use a past rig for target practice once. It's amazing what 00 buckshot will do to a puter. It's a serious attitude 'adjustment' for sure.

So, haven't had time to mess with this until now, until a couple days ago when I rebooted and suddenly stuff didn't work. From what I saw in the logs, it looks like despite having udev 171 and things working previously, my system wasn't mounting more than the root partition at boot. ;(

I've followed all the steps in the first post (didn't even have to change the kernel settings, as I already had them), recompiled the kernel (to fix the nVidia driver bitching about a mis-match), and now I'm about to reboot.

For the record, steveL, your instructions are very easy to follow.

I'll update after this boot to see if it fixes things for me.

[edit] Well, this has improved things. I get far enough into the system where I can manually mount the partitions, which I couldn't do before using your instructions. Having to update this post from my other (currently working) system, however.

Now to figure out why the system claims it can't find /dev/sda to mount my listed partitions, but I can manually mount them after logging in.

What I get now is a message that "fsck.ext3: no such file or directory trying to open /dev/sda3", which is / that its booting from, and "mount: special device /dev/sda5 does not exist", followed by the same for sda6, 7, 8, 9 and 10 (which are /tmp, /usr, /var, /var/log, /var/www and /home, respectively). A few other things bitch about not being able to run, obviously due to /usr not being mounted, then I get a login prompt. When I log in, I'm able to mount all the partitions just fine.

Could being on kernel 2.6.39 be part of the problem? The system I'm on now is running 3.2.1, iirc.

/sigh, and no more time to work on this tonight._________________krenshala
:wq

I rebooted and suddenly stuff didn't work. From what I saw in the logs, it looks like despite having udev 171 and things working previously, my system wasn't mounting more than the root partition at boot.

That sounds like you updated some software prior to the reboot. I'd start by running dispatch-conf or etc-update to make sure you haven't missed any config updates, then try a reboot.

Careful with any changes to udev initscripts! If you get any conflicts there and mess things up, the original scripts are in /usr/portage/distfiles/udev-gentoo-scripts-7.tar.bz2 for udev-171. (I never got anywhere with emerge --config and --noconfmem on my first upgrade.)

Then you could try checking /var/log/emerge.log to see what you built recently. (I don't use any log-viewers atm so can't recommend one; I just use less and hit End ;)

Quote:

I've followed all the steps in the first post (didn't even have to change the kernel settings, as I already had them), recompiled the kernel (to fix the nVidia driver bitching about a mis-match), and now I'm about to reboot.

Hmm that nvidia driver mismatch seems odd, if you hadn't upgraded the kernel. Upgrading the driver package itself compiles against current /usr/src/linux so shouldn't be an issue; you sure that link is correct? (ls -l /usr/src/linux)

Quote:

I get far enough into the system where I can manually mount the partitions, which I couldn't do before using your instructions.
..the system claims it can't find /dev/sda to mount my listed partitions, but I can manually mount them after logging in.

What I get now is a message that "fsck.ext3: no such file or directory trying to open /dev/sda3", which is / that its booting from, and "mount: special device /dev/sda5 does not exist", followed by the same for sda6, 7, 8, 9 and 10 (which are /tmp, /usr, /var, /var/log, /var/www and /home, respectively). A few other things bitch about not being able to run, obviously due to /usr not being mounted, then I get a login prompt. When I log in, I'm able to mount all the partitions just fine.

Hmm that's odd: it sounds like the device nodes aren't there, and are getting setup by udev instead when it does finally run. This post earlier in the thread might be apposite, but afaik the kernel should be setting the node up for you with DEVTMPFS_MOUNT. Also, I'm not sure how udev could be running if rootfs is not being mounted.

I'd try booting with init=/bin/bash (added to end of kernel bootline; hit 'e' in grub to edit it) to see what nodes are available. Though this will probably do exactly the same as you currently have, at least you'll be up with the nodes the kernel makes available in /dev and can check them yourself. If mount -a works there, I'm at a loss as to what's going on (I think I am anyway: can you tell? ;) Only things that come to mind are persistent naming (eg if you have a USB drive plugged in, it used to be the case that it could appear before local drives as sda, not sure if it still is) but you're able to mount via sda once system has come up so that seems wrong, and I know udevadm settle is about, but I don't see how that applies, as kernel module should take care of that.

You do have the modules for the disk-controller as built-ins to the kernel (Y), not as modules (M), right?

Quote:

Could being on kernel 2.6.39 be part of the problem? The system I'm on now is running 3.2.1, iirc.

Hmm not sure: it's worth a try. After all, there are bugfixes to kernel all the time, and I've been on 3.2.1 for a while as well (I also only run stable kernels :)

You should be running with rc-update del xdm default till you get this sorted in any case, so you shouldn't need to worry about nvidia stuff til your system is back.

I checked for missing config updates first thing (when chrooted from the LiveDVD last night) and that wasn't the problem. I also verified I am still using udev 171 (I am).

And yeah, nVidia shouldn't have been complaining, but it was stating it was using 295.xx when the kernel was compiled for 290.xx (don't remember the exact values right now). It didn't take long for a zero change kernel recompile, however, so I didn't worry about that part too much (a minor annoyance compared to the boot process failing, after all).

And yes, I have only a single kernel module - the nVidia driver. Everything else is compiled into my kernel, which is my preference and the primary reason I don't see the need for an initrd being used on my systems.

I guess I'll just have to double check all the steps when I get home from work tonight. If nothing else, I'm learning more about administrating a linux box.

[edit] so, I found a typo in the udev-mount script ( parenthesis instead of braces), but correct that didn't really change much. I've decided to just try emerge --emptytree system from a chrooted environment to see if that unbreaks my system. it can't make it worse, thats for sure. i'll stop cluttering this thread with more on my systems until i'm actually doing something relevent. Thank you again for the help, though. _________________krenshala
:wq

Now to figure out why the system claims it can't find /dev/sda to mount my listed partitions, but I can manually mount them after logging in.

I really would ask on IRC, in #gentoo on chat.freenode.net: treat this as a fresh install, and you're manually configuring your kernel to allow you to move from chroot to booting into console. Once you can get root mounting with fsck running correctly, you're there.

And please, check CONFIG_DEVTMPFS and CONFIG_DEVTMPFS_MOUNT again (especially the latter) and also see what device nodes exist in /mnt/gentoo/dev when nothing's mounted there. If you can boot, even with errors, try disabling udev and udev-mount services, and reboot, then see what nodes exist. These should be the ones provided by the devtmpfs kernel mount, so you can confirm that sda is missing: again #gentoo will be of more help from there.

if not, check that other post I mentioned; summary: use MAKEDEV -n generic in /mnt/gentoo/dev as similar problems to do with /dev/console etc have hit udeved with his port of mkinitcpio (discussion was in #gentoo-dev-help.) Once you're happy with what it will do, remove the -n.

MAKEDEV, from sys-apps/makedev and on live CD, can be used to make any of the device nodes you might need, and should set up correct permissions, ownership etc.

... now I would have been just FINE but for the fact that udev itself now installs into /usr since 181. 171 my systems (/usr on lvm on mdadm) are just fine. Guessing I'm masking >=181._________________There are 10 kinds of people in the world,
those who understand binary and who don't

... I would have been just FINE but for the fact that udev itself now installs into /usr since 181. 171 my systems (/usr on lvm on mdadm) are just fine. Guessing I'm masking >=181.

Don't give up yet. :) That was the situation this setup was designed for: that udev going forward needs /usr (and rules or scripts might need eg /var or /opt) mounted before it starts.

If the patches apply (they're not very complex so shouldn't take much working out but I need to review the new initscripts: if you have them to hand feel free to paste them) then there is no issue: this doesn't start udev til after localmount, so all partitions (including /usr and /var) are available.

We'll only be in difficulty if and when udev has a mandatory requirement for systemd; if that should happen, then I'll use a previous version, and assess the state of mdev at that time versus a fork.

It had been bothering me for a while that udev was starting quite late: this makes it start immediately after localmount (before * didn't work as that just applies across the runlevel) and stop just before, which seems to be working much more nicely.

Don't ask me what the problems were, it was a big upgrade (I lost X for a bit, and decided to get rid of semantic-desktop crap in KDE) and for the life of me I can't remember what went wrong. It certainly works smoothly now, and I feel much more confident about the setup, since all devices are setup before anything else tries to run (apart from device-mapper/ lvm, fsck etc for localmount.)

Going to try this now. Thanks for the instructions Steve. This udev change hosed my system nicely. Especially since I had not been updating for a long time. I got /usr /var /home /opt and /tmp on lvm partitions.

I noticed that in my system the kernel option for the dev tmpfs(CONFIG_DEV_TMPFS) is named differently:

I noticed that in my system the kernel option for the dev tmpfs(CONFIG_DEV_TMPFS) is named differently

Yeah sorry, that was a typo that was pointed out before, so I'd edited the post, but for the new version of the patch, the file on my system still had the old text.

Quote:

As a reminder to myself: since I don't have the initramfs in use: udev must be at the boot runlevel and put the line 'initramfs="no"' in the /etc/conf.d/udev.

There's still a error message shown during the boot: ERROR: cannot start udev as fsck would not start
Trying to figure out where it's coming from.

As well as udev in boot, you also need sysfs and udev-mount added to sysinit runlevel: are you sure those are there? And you have patched /etc/init.d/udev-mount as well as /etc/init.d/udev?

Quote:

Then when shutting down I get these errors:
..
Afaik the first error comes from the udev init script.

The latter errors seem to be some sort of issue with the lvm init script.

That's odd: I've been running with lvm and the new patches for a while now, without any problems at all.

Unless you're on a version of udev greater than 180, it looks to me like either sysfs and udev-mount haven't been added to sysinit runlevel, or the patch to /etc/init.d/udev-mount hasn't been applied.
If udev is waiting for fsck, that sounds like its initscript has been patched, since patched (and with the option set) it needs localmount, which in turn needs fsck to have run. fsck needs dev which is provided by udev by default, but by udev-mount under the new setup.

So make sure udev-mount is in sysinit runlevel, and patched. (Add sysfs as well if that's not there, as it's needed by udev and must be in sysinit runlevel, which was where udev used to be.) For reference, here's what I've been using for quite a while:

But anyway things seem to work. I may have messed up something myself when I was using the earlymounts script. I was experimenting with the run levels and modified some init scripts. Just can't remember which files I edited.

Thank's for the help Steve._________________Everything can be done. There's just a longer delivery time for impossible projects.

Now that you mentioned the modifications to the init scripts I went through them again.
..
After making the above changes the booting up process is smooth again. No errors except consolekit, but hey that's another topic. :)
All the scripts are on correct runlevel. The udev version is 171-r6.

Now the only issue which remains is at the shutdown process. The

Code:

* udev: waiting for localmount (50 seconds)

messages have dissapeared though. But the issue with the lvm partitions is still there:

Well, glad udev isn't complaining any more; sounds like its dependencies are now ok. The "device or resource busy" messages mean something is still using the filesystems, so they can't be unmounted. The classic example is a daemon that doesn't chdir to root when it starts, meaning that its working directory elsewhere in the system is still open. Another typical case would be if another mount-point is still mounted on top of a mount-point one is trying to umount (eg /usr/local when trying to umount /usr.)

But I think you're right, that it's something to do with the earlymounts setup: I note from the second page that the extended lvm version involved a patch to /lib/rcscripts/addons/lvm-start.sh so you might want to check that file. You could always emerge -1 lvm2.

Quote:

Thanks for the help Steve.

You're very welcome :) Hope you get to a nice clean bootup and shutdown soon.

ok, as I've jumped on the /usr separation wagon, and the fact that the other days wine pulled udisks-1.99 which in turn must have udev-187, I think that it is time to lay down the options.
as I didn't followed the topic much, what are the options?_________________Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
ProjectFootball

ok, as I've jumped on the /usr separation wagon, and the fact that the other days wine pulled udisks-1.99 which in turn must have udev-187, I think that it is time to lay down the options.
as I didn't followed the topic much, what are the options?

Well, if you want to avoid an initramfs, you can use the patches in the first post to start udev after localmount, though they will no doubt need tweaking for udev-187. (Hmm must run update as not even showing it in-tree here;)

That works fine provided you didn't need an initramfs before. iow you have drivers for stuff needed to get past localmount built-in to kernel, ie motherboard hard-disk controller and root filesystem: the stuff we've always traditionally sorted out as first part of kernel build.

If you're running lvm, you'll need device-mapper built-in (not module) as well. (I think: that's how I've always done it, but then I've had that built-in for the last few years.) I'd imagine you'd need same for mdadm/luks. Note that rootfs cannot be on lvm, raid or encrypted, which was always the traditional cut-off point to need an initrd, but everything else can.

It's relatively simple to switch back and forth, so long as you change udev's runlevel at the same time. (You'll soon find out if you don't and will need to run rc-update under init=/bin/sh or /bin/bash after a remount,rw of root, then a remount,ro for safety, and a reboot.)

The patches are in much better shape since I tweaked it to start as soon after localmount as possible (it starts directly after here, and everything's been fine on my machine since last summer, apart from missing a revdep-rebuild out after a big KDE upgrade.)

Alternatively, setup an initramfs, and get used to maintaining it. You'll need to keep it in sync with kernel and system upgrades, though it should come down to running a script once you've got used to it and set it up how you want. mkinitcpio looks like a contender.

I don't want to go down that route, as it's another thing to go wrong, in several senses, and no-one has answered the further contradiction inherent in it being both a rescue-system, supplanting the traditional rootfs, and only a trivial few hundred kilobytes (supposedly.)

This is working with the new openrc-0.11.5, which added two new services, swapfiles and tmpfiles.setup, both of which only need localmount.

Just to be safe, I've added them to the before line in the patch (and my local file of course, which I've tested) when we're not using an initramfs: this is for future reliability, to ensure that udev starts before them, and anything that needs them.