Posted: Sun May 30, 2010 8:53 pm Post subject: Guide: Building a Stage1 using Gentoo and System Rescue CD

I noticed some people are a little frustrated that they can't do a Stage1 anymore, or properly..

I've been developing this script for years, finely tweaking stuff here and there. It works rather well on almost any computer. I am using ext4, but you can substitute it for the filesystem of your choice. I usually SSH into the system to build it, that's why "screen" is the first line.

I am using the 4.4 kernel, mostly do to the headers also being 4.4. I always use ~x86 or ~amd64. Also, this is the most recent LTS kernel available.

I switched to the Gentoo stage3 installs and building it from there. It seems to produce less issues while doing a ground-up install, especially with newer packages being a little.. grumpy. There's still a few things to emerge before the bootstrap.

The first thing I would suggest is the latest System Rescue CD. Download this and burn it (or install it on a USB stick), then boot from it. You can use SSH with this method, which is how I typically setup my systems. Just make sure you change the password so you can actually log on.

Code:

screen

# Get exact time
/etc/init.d/ntp-client start

# Make partitions
gdisk /dev/sda
## I've been using gdisk alot lately, mostly to fully utilize >2TB arrays. It works mostly like fdisk, but supports GPT. Grub does like it.

# {-- Since I have multiple systems, I have a separate portage server. So I have the boxes mount via NFS to the portage server. Just make sure you use "mount -t nfs -o vers=3,nolock" if you do so they don't fight over control. --} #

This has worked on a fewer older systems (3x Pentium II's, a couple Pentium III's, several Xeon 2.4's, a K6, etc.) without issues. It also works with a SunFire V100 with a few tweaks (hdc is HDD1 and hda is HDD2, using a Sun partition format, using a Debian CD to install, etc).

So far, I've used this same script to install on a couple WD MyBook Live (2T & 3T) drives, several x86 and x64 systems, SunFire V100's, a Mac Mini G4 1.5GHz, a Raspberry Pi, Cubieboard 1, and a ODROID-C1. My next aim is to put it on the C.H.I.P (which I have 2 of).

For some reason, Gentoo changed the way package.mask/unmask/use works. It now has directories with several different files in them. I find this extremely annoying, so I reverted them using this:

You can tweak them to your liking. I usually add these lines before chrooting:
[code]echo ">=sys-kernel/gentoo-sources-4.2" >> /mnt/gentoo/etc/portage/package.mask
echo ">=sys-kernel/linux-headers-4.2" >> /mnt/gentoo/etc/portage/package.mask
echo ">=sys-libs/db-5" >> /mnt/gentoo/etc/portage/package.mask
db-5 so it doesn't compile both db-4 and db-6 on your system. With some packages, it wants db-4, db-5, and db-6, which is ridiculous.

Changelog
EDIT: Forgot to add to edit /etc/make.conf. I usually set CFLAGS to the CPU itself (ex. K6, pentium3, etc).
EDIT2: Added unmerge libtool so libtool-2.2.10 can compile without being blocked. I also removed the "python-3" mask due to the recent stage1 having it built-in, so you're stuck with 2 versions of Python for now.
EDIT3: Added the latest stage1's package (they're using xz compression now).
EDIT4: Updated to kernel / kernel-headers 2.6.35
EDIT5: Updated the Stage1 location, tweaked the script a little (added emerge --sync and mirrorselect routines)
EDIT6: Added "emerge portage" early due to it blocking Python (update portage to latest, then emerge everything else); Removed "unmerge libtool" since it's not needed anymore.
EDIT7: Updated to kernel / kernel-headers 2.6.36; Added gcc-4.5.1 to be the default compiler (unmasked 4.5.1, unmerged 4.4.4 afterwards). This adds support to the "Atom" chips.
EDIT8: Well, GCC-4.5.1-r1 breaks Apache.. Damn.. So, I'm going back to 4.4.5, which makes the Atom config useless until 4.5.x is fixed. Crap.
EDIT9: Added python 2.7 to the masks table. I don't see the point of having 3 versions of Python in Gentoo (ex. 2.6, 2.7, 3.1). I also added the --update flag to the initial build of gcc (made more sense).
EDIT10: Updated to kernel / kernel-headers 2.6.38; Added new Gentoo file from the Funtoo project
EDIT11: Tweaked the script a bit at the end.. Masked <db-4.8 so only the latest is compiled.. I also masked python <2.7 and >=3.2. I also added at the end to update 2.6.6 to 2.7 and unmerge 2.6.6 so the latest 2.x would be 2.7.x.
EDIT12: Something changed in the btrfs code in 2.6.38 which is causing you to run out of disk space when 1/4 of it is actually used. I'm reverting back to .36 until this issue is fixed.
EDIT13: Been awhile. I updated the links to reflect on Gentoo's Stage3. I removed Python >=3.2 so the latest Python will be installed. I set it up so Kernel 2.6.39 is the latest along with the headers. I also switched it from btrfs to ext4 until btrfs fixes a ton of bugs I've been having here. I might switch it to zfs when it becomes more available. I also removed the atom processor issue due to gcc 4.5.3 coming with the current stage and it seems to be working fine.
EDIT14: Been a bit since my last update. Updated the links again. Removed the update to Python since it's in the latest stage3. I'm masking >=gcc 4.6 due to some serious issues compiling itself. I also added an emerge update to shadow, udev, and openrc before the bootstrap begins. I also removed the headers/kernel matching since vapier has been doing an excellent job at keeping them in sync.
EDIT15: Changed a few lines and did some updates. If using on a SPARC platform, use the 20120912 Stage3 along with the ISO due to segfaulting on newer releases (happening on SunFire V100s).
EDIT16: I edited the directories pertaining to make.conf. They are now in /etc/portage instead of /etc. I also unmasked >=gcc-4.6 due to it compiling normal once again on everything (using -march=native). I also added that gcc 4.5.4 be removed after 4.6.3 was compiled.
EDIT17: I updated the files to the recent versions. I also added an unmerge to the existing gentoo-sources and linux-headers.. along with supplying just the i486 file (since i686 has disappeared).
EDIT18: I updated to recent versions. I also removed the unmerge gentoo-sources and linux-headers unmerge areas from the script due to gentoo-sources not being included in the tarball.
EDIT19: I updated to recent versions. I found that on SPARC, it's best to boot from a Debian disc and build from there. I'm doing the same with a pSeries 520 server that I rescued.
EDIT20: Updated to recent kernel/headers versions. Modified mount layout to handle grub2. Cleaned up a few areas.
EDIT21: Made a few changes. GCC 4.8.3 is now available, so I compiled it, switched to it, and removed 4.7.3-r1. Also, in order to update udev, shadow, and openrc, you need to update procps and udev-init-scripts first.
EDIT22: Separated GCC from BINUTILS and GLIBC due to GCC needing to be built first. Found this out when building on different architectures.
EDIT23: Since GCC 4.9.3 is standard now, I removed the unmerge old GCC lines. Also, I updated it to Kernel 4.1 LTS.
EDIT24: Cleaned up the post a little and updated links.
EDIT25: Updated to compile GCC 5.3.0 and remove GCC 4.9.3. It also gets the @preserved-ebuilds out of the way so it doesn't continuously nag. I also removed the "a" flags so it doesn't ask you before emerging. I found it annoying when I'd copy and paste the entire chunk into a terminal window and not work as expected.
EDIT26: Added texinfo and iproute to the update area before bootstrap due to glibc-2.23-r2 segfaulting across x86 and x86_64 platforms (and possibly more)._________________Router(Nokia IP390,2GB RAM,160GB HDD,8xGigE ports,pfSense) | MyDT(Xeon X3470@3.4GHz,32GB RAM,6x2TB R5,GTX560Ti,2xLG BD-RE,Win10Pro)
MyLT(Asus G53SX,32GB RAM,2x2TB HDDs,BD-RE,Intel 6230,Win10Pro) | Wife(PnmIIX3@3.3GHz,8GB RAM,1TB HDD,DVDRW,Win10Pro)

Last edited by Simba7 on Tue May 10, 2016 2:08 pm; edited 75 times in total

I did my last install with a gentoo stage1 and I got used to the quirks fixing it first. It's really great to see that someone cares. Hope we don't start a flamewar on stage1 installs now _________________Please stand by - The mailer daemon is busy burning your messages in hell...

I find this topic interesting because when one enters in 'stage1 vs stage3 gentoo' into Google (and this has been for 6 years now ! - I search this at least one a year) this link has consistently been in the top three to five hits:

Why is that? Why are folks still curious about Stage1? Is it really just Gentoo 'mythology' born of ricer dreams and whispers across the net that 'real Gentooer's do Stage1'? (lol) Or is it a 'rite of passage' thing for Gentoo folk?

I've always understood the reasoning for deprecating the Stage1 from the Gentoo handbook for new users - no question, it IS conceptually hard for an inexperienced GNU/Linux user to understand why we want (need?) to rebuild so much stuff upfront to get to that satisfying first boot prompt. Moving to the Stage3 process probably brought in many 'new' users to Gentoo, which otherwise would have not had good outcomes with Stage1 their first time out.

Still, with all the multi-core processing power out there on the desktop, Stage1 installs really don't seem like they'd be much of a time hindrance any more.

Anyone know of more recent links/data either supporting or debunking Stage1/Stage3 mythology? Perhaps the speed of the newer hardware has all but wiped out any potential difference and it really is/has been a closed case for Stage3 at this point (?)

This one is ancient, and not particularly relevant. The issues he describes are not with the stage itself, but were with Portage back in those days being...well, powerful, but good and fucked (pardon my French).

Portage shaped up considerably from the time of writing that article. I do remember those pains he describes quite well, started Gentooing back in late '04

He might have had somewhat of a point back then, but not today. And even back then the point wasn't an advantage of a stage1, it was an exposition of non-trivial annoyances in portage._________________Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash

My next step will be "Gentoo From Scratch" : start from the first step of LinuxFromScratch (build a vanilla toolchain), then build python, then portage, then generate stage 1 _________________Kind regards,
Xavier Miller

Why is that? Why are folks still curious about Stage1? Is it really just Gentoo 'mythology' born of ricer dreams and whispers across the net that 'real Gentooer's do Stage1'? (lol) Or is it a 'rite of passage' thing for Gentoo folk?

Stage1 is the only way to get an optimized system for i586. All we have left today for 32-bit x86 are i486 and i686 stage3, the former being "minimal" and the latter "normal". There are still improvements to be had stepping from i486 to i586, but I guess it's sufficiently rare that they don't craft a stage3 for it. I got rid of my last i586 machine (AMD K6-3) a year or two back, but it's still relevant because there are many Via CPUs that are almost i686, but not quite. For these an i586 would be best.

I'm not sure how ricer the difference between i486 and i586 really is, on the Via CPUs._________________.sigs waste space and bandwidth

Nice observation depontius. and I guess that would be right, based on what's produced on the mirrors.

I do understand the link from '05 is now well-worn and as I mentioned, I'm guessing that any proc from a recent dual-core or above would not produce any sort of measurable difference as far as performance itself goes starting with a Stage3 vs a Stage 1. I only observe that the new powerful procs have probably minimized the time constraint involved in a Stage1 making it not quite the chore it once was.

I do understand the link from '05 is now well-worn and as I mentioned, I'm guessing that any proc from a recent dual-core or above would not produce any sort of measurable difference as far as performance itself goes starting with a Stage3 vs a Stage 1. I only observe that the new powerful procs have probably minimized the time constraint involved in a Stage1 making it not quite the chore it once was.

Thing is, that link doesn't even tout any performance benefits to a stage1. Its sole focus is the time taken to get an operational gentoo base system.

In the days of yore, sure, there were some screwy conflicts in portage that made patching things together after a base install relatively painful and time-consuming. Maybe in those instances a stage1 would have been less cumbersome.

But with Portage pretty damn solid now, and with you really needing to do nothing but the steps i mention above, the article is long since deprecated (that is, if it was ever valid to begin with - no idea one way or another. )

As far as performance goes, ponder for a sec just what all is touched with a stage1. I mean, you're rebuilding GCC and the other toolchain bits, rebuilding `system` target, so hey great, the minuscule components that make up a base system *may* be microseconds faster.

But now you're to the point where you have the same basic stuff installed as you would on a stage 3.

Anything else you build is going to have whatever optimisations you've set up in make.conf - regardless of whether youve gone a stage1 or stage3 route. So for i686 or better, what is the point?

I'm fully open to someone more knowledgeable than myself pointing out where this is wrong, but I just can't see where you'd have any performance benefit. You might have leaner build tools, but all of your REAL progs that actually serve a daily purpose, have the same optimizations on a stage3 as they would have had with a stage1.

i.e. no performance benefit. Once you get to x86_64, hell, aside from a small few things, even optimizing for a specific arch is going to yield negligible benefit. It's neat for ricer cred I suppose, but the tangible/measurable difference is tiny if not nil in many cases._________________Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash

Thing is, that link doesn't even tout any performance benefits to a stage1. Its sole focus is the time taken to get an operational gentoo base system.

Agreed - I did intend to imply the 'performance' of the installation procedure itself, not any ongoing (and unmeasurable) performance benefit of starting your baseline system as an S1 or S3. I didn't lay that out just right, my apologies. I was trying to say that, imho, the new procs are so powerful, it seems it would hardly would matter whether you did a S1 or S3 these days - where the difference used to be hours (days?) for bootstrap.h, now that time may be down to less than an hour (on a fast quad core system with decent ram compliment).

Simba7 originally posted that he sensed some were 'frustrated' by (I guess) not being able to do/complete (?) Stage1 installs (anymore?). I'm in agreement cach0rr0 on pretty much all your points. I was trying to understand Simba7's comment as to why users would (continue to?) be 'frustrated' about the topic at all, in any regard. I thanked him for the nice script he provided, which is not unlike procedures that have been around for many years. And I noted the antiquated link on Google that never seems to go away. So, what's the source of the frustration? If it's something users want to do, there's always been enough documentation and interest. There's also been additional excellent input like that of Simba7 around and if they really want to do it, they will/would anyway. It seems you can and always could do Stage1 if you really wanted to. I just didn't get what was meant by '...they can't do a Stage1 anymore, or properly.. '?

no worries, if my response came off as heated or argumentative or posturing of any sort, was most definitely not the intent! Just banter really.

Wanted to attempt to clear up the misconception that I'm sure some have that you'll see some amazing speed boost from stage1 plus -funroll-loops -fomg-rice-awesome if there was any doubt

If I were to hazard a guess, I'd say the "properly" bit comes from support for stage1 being dropped, as you've seen, and any updated scripts capable of doing this without considerable user input being kept "in-house". Don't know.

There's just such little need nowadays to do a stage1, and the demand so low, really not worth the headache of trying to get users through it successfully for such marginal benefit.

Even back then, I think I went the stage1 route maybe once, twice? Usually opted for stage2, and nowadays the stage3 seems like what the stage2 used to be. At least, I don't notice a difference._________________Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash

Run an "emerge --emptytree system" (twice if you really feel like it), and, finally,

Run an "emerge --emptytree world",

then there's not a single bit of difference in any executable code on your system as compared to a stage1 install. If any of you can point to a single element of the resultant system that's better off from a stage1, I'd seriously love to learn about it, as I am unaware of anything. (Well, one very small thing: there might be a few kb of extra cruft left behind from the stage3 that would take effort to find. Not hundreds of kb, either.)

I believe that statements like, "There's just such little need nowadays...," or, "Stage1 is the only way to get an optimized system for i586," are simply not correct. There is, in fact, no need, because there's no difference in the results. I know that there's some redundant compiling going on, but that doesn't affect the end result. Besides, nobody ever argues that the stage1 method is the most efficient way to get a fully optimized system.

A stage1 install, in the final analysis, is, I believe, just a more complicated, deprecated alternate method for achieving the same result. Doesn't mean it's not interesting to the truly hard core, though.

- John_________________I can confirm that I have received between 0 and 499 National Security Letters.

Last edited by John R. Graham on Wed Jun 02, 2010 10:35 am; edited 1 time in total

I believe that statements like, "There's just such little need nowadays...," or, "Stage1 is the only way to get an optimized system for i586," are simply not correct.

I would have to agree with you on this one, even though I stated your second quote. I guess that makes my statement a bit incorrect, but I was still thinking mostly of getting to CHOST="i586-pc-linux-gnu", and stage1 used to be the way to do that, though now that you mention it, I remember seeing "stage1 on stage3" directions a while back, which is basically what you've done._________________.sigs waste space and bandwidth

I believe that statements like, "There's just such little need nowadays...," or, "Stage1 is the only way to get an optimized system for i586," are simply not correct.

I am being diplomatic you see. And also cautious. Over the years without fail whenever I have sworn something is verifiably certain, someone has inevitably come in and said "actually, here's why you're wrong, and here's something you don't know about". So I never speak with any confidence of authority - it is unnecessary unless I am fighting a grizzly bear, or some such._________________Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash

You can still issue all the commands to do a stage1, using the stage3 tarball.
bootstrap.sh was still there last time I looked. You get to set your own CHOST too.

Using the stage3 tarball get arounf the problem of trying to use C++ before C++ has been built, which happens if you use the stage1 tarball. The only cost is the extra downlaod time as the stage3 tarball is bigger than the stage1._________________Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

I recently pulled out an old 386... which was a reason for me to use stage1...

and yes, it's still compiling.

Which is worse, the slow clock speed, or the fact that your address space is so limited? (The i386 didn't bring out the full address bus - I'm not sure anything did before the i686.) How much RAM do you have on that system? What's the clock speed?

I never did have a true 386, but I did get RedHat 7.2 installed on a "Blue Lightning" (386/486 hybrid) with 16MB RAM and a 420MB (not GB) hard drive._________________.sigs waste space and bandwidth

ToeiRei, I think what you're saying is that neither a stage1 nor a stage3 tarball existed with explicit i386 support but you've found a distro that will boot and you're bootstrapping a relatively contemporary stage1 tarball with CFLAGS appropriate for your computer. Is that close?

- John_________________I can confirm that I have received between 0 and 499 National Security Letters.