"So where are the TinyCoreLinux from scratch docs?" - Said the Lemur to the Forum.

"There are none" - Said Hats.

"That's just not right!" Said the Lemur. "I must build my system from scratch. I guess if there are no docs I will have to find my way through the jungle alone..."

This is a chronicle of that journey...

I have a processor setup with special needs. Actually I have several processors with special needs including different architectures. While, I could just compile LFS (Linux From Scratch) I would still have to recreate most of the optimization/customizations that TinyCoreLinux has - such as running from RAM, acting as a network boot, light footprint etc.

I have decided to take it upon myself to document the process of building TinyCoreLinux from scratch to help others that might be interested in producing this low-fat portable Linux. In the process I'll be able to compile for my architectures and get what I need out of it. It would be great if you could provide background info on how TinyCoreLinux was created. Otherwise I'll hack away until I figure it out on my own. Any help would be appreciated...

« Last Edit: December 28, 2008, 03:42:54 AM by Micon Frink »

Logged

Google is your friend: even if they are evil... Lemurs are never your friend. ~ Trust the Lemurs!!!

TC is designed to be a solid base in which the user can easily build around it. Do you mean you want to compile for another processor/platform?

In any case, here is my attempt to answer your question (afaik) since I suppose someone else may want to ask it:

Some 'history':- started building on finnix (might have had some some stuff from slitaz), then the rest was done on TC- init/startup scripts created/included/adapted, for busybox and others- included c++/fltk apps that were ported/recoded from rs's original lua/fltk ones- standard libs included in base- others built with compiletc which itself had its own early stages built on DSL-N (not sure what exactly was (re)built with it - I know at least the kernel was though)- additional fixes, changes

Perhaps you were looking for exact step by step instructions, rather than some history - iirc, there were problems that occurred on the (original) machine that prevents getting the starting data/steps. However, I believe that compiletc is quite robust. I'm pretty sure one could use that to build another TC "from the start", if you wish to go down that path... though I can't see why you'd want to atm.

And LFS is a good place to start and for reference - though I can't say how much was used as a guide

If you want to have it more easily and use TC as the base, optimizing everything has very small effects on the end result; by optimizing the parts you use the most, you can gain almost as much as by optimizing everything, but much more easily. This would mean busybox as it has the shell, the kernel, and then your specific apps and the libs they use.

This was my big bang! I used finnix to create my first cpio initramfs with hello.

Quote

1 ramfs, rootfs and initramfs2 October 17, 20053 Rob Landley <rob[AT]landley[DOT]net>4 =============================5 6 What is ramfs?7 --------------8 9 Ramfs is a very simple filesystem that exports Linux's disk caching10 mechanisms (the page cache and dentry cache) as a dynamically resizable11 RAM-based filesystem.12 13 Normally all files are cached in memory by Linux. Pages of data read from14 backing store (usually the block device the filesystem is mounted on) are kept15 around in case it's needed again, but marked as clean (freeable) in case the16 Virtual Memory system needs the memory for something else. Similarly, data17 written to files is marked clean as soon as it has been written to backing18 store, but kept around for caching purposes until the VM reallocates the19 memory. A similar mechanism (the dentry cache) greatly speeds up access to20 directories.21 22 With ramfs, there is no backing store. Files written into ramfs allocate23 dentries and page cache as usual, but there's nowhere to write them to.24 This means the pages are never marked clean, so they can't be freed by the25 VM when it's looking to recycle memory.26 27 The amount of code required to implement ramfs is tiny, because all the28 work is done by the existing Linux caching infrastructure. Basically,29 you're mounting the disk cache as a filesystem. Because of this, ramfs is not30 an optional component removable via menuconfig, since there would be negligible31 space savings.32 33 ramfs and ramdisk:34 ------------------35 36 The older "ram disk" mechanism created a synthetic block device out of37 an area of RAM and used it as backing store for a filesystem. This block38 device was of fixed size, so the filesystem mounted on it was of fixed39 size. Using a ram disk also required unnecessarily copying memory from the40 fake block device into the page cache (and copying changes back out), as well41 as creating and destroying dentries. Plus it needed a filesystem driver42 (such as ext2) to format and interpret this data.43 44 Compared to ramfs, this wastes memory (and memory bus bandwidth), creates45 unnecessary work for the CPU, and pollutes the CPU caches. (There are tricks46 to avoid this copying by playing with the page tables, but they're unpleasantly47 complicated and turn out to be about as expensive as the copying anyway.)48 More to the point, all the work ramfs is doing has to happen _anyway_,49 since all file access goes through the page and dentry caches. The RAM50 disk is simply unnecessary; ramfs is internally much simpler.51 52 Another reason ramdisks are semi-obsolete is that the introduction of53 loopback devices offered a more flexible and convenient way to create54 synthetic block devices, now from files instead of from chunks of memory.55 See losetup ( for details.56 57 ramfs and tmpfs:58 ----------------59 60 One downside of ramfs is you can keep writing data into it until you fill61 up all memory, and the VM can't free it because the VM thinks that files62 should get written to backing store (rather than swap space), but ramfs hasn't63 got any backing store. Because of this, only root (or a trusted user) should64 be allowed write access to a ramfs mount.65 66 A ramfs derivative called tmpfs was created to add size limits, and the ability67 to write the data to swap space. Normal users can be allowed write access to68 tmpfs mounts. See Documentation/filesystems/tmpfs.txt for more information.69 70 What is rootfs?71 ---------------72 73 Rootfs is a special instance of ramfs (or tmpfs, if that's enabled), which is74 always present in 2.6 systems. You can't unmount rootfs for approximately the75 same reason you can't kill the init process; rather than having special code76 to check for and handle an empty list, it's smaller and simpler for the kernel77 to just make sure certain lists can't become empty.78 79 Most systems just mount another filesystem over rootfs and ignore it. The80 amount of space an empty instance of ramfs takes up is tiny.81 82 What is initramfs?83 ------------------84 85 All 2.6 Linux kernels contain a gzipped "cpio" format archive, which is86 extracted into rootfs when the kernel boots up. After extracting, the kernel87 checks to see if rootfs contains a file "init", and if so it executes it as PID88 1. If found, this init process is responsible for bringing the system the89 rest of the way up, including locating and mounting the real root device (if90 any). If rootfs does not contain an init program after the embedded cpio91 archive is extracted into it, the kernel will fall through to the older code92 to locate and mount a root partition, then exec some variant of /sbin/init93 out of that.94 95 All this differs from the old initrd in several ways:96 97 - The old initrd was always a separate file, while the initramfs archive is98 linked into the linux kernel image. (The directory linux-*/usr is devoted99 to generating this archive during the build.)100 101 - The old initrd file was a gzipped filesystem image (in some file format,102 such as ext2, that needed a driver built into the kernel), while the new103 initramfs archive is a gzipped cpio archive (like tar only simpler,104 see cpio(1) and Documentation/early-userspace/buffer-format.txt). The105 kernel's cpio extraction code is not only extremely small, it's also106 __init text and data that can be discarded during the boot process.107 108 - The program run by the old initrd (which was called /initrd, not /init) did109 some setup and then returned to the kernel, while the init program from110 initramfs is not expected to return to the kernel. (If /init needs to hand111 off control it can overmount / with a new root device and exec another init112 program. See the switch_root utility, below.)113 114 - When switching another root device, initrd would pivot_root and then115 umount the ramdisk. But initramfs is rootfs: you can neither pivot_root116 rootfs, nor unmount it. Instead delete everything out of rootfs to117 free up the space (find -xdev / -exec rm '{}' ';'), overmount rootfs118 with the new root (cd /newmount; mount --move . /; chroot .), attach119 stdin/stdout/stderr to the new /dev/console, and exec the new init.120 121 Since this is a remarkably persnickety process (and involves deleting122 commands before you can run them), the klibc package introduced a helper123 program (utils/run_init.c) to do all this for you. Most other packages124 (such as busybox) have named this command "switch_root".125 126 Populating initramfs:127 ---------------------128 129 The 2.6 kernel build process always creates a gzipped cpio format initramfs130 archive and links it into the resulting kernel binary. By default, this131 archive is empty (consuming 134 bytes on x86).132 133 The config option CONFIG_INITRAMFS_SOURCE (for some reason buried under134 devices->block devices in menuconfig, and living in usr/Kconfig) can be used135 to specify a source for the initramfs archive, which will automatically be136 incorporated into the resulting binary. This option can point to an existing137 gzipped cpio archive, a directory containing files to be archived, or a text138 file specification such as the following example:139 140 dir /dev 755 0 0141 nod /dev/console 644 0 0 c 5 1142 nod /dev/loop0 644 0 0 b 7 0143 dir /bin 755 1000 1000144 slink /bin/sh busybox 777 0 0145 file /bin/busybox initramfs/busybox 755 0 0146 dir /proc 755 0 0147 dir /sys 755 0 0148 dir /mnt 755 0 0149 file /init initramfs/init.sh 755 0 0150 151 Run "usr/gen_init_cpio" (after the kernel build) to get a usage message152 documenting the above file format.153 154 One advantage of the configuration file is that root access is not required to155 set permissions or create device nodes in the new archive. (Note that those156 two example "file" entries expect to find files named "init.sh" and "busybox" in157 a directory called "initramfs", under the linux-2.6.* directory. See158 Documentation/early-userspace/README for more details.)159 160 The kernel does not depend on external cpio tools. If you specify a161 directory instead of a configuration file, the kernel's build infrastructure162 creates a configuration file from that directory (usr/Makefile calls163 scripts/gen_initramfs_list.sh), and proceeds to package up that directory164 using the config file (by feeding it to usr/gen_init_cpio, which is created165 from usr/gen_init_cpio.c). The kernel's build-time cpio creation code is166 entirely self-contained, and the kernel's boot-time extractor is also167 (obviously) self-contained.168 169 The one thing you might need external cpio utilities installed for is creating170 or extracting your own preprepared cpio files to feed to the kernel build171 (instead of a config file or directory).172 173 The following command line can extract a cpio image (either by the above script174 or by the kernel build) back into its component files:175 176 cpio -i -d -H newc -F initramfs_data.cpio --no-absolute-filenames177 178 The following shell script can create a prebuilt cpio archive you can179 use in place of the above config file:180 181 #!/bin/sh182 183 # Copyright 2006 Rob Landley <rob[AT]landley.net> and TimeSys Corporation[DOT]184 # Licensed under GPL version 2185 186 if [ $# -ne 2 ]187 then188 echo "usage: mkinitramfs directory imagename.cpio.gz"189 exit 1190 fi191 192 if [ -d "$1" ]193 then194 echo "creating $2 from $1"195 (cd "$1"; find . | cpio -o -H newc | gzip) > "$2"196 else197 echo "First argument must be a directory"198 exit 1199 fi200 201 Note: The cpio man page contains some bad advice that will break your initramfs202 archive if you follow it. It says "A typical way to generate the list203 of filenames is with the find command; you should give find the -depth option204 to minimize problems with permissions on directories that are unwritable or not205 searchable." Don't do this when creating initramfs.cpio.gz images, it won't206 work. The Linux kernel cpio extractor won't create files in a directory that207 doesn't exist, so the directory entries must go before the files that go in208 those directories. The above script gets them in the right order.209 210 External initramfs images:211 --------------------------212 213 If the kernel has initrd support enabled, an external cpio.gz archive can also214 be passed into a 2.6 kernel in place of an initrd. In this case, the kernel215 will autodetect the type (initramfs, not initrd) and extract the external cpio216 archive into rootfs before trying to run /init.217 218 This has the memory efficiency advantages of initramfs (no ramdisk block219 device) but the separate packaging of initrd (which is nice if you have220 non-GPL code you'd like to run from initramfs, without conflating it with221 the GPL licensed Linux kernel binary).222 223 It can also be used to supplement the kernel's built-in initramfs image. The224 files in the external archive will overwrite any conflicting files in225 the built-in initramfs archive. Some distributors also prefer to customize226 a single kernel image with task-specific initramfs images, without recompiling.227 228 Contents of initramfs:229 ----------------------230 231 An initramfs archive is a complete self-contained root filesystem for Linux.232 If you don't already understand what shared libraries, devices, and paths233 you need to get a minimal root filesystem up and running, here are some234 references:235 http://www.tldp.org/HOWTO/Bootdisk-HOWTO/236 http://www.tldp.org/HOWTO/From-PowerUp-To-Bash-Prompt-HOWTO.html237 http://www.linuxfromscratch.org/lfs/view/stable/238 239 The "klibc" package (http://www.kernel.org/pub/linux/libs/klibc) is240 designed to be a tiny C library to statically link early userspace241 code against, along with some related utilities. It is BSD licensed.242 243 I use uClibc (http://www.uclibc.org) and busybox (http://www.busybox.net)244 myself. These are LGPL and GPL, respectively. (A self-contained initramfs245 package is planned for the busybox 1.3 release.)246 247 In theory you could use glibc, but that's not well suited for small embedded248 uses like this. (A "hello world" program statically linked against glibc is249 over 400k. With uClibc it's 7k. Also note that glibc dlopens libnss to do250 name lookups, even when otherwise statically linked.)251 252 A good first step is to get initramfs to run a statically linked "hello world"253 program as init, and test it under an emulator like qemu (www.qemu.org) or254 User Mode Linux, like so:255 256 cat > hello.c << EOF257 #include <stdio.h>258 #include <unistd.h>259 260 int main(int argc, char *argv[])261 {262 printf("Hello world!\n");263 sleep(999999999);264 }265 EOF266 gcc -static hello2.c -o init267 echo init | cpio -o -H newc | gzip > test.cpio.gz268 # Testing external initramfs using the initrd loading mechanism.269 qemu -kernel /boot/vmlinuz -initrd test.cpio.gz /dev/zero270 271 When debugging a normal root filesystem, it's nice to be able to boot with272 "init=/bin/sh". The initramfs equivalent is "rdinit=/bin/sh", and it's273 just as useful.274 275 Why cpio rather than tar?276 -------------------------277 278 This decision was made back in December, 2001. The discussion started here:279 280 http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.2/1538.html281 282 And spawned a second thread (specifically on tar vs cpio), starting here:283 284 http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.2/1587.html285 286 The quick and dirty summary version (which is no substitute for reading287 the above threads) is:288 289 1) cpio is a standard. It's decades old (from the AT&T days), and already290 widely used on Linux (inside RPM, Red Hat's device driver disks). Here's291 a Linux Journal article about it from 1996:292 293 http://www.linuxjournal.com/article/1213294 295 It's not as popular as tar because the traditional cpio command line tools296 require _truly_hideous_ command line arguments. But that says nothing297 either way about the archive format, and there are alternative tools,298 such as:299 300 http://freshmeat.net/projects/afio/301 302 2) The cpio archive format chosen by the kernel is simpler and cleaner (and303 thus easier to create and parse) than any of the (literally dozens of)304 various tar archive formats. The complete initramfs archive format is305 explained in buffer-format.txt, created in usr/gen_init_cpio.c, and306 extracted in init/initramfs.c. All three together come to less than 26k307 total of human-readable text.308 309 3) The GNU project standardizing on tar is approximately as relevant as310 Windows standardizing on zip. Linux is not part of either, and is free311 to make its own technical decisions.312 313 4) Since this is a kernel internal format, it could easily have been314 something brand new. The kernel provides its own tools to create and315 extract this format anyway. Using an existing standard was preferable,316 but not essential.317 318 5) Al Viro made the decision (quote: "tar is ugly as hell and not going to be319 supported on the kernel side"):320 321 http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.2/1540.html322 323 explained his reasoning:324 325 http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.2/1550.html326 http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.2/1638.html327 328 and, most importantly, designed and implemented the initramfs code.329 330 Future directions:331 ------------------332 333 Today (2.6.16), initramfs is always compiled in, but not always used. The334 kernel falls back to legacy boot code that is reached only if initramfs does335 not contain an /init program. The fallback is legacy code, there to ensure a336 smooth transition and allowing early boot functionality to gradually move to337 "early userspace" (I.E. initramfs).338 339 The move to early userspace is necessary because finding and mounting the real340 root device is complex. Root partitions can span multiple devices (raid or341 separate journal). They can be out on the network (requiring dhcp, setting a342 specific MAC address, logging into a server, etc). They can live on removable343 media, with dynamically allocated major/minor numbers and persistent naming344 issues requiring a full udev implementation to sort out. They can be345 compressed, encrypted, copy-on-write, loopback mounted, strangely partitioned,346 and so on.347 348 This kind of complexity (which inevitably includes policy) is rightly handled349 in userspace. Both klibc and busybox/uClibc are working on simple initramfs350 packages to drop into a kernel build.351 352 The klibc package has now been accepted into Andrew Morton's 2.6.17-mm tree.353 The kernel's current early boot code (partition detection, etc) will probably354 be migrated into a default initramfs, automatically created and used by the355 kernel build.

I opted not to use uClibc and the external cpio for easy remastering.Most of the concepts of design philosophy I created over the last five years as the main developer of the Damn Small Linux project. I ported them and improved them. Actually when I created DSL-N, it too was a tiny core, but I was strongly discouraged from releasing it. The new capability inherit in the 2.6 kernel was exactly what I was looking for. to host my philosophy. Originally I also deployed Siltaz lzma kernel patches for even smaller core, but later dropped lzma for improved speed in booting.

Especially thanks to Hats for the edits to his post which make things much clearer. I understand much better the philosophical composition choices made. Now the real question is finding the list of customize script from the base system.

Roberts:Didn't you keep some kind of development log or something? Seems kinda irresponsible to just post source code without compilation instructions. I guess this is a work-in-progress and not yet well documented. It'd be nice to know exactly what was done from the base system.

- FRINK

« Last Edit: December 28, 2008, 03:51:10 AM by Micon Frink »

Logged

Google is your friend: even if they are evil... Lemurs are never your friend. ~ Trust the Lemurs!!!

OK! ...So I've been snooping around in TinyCore taking it apart. Basically, snooping because nobody is explaining how it works and I've gotta compile for some really different situations... Anyhow, so far I've found the following:

/init is linked to /bin/busybox (this is the initial process run by the kernel)/bin/busybox runs /etc/inittab/etc/inittab starts /etc/init.d/rcS which is linked to /etc/init.d/tc-config/etc/init.d/tc-config is 468 line init script that brings everything up...

Most of /etc/init.d is custom:

/etc/init.d/tc-config - startup script/etc/init.d/rc.shutdown - shutdown script/etc/init.d/dhcp.sh - dhcp init script (called by rcS)/etc/init.d/tc-functions - common include file/etc/init.d/tc-restore.sh - script to restore personal data/etc/init.d/functions.lua - may be residue left over from DSL since there is no LUA in TinyCore/etc/init.d/dropbear - ssh server

The system seems pretty much stock from make install except for these scripts.But I've not gone through with a fine-toothed comb...

- FRINK

Logged

Google is your friend: even if they are evil... Lemurs are never your friend. ~ Trust the Lemurs!!!

everything in /opt/ is custom.everything in /etc/skel/ is custom. (these are all dot files so you have to use ls -a)everything in /root/ is custom. (these are all dot files so you have to use ls -a)

---

To my knowledge these are all the custom files present in the system. I'm sure there are more that I missed. It's a bit frustrating going through things this way but I guess if there is no documentation someone needs to do it. "Leave it to the Lemurs" - I always say.

- FRINK

Coding without documentation is like rice without curry... Oh well, guess I'm providing the curry...

« Last Edit: December 28, 2008, 03:54:43 AM by Micon Frink »

Logged

Google is your friend: even if they are evil... Lemurs are never your friend. ~ Trust the Lemurs!!!

Building software from source should be a community affair. However, the documentation for building the source into a fully functioning TinyCoreLinux is missing. How are others supposed to download the system and improve it to submit changes without a basic "howto from scratch"? Thus I'm going at it piece by piece. You can help document the thing or sit back and watch. It's really your choice...

- FRINK

Logged

Google is your friend: even if they are evil... Lemurs are never your friend. ~ Trust the Lemurs!!!

Hints on starting rolling your own was given, and it was already explained that there was no build system, nor retrievable 'log'. And that busybox was used, scripts are plaintext, etc. I'll still say that I am not sure what you are trying to ask for (vague). To answer your previous question though, currently improvements are done by modifying the corresponding parts of the system.

Hats, I have the utmost respect for the community and for your lack of understanding. Thanks for your input - I really do appreciate it. Sorry I'm not communicating in your brand of English. I'll post a build script when I've got one I guess...

Looking at the unique parts I wonder if buildroot was have used to generate the base system. If this is true it would be helpful to have the buildroot config files. However, I don't find these in the source so perhaps the base file system was made by hand. Then again the init scripts are not in the src either. Hm... I'll be testing out this theory next week and post my results...

I noticed that most of the contents of /usr/bin/ are custom scripts and programs.

These scripts are not included in the TinyCoreLinux source directory so they will need to be copied from the current distro ISO directly. Ideally these should be placed in a tarball in the source tree. Perhaps I can create that tarball.

LEGALITY:Many of these scripts do not bear proper reference to copyright. Some of them don't seem to originate with TinyCoreLinux but have no reference information to their origination. Could it be that these are carry-over from DSL?

All of these are original work for TinyCoreLinux and their source resides in the fltk_projects/ directory of the source tree. There is however, one exception: watcher is in the base of the source tree. None of these apps have a Makefile but usually consist of three files <foo>.cxx, <foo>.h and <foo>.fl

LEGALITY:Some of these programs seem to have originated with DSL but there the references in source are incomplete. I'm not sure if there are any license issues with this as both distros use the same license.

SIDE NOTE: The source tree and releases have been removed from the tinycorelinux.com server and now reside on distro.ibiblio.org/pub/linux/distributions/tinycorelinux/. This was not noted on the main site so I thought I'd mention it here. I was frustrated for a minutes trying to find the source. The link to the download directory on the tinycorelinux.com website has been changed to reflect this new location. Not a big deal just a minor frustration that takes up space here...

- FRINK

« Last Edit: December 28, 2008, 04:31:17 AM by Micon Frink »

Logged

Google is your friend: even if they are evil... Lemurs are never your friend. ~ Trust the Lemurs!!!

Yeah, your're right I don't always put a copywrite banner on every dinky plain text script that I write. And that most of Tiny Core consists of my custom stuff. So what.Now this thread is turned into your personal blog area, where you are the critic down to the minutiae. I find your actions and attitude very curious. You seem to have nothing to contribute but only to complain about absolutely everything.

Yeah, your're right I don't always put a copywrite banner on every dinky plain text script that I write. And that most of Tiny Core consists of my custom stuff. So what.Now this thread is turned into your personal blog area, where you are the critic down to the minutiae. I find your actions and attitude very curious. You seem to have nothing to contribute but only to complain about absolutely everything.

whoa whoa whoa! this is totally unnecessary... no one's said anything to make it necessary to come down on him like this. no one's being disrespectful. this same thing happened with puppy linux years ago... and i've always known dsl and roberts to be far less uptight about these things. in general, i've always counted on the dsl community to if nothing else, have a more professional image, or at least a more meticulous, less sloppy one.

as for "so what," i mean no disrespect either. only that in the open source world, it's expected that you be able to modify things, just as dsl was (THANKFULLY) modified by roberts, and thus made into the best dsl that dsl ever was. some people just take code and aren't careful with it. they mean well, and no one's being accused of anything. only sometimes people want to make a new distro, or adapt it to a new platform (that's the case here,) and they want to follow all the rules.

that means they have to ask where the stuff comes from. so if no one provided copyright information, that's not a problem for roberts, because roberts is the author.

it's only a problem for the original poster, who wants to respect roberts copyright and use it per a gpl license or something. he just wants to respect the rules. so unless roberts gives him formal permission (or writes copyright information on each script, or as a collection of scripts) then anyone trying to respect roberts (and the gpl) has to ask more than once.

please don't be mad at him for that. if anything, he should be applauded. and it is certainly not disrespectful.