What is the most important component of an operating system? Well, arguably, it is the kernel. And if something
goes wrong with the kernel and its associated files, you will not be able to your box, right? Example, the
initrd problem we talked about a few years ago.

But what about glibc? What happens if you delete some of the important C libraries that power your booted
system? How would you go about recovering from such a fiasco? In this tutorial, I am going to engage in some
semi-mild hackery and teach you all sorts of nice self-rescue methods for fixing turbo-broken systems using
busybox and a few other fancy tricks. If you're still wondering, this be a Linux guide. After me.

Scenario

All right. Let's talk about our system and how we can easily destroy it. To wit, a little bit of background on
how Linux works under da hood. I've already explained this in detail in my fourth hacking tutorial, but here's a brief recap. And please excuse the lingo, I
am not going to be 100% accurate here.

Linux programs normally come in two flavors - executables and shared libraries, usually dynamically linked. The
programs are compiled in the Executable and Linkable Format (ELF), which defines how the program code should be
loaded into memory and executed.

The loading of data into memory falls on the shoulders of two other programs, the standard C library (glibc)
and the dynamic linker (ld), which calls on shared libraries with which the programs have been compiled. All
the information on the file structure, including segments, sections, symbols, and such, are stored in the ELF,
and can be parsed using the readelf command.

If you are to remove either glibc or ld, most programs will stop functioning, because they won't be able to
load into memory and run correctly. So what happens if you delete, say /usr/lib64/glibc.so.6 file, which
happens to be this highly critical library?

Anyhow, this is what will happen - I will purposefully NOT write the command that
moves the libc library away from its natural place so you do not accidentally copy it and ruin your box. But
once you do that, any subsequent command you try to run that has not already been loaded into memory will fail:

And if we try to fix by moving the glibc back into its place, again, no joy:

Catch 22. We need libc.so.6 to load other libraries, but it's gone. Hence, nothing will work. At this point,
most people will panic, reboot and then panic some more. Again, no biggie, boot into a live session, copy the files into their right place, and Bob's your uncle. But we will
discuss that separately. Let's focus on our special tools of the trade - Busybox and the LD_PRELOAD hack. But
first, a disclaimer.

Caveats

Now, before we move forward, let's clarify a few points. One, you should always have full data backups and system images, for exactly this type of
scenarios. Two, you should never delete critical system files, so be careful with your sudo or root rights.
Three, you need to be comfortable working on the command line. Four, you do need an open root shell for the kind of work we will do today, otherwise it won't really work. Last
but not the least, do not try this at home unless you really know what you're doing. Bonus, you can always fix
broken Linux systems by booting into a live session. Now, onwards.

Busybox

If you're not familiar with busybox, it's a nice tool. First, it's a special snowflake. Unlike most other
programs, it is designed to run on its own, without any dependency on shared libraries. The reason is, busybox
is the default shell in small, embedded devices, and it needs to be tight and independent, so you can perform
critical functions without relying on tons of data, which you can't store in the highly limited memory on
embedded hardware.

If it's not installed, get it. For instance, on Fedora 23, it's not there, so you
will need to install the whole 1.2MB worth of its equity. Then, let's explore the program's signature. Most
importantly, it is not a dynamically linked executable.

The program is entirely self-dependent. It does not have any fancy symbols, group sections, dynamic section,
relocations, or supported unwind sections. If you're not a developer, some of this stuff may look arcane, but
what it tells us is that busybox can live on its own, without depending on glibc and ld.

The program comes with tons of builtins. Basically, pretty much any command you can think of, Busybox has. For
instance, if you want to see the list of loaded kernel modules, you will then run busybox lsmod. It also comes with important functions like su, mv, cp, ifconfig, and others.
In essence, you should be able to do everything.

And then, the fix simply becomes (as defined in our example):

busybox mv /lib64/libc.so.6.bak /lib64/libc.so.6

Root permissions

Now, all of the above will work only IF you have a root shell inside which you want to run your busybox. It
won't work if you try to do system changes from a user shell. The reasons are obvious, because that would
enable pretty much anyone to gain root access.

A trick would be to setuid on the busybox binary - by root, BEFORE a fiasco. As
it is owned by root, then in theory, yes, anyone running the shell should be able to do things as root.
However, modern security mechanisms like SELinux and AppArmor will hamper your effort, and then, still, this is
not something that should be too easily achievable, for obvious reasons. You really don't want to have a shell
that allows users root access. Think about it. And be careful.

sudo chmod +s /usr/sbin/busybox

Not a silver bullet

Then, even if you do have root, you might not be able to do everything, especially if you've destroyed such a
fine and delicate flower like glibc. I do admit it's not the typical usecase for busybox, but if you try to su
into root with busybox on a system that doesn't have its glibc in the right place, you will fail. Rightly so.

Boot games

You may try to be extra clever and login into busybox in the early stages of the system boot. The same way
you would declare init=/bin/bash on a typical and functional Linux system to reset
the root password, you might go for this here. Except with no glibc, you will probably get a nice, juicy kernel
panic. Then, the same goes if you try to invoke busybox as your init.

What if you don't have busybox?

On some systems, busybox may not be available. Ideally, yes, if cacky hits the fan, your system, while trying
to boot, will drop into a busybox shell, and let you do all sorts of nice magic. But if it's not there, there
are still some cool hidden goodies to help you fix your broken boxen.

LD_PRELOAD

In our example, we only destroyed one file, but ld.so should still be functional. Which means it can load
objects, but the problem is, glibc loads first. Luckily, you can use the LD_PRELOAD environment variable to tell your running shell to load specific objects first,
before others, and this way, hopefully get around the glibc problem.

Speaking of glibc, there is more than one instance of the glibc shared library on the disk, and a few symbolic
links, which means that even if you have moved the glibc.so.6 object away from its rightful place, we still
have an ability to try to recover our system. What we essentially need is to be able to move or copy a file.
Indeed, on most Linux systems, with a slight variation in the location and naming convention, there should be
another copy of the standard library, with the full name. Using Fedora as an example, then you would have
libc.so.6 and libc-2.22.so. Sometimes, the two could be actually symbolically linked. If you're lucky, then the
copy or move fix becomes:

Healthy practices

All right, and now that we've learned all of the above, there are some things you should remember and practice.
Never delete system files, even if you do need them to go away. Move them aside,
to start with, so you can move them back if needed. The exception to the rule are the glibc and the dynamic
linker, which if gone, will cause a lot of problems with your system.

Never do fancy pipes and xargs removals of files when working under system directories. Always echo the outfit
first, always type commands with the hash (#) key as the first character, so if you accidentally hit the Enter
key, nothing bad happens. Try testing with innocent and unimportant files.

Never use environment variables to declare paths and filenames. You
could end up deleting everything. Always go for full paths and names. Always.

Conclusion

And here we are, at the end of this long and nerdy article. But hopefully, you have learned a lot, and I really
mean, a lot of new stuff here: how Linux works, the intricacies of the ELF, what not to do with your system
files and how to avoid tears and pain, and then a slew of recovery methods, including backups, system images,
live CD games, and in more detail, our work with busybox and LD_PRELOAD.

We also learned a little more about the limitations of trying to recover from a badly botched situation, and
why you should potentially have a root shell running on your box, and then all the reasons, why you really
shouldn't let your users have any access to root files. 'Tis a double-edged sword. Anyhow, I'd like to believe
you were enlightened as well as entertained. See you around for more gentle hackerology.