How to master Linux the proper way

In order, for you, to become a master of Linux, there are two things that need to happen. One, you need to invest time studying Linux. Two, you need to do it the right way. Now, a six hundred and eight dollar question. What is the right way?

So let’s say you meet all the preconditions of being someone interesting in expanding their knowledge of Linux. There are many paths, as many as there are distributions, spins, forks, and graphical environments, all combined, to you reaching that level where your skills are admirable. Moreover, you may decide to pursue a career in Linux, which is always a wise thing. So, how do you do it most effectively?

The easy way

If you’re not a great risk taker and you get easily frustrated with the lack of success, you may decide to embark on a shallowly sloped learning curve that begins with an innocent exploration of a newb-friendly distribution, something like Mint, Ubuntu, openSUSE, or any one of these. Depending on your luck, things may go smoothly. At this stage, you will probably want as few problems as possible, however, the learning demons will demand that there be errors and bugs, so that you be forced to learn.

Even so, this kind of study will probably familiarize you well with the Linux desktop, most often a particular version thereof. Your initial choice will stick for a long while, and you will get to know and do things the way your distribution demands it.

The hard way

The total opposite of beginning with a rich GUI desktop is to try one of the less friendly distributions, like Arch or Gentoo. Here, you will be spared no gory mess, and you will spend hours tinkering, often running commands you do not fully understand, waiting for things to compile or fail, searching for errors to figure out how you may fix them.

This shellshock method may have its merits, but just like the first one, it will firmly set your ways for years to come, blinding you to the possibilities and hardening the way your mind works and approaches problems.

The RIGHT way

Neither of these methods will really make you a master. In fact, they will make you the opposite of a master. You will become a one-degree expert. You will know well what you want and need, but your thinking will be limited to a very narrow subset of who-cares. Let me elaborate.

We will first define what being a Linux master is. Some may say, being able to administer the Linux box, top to bottom and then back. Firewalls, access control lists, startup scripts, you name it. How noble, but entirely misplaced.

Next, you may think repeating 43 commands letter to letter in order to get your Wireless working counts as a feat. In my lexicon, it counts as futility. Not only should you not waste time trying to configure hardware, as it serves no higher purposes, insisting on this kind of approach will make you all too used to manual, laborious tasks, you will get used to higher-than-needed levels of efforts for most of the problems you are trying to solve, and you will dig so deep down, you won’t be able to see anything around.

Third, if you think knowing a bunch of commands and scripts makes someone an expert, then you are in for a surprise. Being a great Linux master is definitely not about remembering all of the little sed and awk flags by heart, or being able to cron yourself to sleep. Nope.

So what is being a master all about really?

All right. Let’s begin with an example. Say you’re facing a network related issue. Your torrent client is not maxing its available connections or bandwidth. As a Linux guy/gal, you want to solve this problem, because it irks you. What do you do?

Unlike Windows folks, and yet, very much like them, you Google for a quick solution. Of course, you land in a dozen different forums, where a million other people, plagued by completely random and utterly unrelated problems are discussing similar and tangent issues. In the whirling, boiling sea of answers, you find a bunch of suggestions about increasing the window scale, the number of TCP connections and more. Because it all sounds so sexy and involves echoing parameters under /proc, you are very much excited, and you run them. It works. You’re happy. Your Hax0r lever has just increased a notch.

You iz a hax0r, be proud.

What you have just done is the greatest possible mistake, and distanced yourself from becoming a master one more step.

The thing is, most people have no real clue what most tunables under /proc and such really do. And yet, they are perfectly comfortable issuing commands that alter them. Most people do not understand how the network stack works, what different layers do, and how things may be connected. Going up, looking more widely, most people fail to make the necessary connection between different applications and their operating system, their system and their hardware, the router, the modem, the ISP, and beyond. Most people restrict themselves to knowing Linux as the four walls of their little desktop, and that’s it. They may be really leet in that tiny square of ignorance, but that’s not what being a Linux master really means.

Being a Linux master is understanding the big picture. Unlike most folks out there monkey-clicking GUI, you have the distinct advantage of thinking about content and style, or rather functionality and data, as separate entities. If you can do this, you can abstractize your problems. If you can do this, then your torrent slowness is no longer a matter of tuning something until it’s gone, it’s trying to figure out whether there is a problem in the first place! And if you’re led to believe that there is, then figuring out which component is at fault.

You’re better than that; image taken from Wikimedia Commons, licensed under CC BY 1.0.

I’ve seen so many people fail this little test. They may be brilliant engineers with advanced degrees in this and that, but they are awful problems solvers, because they cannot bring their minds to think on a level that is above the manifestation of a potential problem. Being able to compare healthy and unhealthy systems, performing a component search, or a simple design of experiment sounds trivial or boring or unrelated, but these are in fact the most important things in any problem investigation, and the most precious tool in the hands of the Linux master.

A Linux master does not restrict themselves to just debugging a problem. They try to fix the world around the problem, and then, make the problem go away forever. This means that a Linux master is not just an expert in Python and Java and /proc, he or she is also extremely knowledgeable in everything there is, anything that could possibly affect their field of work. Even Windows.

So you have a problem with your torrents, then you should be an expert on the kernel as well as network, and understand the network protocols in solid depth. You should also understand the global networking infrastructure. You should understand the application you’re currently working with. And if they land you in a completely new setup tomorrow, you will figure out everything, just because you have the right intuition to solving problems.

Extrapolate now, please. If you do not have even a basic understanding of storage solutions, like NAS, SAN and friends, or network file systems, then your ability to resolve issues in a multi-user environment with shared resources will be extremely weak, even though all of the problems WILL manifest in the Linux you know so well. You cannot be a security expert, if you do not understand process and memory management well. Just because you can configure Snort, AppArmor or SELinux does not make you into anything more than a knowledgeable configuration trooper.

You will not be a Linux master if you don’t have passable skills of virtualization technologies, all of them. You will not gain this status if you keep thinking about problems in terms of isolated instances or snippets of system messages or vague complaints from your users, maybe yourself. Most definitely, you will not become an expert unless you’re really, really lazy.

Being lazy is the key; image taken and adapted from Wikimedia Commons, licensed under CC BY 2.0.

That’s right. Lazy people hate work. So they invent everything they can to make themselves work less. Lazy people are the true innovators, because they will tailor scalable, long-lasting solutions that are fully automated and can be maintained by someone below their skill level.

This means that the more you know the more you should share and teach others. Never make yourself the one indispensable link in the chain, because they will never let you go, and you will remain stuck doing the same thing over and over. That’s boring. A master is always bored. After they’ve done it once, they want to move, never fiddle with the routine again. Routine is for someone else.

In fact, as a master, you will do very little. You will sort of dispense magic, and things will happen, because you have the vision, and you have the ability to unravel complex problems. Someone may spend days trying to bruteforce their way using lots of commands and scripts. You may suggest a trivial thing they would never think of, and it will fix everything in a fraction of time. Just because you hold end-to-end knowledge in your hands, and you can make that diabolic, magical connection between a flickering LED on a storage segment in another data center and users complaining about slow version control check-in times half across the globe. Because you know how it all connects. Hardware, cables, appliances, radical new solutions, algorithms, everything. And maybe some Linux too.

For you, Linux should be the glue, the manifestation of this broad understanding and knowledge. It works the other way around, too. But the chances of someone not being a Linux person and having this amazing ability is much, much lower. That’s just the way the world works. The modularity and complexity of Linux can teach you simplicity and abstract thinking. Provided you do the right things.

I’m convinced, what now?

Now, how do you actually achieve all of them above? There’s a lot you can do. What you should never do is assume that lots of keyboard typing is synonymous with productivity. Get that out of your head. You should also not feel like being a hero for having figured out Bluetooth from command line, all on your own.

My suggestion, and it’s entirely personal, take it or leave it, is to create a rich, complex world, full of Linux, and then use it to slowly build your master knowledge. We shall begin with the choice of a distribution. No, it’s not going to be Ubuntu. Nor Arch. Nor Gentoo. It’s going to be RedHat, and for those who want stuff for free, CentOS. Debian is a second option.

If you want to be a real expert, then you need to study in an environment that simulates real-life scenarios as much as possible. To wit, you want an enterprise flavor of Linux, and RedHat is the most obvious choice here. SUSE is another good candidate, as it kind of rules the supercomputing world. Ubuntu is slowly getting there, but it’s not there yet. Debian is fairly popular for Web hosting, but it’s not in the same league as the big RedHat and Novell players. Still ok. If you want the high-performance computing environment, if you want by-the-book problems, then try RedHat and family first.

Now that you have selected a distribution, install some five or six instances. Your first challenge, learning how to do automated installs, so you don’t come across as a little monkey. You should make it all virtualized, of course, because that will force you to learn another technology. More scripting, more automation. We’re just getting started.

Once the servers are running, how about you throw in shared storage and a directory service, something like NFS and NIS or LDAP. Host the user home directories on that shared server. Add some firewall rules to limit access. Throw in a private DHCP and DNS server. This will force you to spend days learning about protocols you never thought important and relevant. Good. Nothing like being at the edge of the comfort zone to get the best from yourself.

While you’re sweating, you will occasionally make use of the command line. This will be your gateway to Linux knowledge. You will learn about TCP, UDP, ARP, and DHCP, plus a dozen other tools and utilities, but you will actually be doing that in Linux, with Linux. That’s how it’s done.

Once your private little computer farm has a dozen machines, it’s time to introduce still more noise and complexity. Web, FTP, SQL, throw everything in there, and keep suffering some more, and try to figure the best, easiest way to wriggle out of your misery. Try to be lazy so that your solutions end up being ironclad and eternal.

You will not be able to replicate the work conditions that real companies have, not in the terms of network throughput, complexity or actual performance, but you will surely learn a whole lot more than you ever will by just playing with your one little machine. You may doubt this advice, but once you start sorting out RPC errors to your NFS server and analyzing packet logs and stepping through application crash cores, you will start to make the abstract connections that are absolutely necessary for you to become a master.

In the end, you will not know any one field too well, although you will develop affinity for a few topics, but you will know a lot about everything. That’s your first step to becoming a master. A jack of all trades, and you have the best chances of succeeding, because you’re using Linux, and that’s the best tool for teaching someone the skills. After a couple of years, you will be able to relax, because the abstract way of problem solving will have become your second nature, and then and only then, will you be able to invest all of your intellectual powers in becoming a ninja in what you like most.

Still, as you expand, you will encounter new limits, new challenges. Leave no tunable under /proc and /sys unturned. Learn everything you can about kernel internals, even if you never write a single line of code. You must, because otherwise, you will be just a borg pressing buttons and hoping for the best.

I’ve written a lot, so you may be wondering what the core message is. Simple. A sys admin or a smart guy is like a car driver trying to replace a flat tire. A master is someone trying to create a new type of rubber.

Conclusion

If you’re even semi-serious about becoming professionally involved in Linux, or just gaining mastery of this marvelous world of technology, then Linux is your best tool, your best bet, your free and colorful playfield and sandbox. Start big, because it won’t work otherwise. Create yourself a world and then subvert it to your will.

Finally, everything you learn, share, pass on. That’s the real test of your knowledge. Not until you’ve taught someone else will you really know how good you are. But if you’re just looking for technical details, a recipe, and you don’t care about the enlightened spiritual message of this essay, then take the following from this article. Be lazy, be easily bored, stick to the enterprise flavors, learn everything, and create complex setups that go way beyond anything you normally need. Only when you push yourself to the limit will you know how good you are half way there.

He’s right. Nothing was ever quite as humbling as taking the Red Hat skills assessment test. Do you need to know the nuts&bolts to use linux? Nope. Does it help? Yes…yes it does. Might even keep you gainfully employed.

wdhpr

Way to much work thats why we have masters. Learning is not about what you know its knowing where to look for what you need to know.

So, to summarize this preachers article in one sentence, go to where the money is and become a novell&suse technology dwarf in their company world to become a “master” linux user? What a limited, boring view of “mastery”. Did Linus do this? Did RMS, Paul Graham, Mark Shuttleworth or any of the 80% other leading linux gurus ever do this? NO! They are not like dedoimedo apparently believing in their small consultant based master and servant worlds. Instead they choose a very simple approach. They created their own world, their own eco-system of followers, friends and believers.
The subtle message of this slave text is: But we possibly cant dream of that. We cannot be one of them, can we? We are only the small creatures, the small bloggers endlessly repeating the boring dedo-Dogma of you are only good if you are able to earn money from “big” companies with some ridiculous piles of qualification papers.
Again, NO! You dont have to be a dedo-dwarf. You can choose your way of social environment yourself, agree with a social contract you respect.
If you want to absorb the right linux knowledge, enabling you to be able to shape the world around you, just google for people who did this already. Look for people like Joey Hess, look for all the “normal” guys integrating with Y-Incubator, look for people who are forging their social environments instead of earning money the frustrating dedo-the-shiny and hollow guru way.

pmu

Excellent Post. Eye Opener. Please keep writing such posts. This is what I call the “True Hall of Fame” Write up.

Matthias Schuster Scharmer

Thanks a lot for this inspiring essay, i integrate it into our forum :D