Posted
by
Soulskill
on Monday February 14, 2011 @09:57PM
from the no-tolerance-for-stupidity-and-epic-beards dept.

snydeq writes "Deep End's Paul Venezia offers a field guide to understanding your resident Unix veteran, laying out the nine traits common to this grizzled, hardcore set. From not using sudo, to wielding regular expressions like weapons, to generally assuming the problem resides with whomever is asking the question, each trait is key to 'spotting these rare, beautiful creatures in the wild,' Venezia writes. 'If some of these traits seem anti-social or difficult to understand from a lay perspective, that's because they are. Where others may see intractable, overly difficult methods, we see enlightenment, borne from years of learning, experience, and overall, logic.'"

vim? svelt? Puhleez. When not using ed(1), Real Unix vets use Bostic's One True vi, not some fagged-up Vegas showplace of an editor like vim.

That's like saying Real Unix vets still use telnet and rsh to remotely administer machines. Sometimes it's nice to be able to move up and down lines without having to leave edit/write mode. vim is used by Real Unix vets who have kept up with the times, just like ssh. Only washed up has-beens don't learn to eventually use better tools.

I have to go with the GP on this one. It's not that vim is inherently bad, it's just that it's *unnecessary*. It includes tons of features that an old-school admin has no use for. Sure, most of us use it anyway because modern Linux systems usually don't include our good friend vi (it's usually just a symlink to vim), but we rarely use the features that separate vim from vi. Moving in and out of edit mode is second nature. Hell, even using the arrow keys to move around is a needlessly inefficient waste of motion, since the arrow keys are usually far from any other useful key on the keyboard. The first thing we do in vim is turn off syntax highlighting and silently curse whoever keeps turning it back on.

The comparison to ssh is inaccurate as well. We use ssh because ssh gives us clear security advantages over telnet and rsh, and allows us to use one tool where we previously needed two. Any Unix admin worth his salt, no matter how long and luxurious his neck beard, will gladly upgrade his tools to improve his own efficiency or increase security. SSH does both of those things, while vim does neither.

The first thing we do in vim is turn off syntax highlighting and silently curse whoever keeps turning it back on

I second the other poster's WTF here. It's one thing not to rely on the advanced features of vim - after all, the Single UNIX Spec only mandates vi, so it's the only thing you can guarantee being available, but it seems crazy to not take advantage of them when they are available. Do you also turn of auto-indenting and multi-level undo so that you can spend more time typing and less time actually being productive?

Any Unix admin worth his salt, no matter how long and luxurious his neck beard, will gladly upgrade his tools to improve his own efficiency.

Which is why, as a seasoned UNIX admin, I remove all vi from my systems (to prevent broken programs that do not use $EDITOR or/etc/alternatives from dropping me into vi) and use emacs exclusively. Or jove on small systems.

C-X-( and M-x yank-rectangle improve my efficiency vastly for the one-off scenarios which it is not worth scripting.

Original article author apparently has never met seasoned admins of the emacs species. Specist!:-)

When not using ed(1), Real Unix vets use Bostic's One True vi, not some fagged-up Vegas showplace of an editor like vim.

Indeed.

A particularly nasty vim change is that:
- In vi "u" undoes the last change - even if it was another "u". Hit "u" repeatedly and the screen flips between with and without the last change, making it blink. Useful if a multi-change command may have done something goofy.
- In vim "u" backs you through a history of changes. Handy sometimes. But to redo you have to "control-r". Forget about blinking the screen to use your eye's motion sensors to spot the trouble. Also watch out for accidentally undoing a bunch of changes just before you close and exit. Oops!

Geez, guys. If you want to add new stuff (like a history to go backward through), don't change the behavior of an existing command. Add another, or at least make it configurable - preferably with the default the old behavior.

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
- Jamie Zawinski

I think the point is that what separates the true Unix gurus from the wannabees are that the gurus have no problems with their regular expressions.

Still, it's sad though. I remember the day when regular expressions were considered basic knowledge, and what distinguished the true Unix gurus was the ability to read and write sendmail configuration files by hand.

We exist. It's really pretty simple, achieve enlightenment, then the regular expressions will get you. It's much harder the other way around.

I once wrote a program that read tables from a database to generate regular expressions to parse really messed up addresses (third world, not easy US style). The regex generated the Perl variables that would populate the database table. I think the largest of the expressions was ~43k, hence why the regex was generated, being a concept that was used in the code, but never appeared, for the intense fear it would incite. 10 pages of code and a few tables to replace an expensive program we bought that didn't work.

As with any type of programming, you have to look at it from the computer's perspective and understand how the computer will feel about the code you're feeding it.

That's very true, if it is ultra-high traffic code I wouldn't use regular expressions for much, although you can improve performance a bit (even in the above example) by pre-compiling the pattern and using a static reference to it. How many times though do you run across terrible abuses of chained/nested substring calls (usually without bounds checking) to accomplish things like the above? This is just one area where regex is a godsend.

The requirement actually comes from a popular URL shortening service (I don't know why I'm resisting identifying it, but I am). The "url" variable is a plain String generated in one of many other parts of the code - these other parts guarantee that the URL is valid by the spec, but as you point out this particular requirement is beyond the spec. And anyway, to gain the advantage of the URL class's built-in validation the string would have to be used to instantiate a URL instance, and a failure to validate

All I know is that when first saw regular expressions, I thought my modem had been disconnected and I was about get that damn "CARRIER LOST" signal again. Then I realized I was using Ethernet. After changing my shorts, I realized how [regex] I was.

If an off-by-one error results in the monthly report generation failing if the last day of the month falls on a saturday, then we'll quietly fix the code and tell the user this was a one-off problem most likely caused by incorrect user input but if it ever happens again to let you know

Take #9: "Our thinking here is there's no reason why a reboot should ever be necessary other than kernel or hardware changes, and a reboot is simply another temporary approach to fixing the problem.". When a run away program fills the disk or sets off the OOM killer then after fixing the problem itself rebooting is the obviously wise thing to do - who knows what random proceess got put in a bad state by the resource exhaustion best reboot and get everything into a known good state.

And of course have fun when the machine does need to be rebooted for a "kernel of hardware change" and some vital service doesn't restart because no one checked that the damn init script was enabled.

Not stupid at all. This guy is into root cause analysis as a process of understanding faults and finding lasting solutions. (See reference to "bandaids".) Covers up your tracks until the next crash. A fully functioning fault-free system working as designed should not require a reboot except for the cases outlined. Unless unix systems aren't as reliable as people like to assume...

I used to routinely work on systems that had three or more years of uptime. We didn't like that, but not because rebooting had any value. We didn't like the fact that they weren't applying their operating system patches at least yearly.

The reason that Unix SAs don't like to reboot is deep seated in the history of Unix running decades ago on hardware for which a reboot cycle meant interrupting potentially dozens of people all sharing the same machine for a sequence that might take 10 to 20 minutes if nothing went wrong. Rebooting was correctly viewed as something to avoid whenever possible.

Windows was not engineered for long uptimes until NT 4.0 and is a johnny-come-lately OS in comparison. Windows didn't run on significant (read: capable of simultaneously supporting more than one user in a non-trivial way) hardware until, what, 1994 or 1995? Meanwhile Unix and its intellectual antecedents had been supporting multi-dozen-user installations for nearly three decades.

When there's only one user, rebooting isn't nearly as big a deal as when there are 20, 30 or more. That dichotomy alone drove the reliability of Unix, and the the lax attitude of Windows.

Personally, even though rebooting my desktop Linux computer with it's fast processor, SSD and RAID disks, takes well under a minute, I still don't like doing it. There's something wrong: It shouldn't need to be done. If I'm rebooting for a non-hardware related issue, it's because I'm being sloppy.

The only admins I've seen that are allergic to rebooting are basically children who are new to a paying job. Most experienced Unix admins I've known have no problem with rebooting, when they feel it's the most appropriate option. They're pragmatic, above all else.

True. I will reboot a host without a second thought if a specific type of issue comes up. Root cause analysis is great, but when you're in production, the host really needs to be running. You can figure out the RCA from the logs later on, if need be.

However, most issues with a box can easily, and much more quickly, be fixed by simply restarting specific processes. I've had plenty of hosts stay up for years at a time with nothing else needed but to occasionally restart some processes. And that is the way it should be. Rebooting isn't some magical maintenance cycle where the oil gets changed and the hamsters are replaced, its just a wipe of the RAM and reload of the OS to an initial state. It doesn't fix hardware errors, bad code or poor planning.

Unfortunately, most of the time, the order comes from someone who doesn't know what they are doing. In that case, as long as it doesn't make things worse, if it takes longer to convince them to not reboot the host than it does to actually reboot it, you might as well go, "Right away, sir." and get it over with.

You must not have ever admin'ed many big iron servers. You know, the ones which take upwards of 30 minutes to reboot and are typically servicing 100+ users or mission critical applications. As the rest of the article says, simply rebooting doesn't fix the underlying issue. All you do is set yourself up for a repeat down the road. You are simply kicking the can down the road for someone else to pick it up and put it in the trash/recycle bin...

Your shell, however, is yours and yours alone for the lifetime of the process. If you don't trust yourself not to type something stupid, you shouldn't be working as an administrator. Period. Sudo was never intended to function as training wheels. To imply its necessity is to claim you, the junior admin, knows better than the senior admin. When you've done your trench time, you too will trust yourself.

You, like most others in this discussion and TFA's author, do not understand what sudo is for.

Of course. There's always "sudo -s", but that requires more keystrokes.
If you're doing something like editing a bunch of configs in/etc typing sudo at the beginning of every single line gets a bit old. (Besides, if you're paying for all that backup media, you might as well use it every now and again).

Really? When your job is entirely about being root, sudo is just getting in the way. I happen to have run systems in a serious environment, and we never used sudo. I would say if you have something to do that ISN'T root, you sir are teh nub.

bullshite. I've been syadmin for 16 years, you NEVER get on as root unless stuff is seriously broken. if you don't know how to properly use ACLs, SELinux contexts, etc - then you're not a sysadmin, you're a hobbyist. No serious, experienced UNIX admin would spend more than 1% of their time as root.

I dare you to list 10 *Administrative* tasks that don't require root privileges.

Seriously, if you're not starting or stopping a service, editing configs that only root can edit, or reading logs that only root can read -why are you even logged as anything but a user -placing you squarely outside the context of this conversation?

ACL's and SELinux contexts are not the point, we're not talking about uploading changes to the corporate website. We're talking *specifically* about sudo (root) vs sudo su - (root).

Really. I consider it a sign of inexperience and an indicator that the admin has never had to clean up after someone else screwed something up as root. That may be the case if you are super meticulous and you've been the only admin everywhere you've been, but no serious environment only has one root level admin and I have yet to meet anyone who was really good and super meticulous all the time.

I'm doing sysadmin, maybe one out of 20 commands I type *have* to be run with root access. If I am doing them all as root then there is a much greater chance of making a mistake and committing that system destroying action or, even worse, doing something subtly bad that nobody knows about until later when it's too late. It also makes me think twice (instead of just once) before executing that command as sudo.

Sudo logs commands that were run, by whom, and when. Even if I didn't care about whether I was root all the time or not, having a log of what was done with that access can be an indispensable tool when doing system troubleshooting. It's also a handy way of telling if someone screwed something up or if j00 wuz pwndz.

To me, running around as root and not using sudo is like using vi to look at a config file you have no intention of editing or similar. It's too easy to slip up and do something wrong once you get "in the groove". Add a page at 4am to that or a situation where you're at the tail end of a 30 hour emergency maint and it's beyond easy to screw things up.

You sir, are full of shit. Sudo IS ROOT. Repeat after me a second time, sudo IS ROOT. It's just annoying root. If users need root to do their mundane work--which is what sudo is for--there's something wrong that needs to be handled by code.

As far as slipping up, uh, what are you going to screw up? I mean, all your configs are version controlled in CFEngine/svn/cvs right? And you have an extensive library of base system images right? You have a bootstrapper right? And your data is on external snapshot

On a server, what could I possible need to do that doesn't require root?

"Man, if you gotta ask, you ain't never gonna know!"

For my file server 159 out of the last 500 commands featured sudo in them. On my database server, the number is 199.

On one of my main application servers, the user account for the service itself isn't even in/etc/sudoers. All of the maintenance and administrative tasks are done without resorting to extra permissions.

Ok, logging into to a database is *not* being a Unix/Linux Admin, it is managing a database.

Listen. The people who log in to make changes to the database, and update the website(sorry, application server), are *USERS*, even if some of them are in sudoers, they are not the Admin (who put them in sudoers).

You just prefix each command with "sudo". It becomes a reflex. You've essentially turned your regular account into a root account. You no longer really have a regular account.

That "#" prompt was invented for a reason: it provides a reminder. While you certainly could be oblivious to such a reminder, that is less a risk than always being one thinko away from doing root actions while feeling all safe and usery.

Sudo everything provides an actual audit trail to the actions taken by an admin. which is essential in environments where governance and acountability are required.

Provided you don't trust it to actually do those things. If someone can run 'sudo su -' then they own the system and can make the sudo log files say whatever they want, including removing the fact that they ran 'sudo su -'. Ditto 'sudo emacs', 'sudo dd', 'sudo mv' or any other command that as root will execute subsidiary commands, write specified data to specified files or any various other routes to a root shell. And in most cases you don't even need to muck about modifying logs: Just 'sudo emacs/etc/something/innocuous' and nothing untoward appears in the sudo log but you can run unlogged commands from within emacs, etc.

sudo can be useful in situations where you have a very limited subset of commands that a user should be able to execute as root and each will run as root without allowing the user to achieve a root shell. The trouble is that most commands that aren't already setuid aren't especially designed that way, so that scenario doesn't happen very often.

I guess you could say it's useful if you want to have some kind of faith-based auditing mechanism where you assume your sudoers aren't malicious and therefore won't modify the logs or work around the logging. But if you trust your sudoers that much then why do you need to audit them? It smells like a mechanism to provide the appearance of auditing so that someone's PHB can be satisfied that auditing exists, even though you can't really trust the audit log to be complete or correct.

Provided you don't trust it to actually do those things. If someone can run 'sudo su -' then they own the system and can make the sudo log files say whatever they want, including removing the fact that they ran 'sudo su -'. Ditto 'sudo emacs', 'sudo dd', 'sudo mv' or any other command that as root will execute subsidiary commands, write specified data to specified files or any various other routes to a root shell. And in most cases you don't even need to muck about modifying logs: Just 'sudo emacs/etc/something/innocuous' and nothing untoward appears in the sudo log but you can run unlogged commands from within emacs, etc.

Yes. This is why you disable all those backdoors and only specify particular commands that can be run in/etc/sudoers, with others either denied, or lighting up your IDS/system monitoring like a Christmas tree.

(This also serves as a handy reminder that properly securing systems is *extremely difficult*.)

This guy lost me with the first thing on the list. Going directly to root is great - if you're a noob in mom's basement. Nobody who has ever run systems in a serious environment mucks around as root as an alternative to something like sudo.

Contrary to the article's stated rationale, there is a real reason not to use sudo: If you run some malicious code as the wheel user, it can modify the wheel user's environment. Then the sudo command runs in the wheel user's environment with e.g. a malicious $PATH and you go from having a compromised user account to having a compromised system.

Incidentally, this is also why you should never run "su" and always run "/bin/su -".

Uh, sudo isn't a tool, its a wrapper to audit trail your ass and limit what you can run. The only reason to have it apply to admins is to watch them. Otherwise it just gets in the way. Its not like it adds something to the experience for anyone. It doesn't keep you from making a mistake, it just keeps you from running commands that someone else has decided that you don't need access to. That's like saying that locking away guns prevents you from shooting yourself in the foot. Its does... unless you are in the Army and your job is to tote the thing around on patrol for days at a time. There are simply some times in life where you have to know how not to shoot yourself in the foot.

The only place I ever had it applied to me was when I worked in the financial services industry, and I understood their position. Even then, the sudoers file was so badly conceived that, had we wanted to, it would have been a simple matter to get a root shell. Its difficult to keep out the very people that you need to keep the system running. I'd argue that it is generally not even worth trying it at all unless a very unforgiving regulatory commission is breathing down your neck.

If you don't know how to use root access in a way that doesn't screw up your box, you don't deserve to have a job as an admin, period. Its not like its easy to bumble around on a command line and screw something up. There's really only one really easy thing to do that will potentially demolish a box and that's rm -rf * in the wrong directory and it is a rare sudoers file indeed that prevents a sysadmin from running 'rm'. You could argue a reboot or shutdown on a box with something like a database might be a problem as well, but after that, you actually need to put some effort in screwing up your host with the other commands. Even format/fdisk requires you to think about how you are going to reformat your disk.

I use sudo a lot... to make sure developers can't screw up boxes and do cute little tests in production. But my rule of thumb as an admin is that sudo is something that is inflicted on someone else.

Of course, any tool, be it "su(1)" or "sudo(8)" can be used, and it can be abused. Everyone I know who ever runs systems in a serious environment uses multiple less-privileged users to segregate subsystems, so that you can only kill one of them at a time. In my own work, I *never* use sudo from the command line. I do use it in scripts, and I mean scripts written in several languages. I use setuid as seldom as I can, with *lots* of armor around the API to prevent trouble.
Indeed, "su -ul <subsystem

This guy lost me with the first thing on the list. Going directly to root is great - if you're a noob in mom's basement. Nobody who has ever run systems in a serious environment mucks around as root as an alternative to something like sudo.

Nonsense. Su is much simpler software and much less likely to have security vulnerabilities.
Sudo has had many. Allowing the 'sudo' binary to be setuid root in a serious environment is considered a major risk.
The 'su' binary is much simpler code, and slapping the setuid bit on it is considered much safer.
Well, on BSD 'su' binary is extremely safe. On a GNU/Linux OS, system's setuid su might be linked against a nightmare called GLIBC, but then sudo would be also.
Sudo has the issue of 'subtle/sneaky ways around' any configured policies. And sneaky ways to gain sudo permissions not assigned by policy.

Assigning full Sudo privileges to ANY user is a serious security risk, since you have reduced the number of passwords that must be guessed to gain full root privileges to one password, because sudo requires a password that is the same as the user's login password. The security of requiring knowledge of the user password and separate root password is considered stronger; when you disable root login and require wheel/admin group membership to 'su'. If you assign full sudo privileges to any user then only that user's password is required to gain full root access, which is a reduction in root authentication security strength.

Also 'idle timeout' for root logins is ineffective when sudo is used. If a third party gained access to any logged in ssh
session they can run sudo commands; 'sudo command timeout' can be defeated by merely staying logged in until the legitimate user logs in somewhere else and runs a sudo command; once any legitimate user types the proper sudo password, ALL terminals/remote login sessions under the same username can use any 'sudo' command without a password reprompt,
due to the way sudo is designed.

Su is used in serious environments all the time for the purpose of system maintenance and is considered preferable to sudo for such purpose --- hardly anyone ever even imagined using sudo for that purpose until 5 or 6 years ago. Sudo is a relative newcomer, not installed by default on most systems, and the purpose it was created for is misunderstood if you suggest actual admins use it to perform commands. Sudo is for enabling non-admins to perform some tasks that require UID 0 privileges, under rules established by the system admin; that is the reason the Sudo tool was created, its purpose for existing, and it has nothing to do with the root/sysadmin performing their own duties which actually require root.

That is 'sudo is for partial roots / guest root users, "guests" who are to temporarily have root access but not be one of the persons entrusted with the root passwords.

If you don't login to the system to perform administrative tasks, it is better to simply login as a normal user and then 'su' when you need it.
That way you have to know two different passwords to do things as root; which is strong security.

My environment has some critical systems with minimal installs, where sudo isn't even installed and won't be;
root filesystem being read-only and requiring signed binaries and all. Sudo not even available for some older OS flavors.

It's common to have some paths non-root isn't even allowed to CD into; and this is an improvement for security,
but of course sudo is useless here: Hint: there is no such thing as 'sudo cd'; if you think otherwise, you need to lookup what the cd command does again.

The fact of the matter, is, when you are performing system administrative tasks, typing 'sudo' after each command is too cumbersome. Convenience, and speed matter, as they have a direct impact on admin performance and efficiency.
Sudo introduces inconveniences that are likely to result in serious system-e

Su to root, solve the problem, get out. I don't see what isn't methodical about that?

By using sudo, you get to skip the last step.

Only if your solution to the problem only involves one command. Which, in my experience it seldom does.Sudo is a crutch for home-users who don't trust themselves to remember if they're in a root shell or not. "Real Unix Admins" don't have that problem and don't need sudo.

Reads the story.
Opens the reply form.
Searches our databases for related information.
Creates a whimsical yet topical reply.
Inserts it into the comment box
And then counts down the seconds due to the damned 15 seconds required between opening the reply form and submitting it. Mine usually gets it in with 14 seconds to spare.:)