Posts Tagged ‘sysadmin’

If you’re a sysadmin which needs to deal with Microsoft Windows servers locally or remotely via Remote Desktop RDP client (MSTSC.EXE) or inside a Windows Domain Controller, you will have to frequently debug Windows isseus or Application caused errors by reviewing debug information stored in Event Logs.

Event Viewer is a precious tool to debug often errors with missing libraries or failing programs on Windows boot and thus on M$ Windows it is the Swiss Army knife of sysadmin.
However as staring Event Viewer using the GUI menus, takes a lot of step and looses you time, e.g., you have to navigate to menus:

If you used to be a long years Linux / UNIX sysadmin and you suddenly have to also admistrate a bunch of Windows hosts via RDP (Remote Desktop Protocol) / Teamviewer etc. and you need to document The Load Avarage of a Windows XP / 7 / 8 servers but you're puzzled how to get an overall load avarage of Windows host via command in a UNIX way like with the good olduptime Linux / BSD command e.g.

Then its time to you to get used to WMIC – WMIC extends WMI for operation from several command-line interfaces and through batch scripts. wmic is a wonderful command for Command addicted Linux guys and gives a lot of opportunities to query and conduct various sysadmin tasks from Windows command prompt.

There are plenty of security schemes and strategies you can implement if you're a Shared Web Hosting company sysadmin however probably the most vital one is to install on Apache + PHP Webserver SuPHP module.

# apt-cache show suphp-common|grep -i descrip -A 4

Description: Common files for mod suphp Suphp consists of an Apache module (mod_suphp for either Apache 1.3.x or Apache 2.x) and a setuid root binary (suphp) that is called by the Apache module to change the uid of the process executing the PHP interpreter to the owner of the php script.

So what SuPHP actuall does is to run separateCPanel / Kloxo etc. Users with separate username and groupid permissions coinciding with the user present in /etc/passwd , /etc/shadow files existing users, thus in case if someone hacks some of the many customer sites he would be able to only write files and directories under the user with which the security breach occured.

On servers where SuPHP is not installed, all systemusers are using the same UserID / GuID to run PHP executable scripts under separate domains Virtualhost which are coinciding with Apache (on Debian / Ubuntuuid, gid – www-data) or on (CentOS / RHEL / Fedora etc. – user apache) so once one site is defaced exploited by a worm all or most server websites might end up infected with a Web Virus / Worm which will be trying to exploit even more sites of a type running silently in the background. This is very common scenarios as currently there are donezs of PHP / CSS / Javasripts / XSS vulnerability exploited on VPS and Shared hosting servers due to failure of a customer to update his own CMS scripts / Website (Joomla, Wordpress, Drupal etc.) and the lack of resource to regularly monitor all customer activities / websites.

Therefore installing SuPHP Apache module is essential one to install on new serverslarge hosting providers as it saves the admin a lot of headache from spreading malware across all hosted servers sites ..
Some VPS admins that are security freaks tend to also install SuPHP module together with many chrooted Apache / LiteSpeed / Nginx webservers each of which running in a separate Jailed environment.

Of course using SuPHP besides giving a improved security layer to the webserver has its downsides such as increased load for the server and making Apache PHP scripts being interpretted a little bit slower than with plain Apache + PHP but performance difference while running a site on top of SuPHP is often not so drastic so you can live it up ..

Installing SuPHP on a Debian / Ubuntu servers is a piece of cake, just run the as root superuser, usual:

# apt-get install libapache2-mod-suphp

Once installed only thing to make is to turn off default installed Apache PHP module (without SuPHP compiled support and restart Apache webserver):

# a2dismod php5 …

# /etc/init.d/apache2 restart
…

To test the SuPHP is properly working on the Apache Webserver go into some of many hosted server websites DocumentRoot

You might be stunned to find out Mac OS X has a server variant called Mac OS X server. For the usual admin having to administer a Mac OS X based server is something rarely to do, however it might happen some day, and besides that nowadays Mac OS X has about 10% percentage share of PC desktop and laptops used on the Internet (data collected from w3cschools log files). Thus cause it is among popular OSes, it very possible sooner or later as a sysadmin you will have to troubleshoot issues on at least Mac OS X notebook. Mac has plenty of instruments to debug OS issues as it is UNIX (BSD) based.

Mac OS X has already a GUI tool called Activity Monitor (existing in Mac OS 10.3 onwards) in earlier verions, there was tool called Process Viewer and CPU Monitor.

To start Activity Monitor open Finder and launch it via:

Applications -> Utilities -> Activity Monitor

As a Linux guy, I like to use command line and there Mac OS X is equipped with a good arsenal of tools to check CPU load and Memory. Mac OS X comes with sar – (system activity reporter), top (process monitor) and vm_stat (virtual memory statistics) command – these ones are equivalent of Linux's sar (from sysstats package), top and Linux vmstat (report virtual memory statistics).

That gathers 10 sets of metrics at 10 second intervals. You can then extract useful information from the output file (even while it's still running), this will get you cpu load on Mac OS system spitting stats every 10 seconds.

If you're coming from a small or middle-sized company to a corporations like HP or IBM probably you will not a clear defined idea on the 2 types (2 dimensions) of system Scaling (Horizontal and Vertical scaling). I know from my pesronal experience that in small companies – all needed is to guarantee a model for as less probels as possible without too much of defining things and with much less planning. Other thing is being a sysadmin in middle-sized companies, often doesn't give you opportunity to discuss issues to solve with other admins but you have to deal as "one man (machine) for all" and thus often to solve office server and services tasks you do some custom solution.
hence for novice system administrators probably it will be probably unclear what is the difference between Horizontal and Vertical Scaling?

Vertical Scaling(scale vertically or scale up) :- adding more resources(CPU/RAM/DISK) to your server (database or application server is still remains one).Vertical Scaling is much more used in small and middle-sized companies and in applications and products of middle-range. Very common example for Virtual Scaling nowdays is to buy an expensive hardware and use it as a Virtual Machine hypervisor (VMWare ESX). Where a database is involved using Vertical Scaling without use of multiple virtual machines might be not the best solution, as even though hardware might suffice (creation of database locks might impose problems). Reasons to scale vertically include increasing IOPS (Input / Ouput Operations), increasing CPU/RAM capacity, and increasing disk capacity.
Because Vertical Scaling usually means upgrade of server hardware – whenever an improved performance is targeted, even though if Virtualization is used, the risk for downtimes with it is much higher than whenever horizontal scaling.

Horizontal Scaling (scale horizontally or scale out):- adding more processing units (phyiscal machine) to your server (infrastructure be it application web/server or database).Horizontal scaling, means increasing the number of nodes in the cluster, reduces the responsibilities of each member node by spreading the keyspace wider and providing additional end-points for client connections. The capacity of each individual node does not change, but its load is decreased (because load is distributed between separate server nodes). Reasons to scale horizontally include increasing I/O concurrency, reducing the load on existing nodes, and increasing disk capacity.Horizontal Scaling has been historically much more used for high level of computing and for application and services. The Internet and particular web services gave a boom of Horizontal Scaling use, most companies nowadays that provide well known web services like Google (Gmail, Youtube), Yahoo, Facebook, Ebay, Amazon etc. are using heavily horizontal scaling. Horizontal Scaling is a must use technology – whenever a high availability of (server) services are required.

A classic sysadmin scenario is to configure new Apache webserver with requirement to have an SSL ceriticate installed and working on port 443 and all requests coming on port 80 to be redirected to https://.
On Apache this is done with simple mod_rewrite rule:

Before applying the rule don't forget to have Apache mod_rewrite enabled usually it is not enabled on default most Linux distributions by default.
On shared hostings if you don't have access to directly modify Apache configuration but have .htaccess enabled you can add above rules also to .htaccess

On Debian based GNU / Linux distros, there are some critical packages which need to be disabled to update during the common routine apt-get update && apt-get upgrade which is a almost daily part of Debian sysadmin living. Example for packages which are good to mark not to upgrade are for example; linux kernel, java virtual machine, adobe flash plugin, etc.

Setting a package to omit upgrade on system package update for adobe flash plugin for example is with:

For people interested into statistics of how Linux existing users are spending, there log in times and what kind of commands each of users is executing, take a look at acct
acct is existing on all mainstream Linux distributions is a great sysadmin tool. acct is a great tool whether you have a system where a multitude of users you don't trust has to be monitored. It is an absolutely must have for anyone willing to run, lets say experimental honeypot or free shell host. acct is useful for paranoid sysadmins who like to always knows what there users are running as well as in situation where some of users is suspected to be a potential cracker trying to root the host.

Below is description of acct package on Debian:

# apt-cache show acct| grep -i description -A 8
Description: The GNU Accounting utilities for process and login accounting
GNU Accounting Utilities is a set of utilities which reports and summarizes
data about user connect times and process execution statistics.
.
"Login accounting" provides summaries of system resource usage based on connect
time, and "process accounting" provides summaries based on the commands
executed on the system.
.
The 'last' command is provided by the sysvinit package and not included here.

# If you want to keep acct installed, but not started automatically, set this
# variable to 0. Because /etc/cron.daily/acct calls the initscript daily, it is
# not sufficient to stop acct once after booting if your machine remains up.
ACCT_ENABLE="1"

A lot of the initial commands shown to run on pts/1 is not actual commands, by the user but are just stuff run on user login time via /etc/bash.bashrc, /etc/profile, ~/.bashrc. ~/.bash_profile.

lastcomm displayed output from 2nd column is a special flag giving more information on how and for what purpose command was executed. In above output
F – indicates the command run after a fork. X – is returned if a command exit with SIGTERM (kill signal)D – in case of generated command core dump (D is good one to look for whether checking a suspicious user profile, as it is so common exploits use core dumping to get root superuser access)S – means the command is run with superuser privileges (this one you will see usually whether inspecting user profile of a cracker who run exploit using core dump – a lot of Ds followed by some shell code to run as superuser)

2. Get statistics on CPU use time of services (daemons) and user accounts

psacct is very handy, whether you have CPU server overloads and you have difficulty finding out what are the "CPU hungry processes". To get those use summarized accounting information tool;

Another good use of lastcomm command is to grep over all users executed command for precise commands of interest. One very good use case is if you catch a system abuser running certain exploit or DoS tool on the host and you want to make sure no-one else on the system doesn't try running it.

There is one tool called ac, which is similar in what it does to last command, just like last it uses /var/log/wtmp binary log file to get its user login times stats . The difference is ac provides more and better structured user login time length info.

Its very useful if you want to have idea, which user spends most time connected to host.

$ ac -p
sic 4.86
hipo 4.80
root 25.80
play 0.02

To get general info on how much overall hours all existing users spend doing stuff on node;