Archive for the 'Linux' Category

I am playing since recently with FreeIPA, Red Hat's identity management solution built on top of Red Hat's DS389 directory server. One of the main reasons why I decided for FreeIPA (apart from integrated Kerberos for single sign-on and possible integration with Microsoft Active Directory in the future) is, also integrated, NIS server - proxy system that receives requests from NIS clients, gets the data from LDAP server and sends it back to clients. Now, in order to understand why I need to support both LDAP and NIS you need to know few things about the environment I'm in charge of.

I'm working for a software development company. We produce billing software for telecommunication operators - mainly used by mobile telecommunication companies. That means, when you make a call, your call needs to be tracked, recorded and properly billed on the end, all done by our software (called BSCS btw). Sounds simple enough. Multiply that by one hundred million customers making calls and it's not so simple anymore. :) Anyway, our customers use our software on different platforms, most of them use HP-UX, some are on Solaris, some AIX, some are on Linux and we even have some customers on Tru64. In order to provide support to all those customers we need to have all those systems as well. So on the end we end up 100+ servers of all types of UNIX systems. That's not a big problem, it's even interesting, but the problem comes up when those systems are not being upgraded. We have Solaris 2.6 servers and Tru64 4.0D servers, until recently we even had AIX 4.3 and HP-UX 10.30 servers. All of the mentioned systems are 13 years old! Scary!

As you can imagine, those outdated systems do not support many things we take for granted today. Shadow passwords and LDAP authentication are few of those things. And this gets us back to the main topic of this post. FreeIPA (or rather DS389) provides integrated NIS server for unlucky people like myself via SLAPI-NIS plugin. All you have to do in order to use it, is to enable compat and NIS plugin.

# ipa-compat-manage enable
Directory Manager password:

Enabling plugin
This setting will not take effect until you restart Directory Server.

# ipa-nis-manage enable
Directory Manager password:

Enabling plugin
This setting will not take effect until you restart Directory Server.
The rpcbind service may need to be started.

By default only passwd, group and netgroup maps are supported but other maps can easily be added. In our environment we are heavily relaying on automounter maps so I had to find a way to add them into FreeIPA NIS server. Luckily, as everything else with FreeIPA, this is very simple. First let me show you how to add automount entries in FreeIPA, it is surprisingly easy.

When it comes to automounter, FreeIPA has support for different locations. So for example, you can have different maps for your production environment, test environment and DMZ environment. Pretty neat. In my example, I will create a new location for our DMZ environment.

# ipa automountlocation-add dmz
Location: dmz

New location is automatically created with auto.master and auto.direct maps.

# ipa automountmap-find dmz
Map: auto.master

Map: auto.direct
----------------------------
Number of entries returned 2
----------------------------

I would like to add a new map for user home folders.

# ipa automountmap-add dmz auto.home
Map: auto.home

Then we need to add an entry in auto.master map to associate /home mount point with auto.home map.

And voila, when user miljan logs-on he will have his home folder mounted.

Final step would be to have this in NIS as well. For this we need to manually add few entries into LDAP server. In the example below we add support for auto.master map. There are probably few things you would need to change, though. First, the domain name in DN and nis-domain lines - in the example I am using example.com as a domain. Second, nis-base line - value of this attribute needs to be the DN of your automount map.

I was recently confronted with the task of migrating logical volume holding MySQL databases to a separate physical volume. Due to important application running on top of MySQL any downtime was out of the question.

Another, and somewhat more tricky way of doing this is to use LV mirroring. I say more tricky because it requires more work and the last part of the operation is not really intuitive so it may be prone to errors.

Hm, confusing error. :) The reason for this is that Linux LVM implementation requires mirror to use a log volume, and this log volume needs to reside on a PV of its own. In order to go around this we can instruct LVM to place the log in the memory. This is not recommended for situations when mirroring is needed on the long run, since mirror needs to be rebuilt every time the LV is activated. But this is the perfect solution for our needs as our mirror will not last for long.

Last step, the one where we have to be more careful, is removing mirror copy from old PV. We have to be more careful because the PV name in this case is the name of a PV we want to remove from mirroring, which is the opposite from what we had when establishing the mirror. If we use wrong PV name we will end up on the begining, with LV on a wrong PV and without mirroring.

Few years ago I went on a quest to find a perfect shell prompt. I asked the mighty Internets for ideas, but it seemed futile. I tried many things, simple prompts, complex prompts, but nothing could satisfy my requirements (I don't even remember what were my requirements back then.) So I picked best of both worlds and got this little monster.

:) oscar:~#

Happy face! And in case of an error, it looks sad.

:( 2 oscar:~#

Cute, a? It even prints the exit code. Useful and cute at the same time!
And here is definition of the prompt. As you can see it uses simple function to determine return code of executed command and adjust its feelings accordingly.

I just found this old screenshot from one of my previous jobs. It was taken on December 9th 2003, while one of the web hosting servers went woowoo due to badly optimized web site. Load average went sky high to 682! Anyone else had such a high load before or am I the absolute champion? :)

Small note for everyone planing to use Linux LVM snapshots and using XFS at the same time. XFS has UUIDs which are unique identifiers of the filesystem. Two file systems with same UUID can not be mounted on the same server. Now, if we know that a snapshot of a logical volume represents a point-in-time copy of the original logical volume it doesn't take much time to realize that the filesystem on the snapshot is also a copy, thus it will have the same UUID as the filesystem on the original logical volume. So here is what happens when you try to mount the snapshot:

I have just noticed that I missed to mention one very important thing in my previous post.

File /proc/meminfo contains a very useful field named Committed_AS. This field indicates TOTAL value of committed memory. If all applications would require all memory allocated to them your server would need this amount of memory.

If we look in the example from my previous post we would find following values:

So my server has 32GB of RAM, but total amount of memory allocated is 48GB. That is 150%! If all this memory would be required at once server would crash pretty bad (or OOM killer would start butchering my Oracle databases to get some memory back!). :-)

Yesterday I had a request for memory usage report on Oracle servers in my company. As we are using Centreon, Nagios frontend which makes good use of performance data reported by Nagios plugins and makes nice graphs out of it, it was a matter of pasting the images into the mail and sending it. But than interesting question was raised: how come on a server with 32GB of RAM and with 30+ databases running, only 5GB of RAM is reported as used? Strange indeed.

Usually, used memory on Linux is calculated as (Total Memory – (Unused Memory + Buffers + Page Cache)). Why buffers and caches are not counted into memory usage? Simply because it contains data that is not really critical for operating system and applications running. It contains data that can be flushed and removed from the memory at any time.

So in my case that was:

33274944 – (198580 + 20 + 27439580) = 5636764

OK, this matches output from free command. But what about those 20GB of allocated shared memory?

Next few hours I spent in searching and reading Linux documentation on memory management and found few interesting things.

Linux uses principle of memory overcommitment. Basically, what this means is that when application requests memory to be allocated, kernel will always “give: the memory hoping that application will not really use it, or at least not the whole size allocated. Only when application tries to write the data into the memory, kernel will mark the memory as used. This can lead to situation where the size of allocated memory is actually higher than the size of physical memory inside the machine. But as long as there is no demand for allocated memory, system is running without problems.

And this is the core of my dilemma. Shared memory is allocated, but since there is no data in it, it is not counted into used memory.

overcommit_memory â€” Configures the conditions under which a large memory request is accepted or denied. The following three modes are available:

0 â€” The kernel performs heuristic memory over commit handling by estimating the amount of memory available and failing requests that are blatantly invalid. Unfortunately, since memory is allocated using a heuristic rather than a precise algorithm, this setting can sometimes allow available memory on the system to be overloaded. This is the default setting.

1 â€” The kernel performs no memory over commit handling. Under this setting, the potential for memory overload is increased, but so is performance for memory intensive tasks (such as those executed by some scientific software).

2 â€” The kernel fails requests for memory that add up to all of swap plus the percent of physical RAM specified in /proc/sys/vm/overcommit_ratio. This setting is best for those who desire less risk of memory overcommitment.Note This setting is only recommended for systems with swap areas larger than physical memory.

overcommit_ratio â€” Specifies the percentage of physical RAM considered when /proc/sys/vm/overcommit_memory is set to 2. The default value is 50.

As requested by bleketux, I made some modifications to pidgin_status.py script.

Main news is that now it is possible to change Pidgin status message periodically. Script will go to background (it is a real daemon now :P), change the status, and wait for the set time interval until it changes the message again, and then all over again, wait-change-wait-change.

To change status message every 5 minutes with a random line from file /home/miljan/quotes/dusko_radovic.txt:

pidgin_status.py -d -t 5 -f /home/miljan/quote/dusko_radovic.txt

And in Pidgin you would get something like this every five minutes:

To show the song you are listening to as status message:

pidgin_status.py -s “Mukeka di Rato – Kustapassaaessedrmobral”

And in Pidgin it would look like:

You can see all possible options by running script with -h argument for help.

Strace is a tool that should be in a toolbox of every system administrator. Not only that it can help in troubleshooting simple problems (ie. missing libraries in newly created chroot, which ldd mysteriously misses to report) but it also helps in debugging very complex system problems and performance issues.

Recently I experienced a very strange problem with one of the RHEL 3 servers we’ve got. Problem manifested in a very strange way, SSH and su logins hanged, other daemons were also hanging during the startup, only way to reboot or shutdown the server was to physically press the restart/power off button, etc. All this could have been caused by problems on both software and hardware level. First suspicious was bad RAID controller, but after tests this proved to be a mislead. After more tests and brainstorms hardware problems were definitely excluded, so problem has to be on the software side. But what could be the problem?

After few more misleading steps I tried to trace system calls created by su command and found very interesting results.

And this is where the strace output ends and su command hangs. Audit device file is opened (file descriptor 3) and as soon as the first request is dispatched to this device (ioctl system call to file descriptor 3) command freezes. According to this I should just disable audit on the server and the problem will be gone.
As a test, audit daemon was temporarily stopped and I tried to switch to another user and the problem was indeed gone.

When the free space in the filesystem holding the audit logs is less than 20%, the above notify command will error out and auditd will enter suspend mode. This causes all system calls to block.

So this behavior is not a bug but actual feature of the software. :o) From security point of view this is expected behaviour – attacker could fill up filesystem where audit logs are stored before the attack and audit will be disabled, meaning no logs of his activity, so better not to allow ANY activity on the system if audit is not able to write to its logs. But still, this kind of behaviour renders the system completely useless to legitimate users.

The topic of this post is not audit, so I will stop here. Important thing is that strace led us directly to the main source of the problem. Resolution of issues like this would be much more complex and time consuming without this great little tool. :)

I was very bored today. Tired from working on Ratuus (don’t go there, site is under heavy construction :)) I needed something to help me take my mind off everything. And what better way to do it, than playing with Python, Pidgin and D-BUS. :D

To cut the long story short, I needed something that will update my Pidgin status message with the information about the current song I am listening. Till recently I was using Rhythmbox player and there is a perfect little Pidgin plugin called Current Track that worked with this player. Last week I discovered gmusicbrowser and fell in love immediately. It is fast, rich with functionalities but still simple to use. Exactly what I want from audio player. (Hm, I just noticed it is written in PERL. Now when Python is used for everything this comes as a big surprise.)

gmusicbrowser already has a plugin called NowPlaying. It will trigger some command whenever song is changed. I just needed to write this command that will inform Pidgin about the change. So, this seemed like a perfect exercise for slow Saturday. :)

Quick search on Pidgin and D-BUS showed extensive documentation about Pidgin API accessible through D-BUS. There is even a working example of how to change the status message! :)

But that was too simple, so I got another idea. Some time ago, I wrote a small daemon in C that will bind to a specific port and display random bofh-excuses fortune messages when someone would telnet to it. (Seems like I have a lot of spare time. I should really find some hobby!) Something similar to telnet bofh.jeffballard.us 666 (here for more information). So I was thinking about implementing the same for my Pidgin status. Random BOFH excuses in your status message! How cool geeky is that!

The result of all that is short (~60 lines of code) Python script that will set your Pidgin status message to:

a) you current song

pidgin_status.py -m The Real McKenzies – Outta Scotch

b) random line from a file

pidgin_status.py -f /usr/local/share/bofh-example

c) anything you give as the command line argument

pidgin_status.py Some very interesting and funny status message

Only difference between a) and c) is the type of the icon that will be shown. In example a) there will be a small musical note, while in example b) and c) nice arrow pointing to right side will be show.

In the middle of testing I noticed this strange message:

Being from Serbia myself, I find this extremely funny. Although, I didn’t know Serbian hackers are so notorious! :)

I hope someone will find it useful. In any case, I am accepting donations for some long and adventurous vacation. As you can see, I really need it! :D