Posts Tagged ‘config files’

Generally upgrading both RHEL and CentOS can be done straight with yum tool just we're pretty aware and mostly anyone could do the update, but it is good idea to do some steps in advance to make backup of any old basic files that might help us to debug what is wrong in case if the Operating System fails to boot after the routine Machine OS restart after the upgrade that is usually a good idea to make sure that machine is still bootable after the upgrade.

This procedure can be shortened or maybe extended depending on the needs of the custom case but the general framework should be useful anyways to someone that's why I decided to post this.

Before you go lets prepare a small status script which we'll use to report status of sysctl installed and enabled services as well as the netstat connections state and configured IP addresses and routing on the system.

2. Clear old versionlock marked RPM packages (if there are such)

On servers maintained by multitude of system administrators just like the case is inside a Global Corporations and generally in the corporate world , where people do access the systems via LDAP and more than a single person has superuser privileges. It is a good prevention measure to use yum package management functionality to RPM based Linux distributions called versionlock.versionlock forthose who hear it for a first time is locking the versions of the installed RPM packages so if someone by mistake or on purpose decides to do something like :

[root@redhat: ~ ]# yum install packageversion

Having the versionlock set will prevent the updated package to be installed with a different branch package version.

Also it will prevent a playful unknowing person who just wants to upgrade the system without any deep knowledge to be able to run

[root@redhat: ~ ]# yum upgrade

update and leave the system in unbootable state, that will be only revealed during the next system reboot.

If you haven't used versionlock before and you want to use it you can do it with:

[root@redhat: ~ ]# yum install yum-plugin-versionlock

To add all the packages for compiling C code and all the interdependend packages, you can do something like:

3. Check RPC enabled / disabled

This step is not necessery but it is a good idea to check whether it running on the system, because sometimes after upgrade rpcbind gets automatically started after package upgrade and reboot. If we find it running we'll need to stop and mask the service.

yum install package-name / yum upgrade keeps downloaded packages via its operations inside its cache directory structures in /var/cache/yum/*. Hence it is good idea to check what were the previously installed packages and their count.

This step is mandatory to make sure you're upgrading to latest packages from the right repositories for more concretics check what is inside in confs /etc/yum.repos.d/ , /etc/yum.conf

6. Clean up any old rpm yum cache packages

This step is again mandatory but a good to follow just to have some more clearness on what packages is our upgrade downloading (not to mix up the old upgrades / installs with our newest one). For documentation purposes all deleted packages list if such is to be kept under /root/logs/yumclean-install*.out file

8. Apply the actual uplisted RPM packages to be upgraded

9. Monitor downloaded packages count real time

To make sure yum upgrade is not in some hanging state and just get some general idea in which state of the upgrade is it e.g. Download / Pre-Update / Install / Upgrade/ Post-Update etc. in mean time when yum upgrade is running to monitor, how many packages has the yum upgrade downloaded from remote RPM set repositories:

10. Run status script to get the status again

11. Add back versionlock for all RPM packs

Set all RPM packages installed on the RHEL / CentOS versionlock for all packages.

#==if needed # yum versionlock \*

12. Get whether old software configuration is not messed up during the Package upgrade (Lookup the logs for .rpmsave and .rpmnew)

During the upgrade old RPM configuration is probably changed and yum did automatically save .rpmsave / .rpmnew saves of it thus it is a good idea to grep the prepared logs for any matches of this 2 strings :

If above commands returns output usually it is fine if there is is .rpmnew output but, if you get grep output of .rpmsave it is a good idea to review the files compare with the original files that were .rpmsaved with the substituted config file and atune the differences with the changes manually made for some program functionality.

What are the .rpmsave / .rpmnew files ? This files are coded files that got triggered by the RPM install / upgrade due to prewritten procedures on time of RPM build.

If a file was installed as part of a rpm, it is a config file (i.e. marked with the %config tag), you've edited the file afterwards and you now update the rpm then the new config file (from the newer rpm) will replace your old config file (i.e. become the active file). The latter will be renamed with the .rpmsave suffix.

If a file was installed as part of a rpm, it is a noreplace-config file (i.e. marked with the %config(noreplace) tag), you've edited the file afterwards and you now update the rpm then your old config file will stay in place (i.e. stay active) and the new config file (from the newer rpm) will be copied to disk with the .rpmnew suffix.See e.g. this table for all the details.

In both cases you or some program has edited the config file(s) and that's why you see the .rpmsave / .rpmnew files after the upgrade because rpm will upgrade config files silently and without backup files if the local file is untouched.

After a system upgrade it is a good idea to scan your filesystem for these files and make sure that correct config files are active and maybe merge the new contents from the .rpmnew files into the production files. You can remove the .rpmsave and .rpmnew files when you're done.

If you need to get a list of all .rpmnew .rpmsave files on the server do:

[root@redhat: ~ ]# find / -print | egrep "rpmnew$|rpmsave$

13. Reboot the system

To check whether on next hang up or power outage the system will boot normally after the upgrade, reboot to test it.

you can :

[root@redhat: ~ ]# reboot

either

[root@redhat: ~ ]# shutdown -r now

or if on newer Linux with systemd in ues below systemctl reboot.target.

15. Clean up any versionlocks if earlier set

16. Check services and logs for problems

After the reboot Check closely all running services on system make sure every process / listening ports and services on the system are running fine, just like before the upgrade. If the sytem had firewall, check whether firewall rules are not broken, e.g. some NAT is not missing or anything earlier configured to automatically start via /etc/rc.local or some other custom scripts were run and have done what was expected. Go through all the logs in /var/log that are most essential /var/log/boot.log , /var/log/messages … yum.log etc. that could reveal any issues after the boot. In case if running some application server or mail server check /var/log/mail.log or whenever it is configured to log. If the system runs apache closely check the logs /var/log/httpd/error.log or php_errors.log for any strange errors that occured due to some issues caused by the newer installed packages. Usually most of the cases all this should be flawless but a multiple check over your work is a stake for good results.

Ansible can be used to configure or deploy GNU / Linux tools and services such as Apache / Squid / Nginx / MySQL / PostgreSQL. etc. It is pretty much like Puppet (server / services lifecycle management) tool , except its less-complecated to start with makes it often a choose as a tool for mass deployment (devops) automation.

Ansible is used for multi-node deployments and remote-task execution on group of servers, the big pro of it it does all its stuff over simple SSH on the remote nodes (servers) and does not require extra services or listening daemons like with Puppet. It combined with Docker containerization is used very much for later deploying later on inside Cloud environments such as Amazon AWS / Google Cloud Platform / SAP HANA / OpenStack etc.

0. Instaling ansible on Debian / Ubuntu Linux

Ansible is a python script and because of that depends heavily on python so to make it running, you will need to have a working python installed on local and remote servers.

sshpass needs to be installed only if you plan touse ssh password prompt authentication with ansible.

Ansible is also installable via python-pip tool, if you need to install a specific version of ansible you have to use it instead, the package is available as an installable package on most linux distros.

Ansible has a lot of pros and cons and there are multiple articles already written on people for and against it in favour of Chef or Puppet As I recently started learning Ansible. The most important thing to know about Ansible is though many of the things can be done directly using a simple command line, the tool is planned for remote installing of server services using a specially prepared .yaml format configuration files. The power of Ansible comes of the use of Ansible Playbooks which are yaml scripts that tells ansible how to do its activities step by step on remote server. In this article, I'm giving a quick cheat sheet to start quickly with it.

1. Remote commands execution with Ansible

First thing to do to start with it is to add the desired hostnames ansible will operate with it can be done either globally (if you have a number of remote nodes) to deploy stuff periodically by using /etc/ansible/hosts or use a custom host script for each and every ansible custom scripts developed.

a. Ansible main config files

A common ansible /etc/ansible/hosts definition looks something like that:

There is a lot of things to say about playbooks, just to give the brief they have there own language like a templates, tasks, handlers, a playbook could have one or multiple plays inside (for instance instructions for deployment of one or more services).

The downsides of playbooks are they're so hard to write from scratch and edit, because yaml syntaxing is much more stricter than a normal oldschool sysadmin configuration file. I've stucked with problems with modifying and writting .yaml files and I should say the community in #ansible in irc.freenode.net was very helpful to help me debug the obscure errors.

yamllint (The YAML Linter tool) comes handy at times, when facing yaml syntax errors, to use it install via apt:

# apt-get install –yes yamllint

a) Running ansible in "dry mode" just show what ansible might do but not change anything

The available packages you can use as a template for your purpose are not so much as with Puppet as Ansible is younger and not corporate supported like Puppet, anyhow they are a lot and does cover most basic sysadmin needs for mass deployments, besides there are plenty of other unofficial yaml ansible scripts in various github repos.

I use Midnight Commander console file manager on any UNIX like servers, since my early days as system administrator. mc comes with its own text editor mcedit which is often very handy for reading config files or pieces of source code. Many times I had to modify files which were spitting errors which I couldn't track in VIM, jor or whatever text editor had on server at hand and after checking file with mcedit I caught my config or source code mistake. I guess many other admins has similar nice experiences with mcedit – Internal file editor of GNU Midnight Commander. Nowadays, I install mostly Debian Linux on new configured servers and using mc to navigate in file system is very useful. I prefer mc to open files for edit with F4 (Edit – kbd shortcut) with its default mcedit, however for some reasons most of Debian / Ubuntu and other Linuxes, nowadays has set global environment text editor to nano. I totally dislike this text editor and like changing mc always to use mcedit. This is done straight from MC menus by: