2010-08-16

Short war story

One of my clients upgraded their network cabling over the weekend. Today morning, all connection attempts to all Oracle database servers started taking too long to materialize. The systems administrator called me in the afternoon, apparently, after exhausting all possible solutions that he could think of.

I asked him if ssh to those database servers (all of which run Linux) was connecting immediately. He checked, and replied that it was. Initially, I too was confused. I then asked him if the same IP addresses (and not any other virtual addresses) were being used in tnsnames.ora on the application servers.

Then came the answer. He checked, and replied that tnsnames.ora did not have any IP addresses at all. It had only host names! The obvious culprit, then, was DNS. When the DNS server was restarted, the world returned to normality.