April 26, 2008

It is sunny Saturday for London and I just stopped struggling with our central machine which has Quest Big Brother (pretty handy tool with less price then HP Open View)and an Oracle Database for our centralized checks running on it .

Before stopping by office, All I wanted was using internet banking securely, but could not ignore the errors on BB Screen for not being entitled as an invalid DBA :)

When I checked the BB Screen everything was in unchecked status. At first I thought Oracle instance was down. I checked the instance and it was alive so I logged on the machine to see whats going on. It is a Windows 2003 Box with enough hardware. Memory was fully utilized and there were 180 NMUPM.EXE process explorer running with 11MB memory usage each. (I use to see windows processes instead of task manager so I can investigate more deeply about a running process. ) Process Explorer was saying that it was an Oracle Enterprise manager related process. This was the first time I saw this process. I searched through metalink and saw that NMUPM.EXE sometimes utilize CPU 99 percent and sometimes spawns too many processes. CPU utilization is caused because of .Net Framework installation NoteID:203224.1(not my case), spawning is caused by “pdh.dll (Performance Data Helper) found in $ORACLE_HOME/bin which contains APIs to collect performance data from Windows Environments. It is spawned after EM usageNote ID:436138.1. There have been other bugs in the past about nmupm performance based on other dlls being registered with pdh.”

The solution was removing this pdh.ddl and restarting the DB Console service.

The steps I tried to solve the problem;

1- Tried to stop DB Console service. (did not stop)

2- Killed the DB Console Service but it did not helped to kill the other NMUPM.EXE processes.

3- Tried to kill NMUPM.exe but got bored of one by one killing limitation of both task manager and process explorer.

4- Tried to restart eh machine from remote desktop. It did not work. Machine was alive and I lost my option to log on the machine via remote desktop. Tried to stop services by remote services administration option of windows but did not worked

5- Tried to do shutdown command for remote machine like below. It said the machine is shutdown in progress :(

shutdown -r -m \\uklondmeyregdu1

5- I dont have option to go server room and reset server via button so I Did some googling for how to see / kill processes running on remote machines. What I found is magical and very very useful command line tools named tasklist and taskkill. I checked the running processes by tasklist like below

tasklist /s \\REMOTESERVER /u DOMAIN\USERNAME

After I saw that processes are still running I ran the best command ever (for me)on windows, TASKKILL, which allows multiple killing with one command by just giving the image name

taskkill /s \\REMOTESERVER /im nmupm.exe /u DOMAIN\USERNAME

It killed all the nmupm.exe processes and restart kicked at the end.

I feel embarrassed about now knowing or searching for this two commands especially the taskkill before. Maybe you know it for ages but it is worth to stay at office and write on this sunny London Saturday.

In conclusion I still hate to run Oracle on windows :) however, it is really good to know how to use Windows effectively.