Monday, August 26, 2013

Get Rid of Deleted Open Files

Sharing a nice articles which I found on web. Hope it will be
useful to you as well..

You might have this scenario; Logfiles deleted while the process
is still running. That's annoying: On your Linux-Server the /var filesystem is
nearly full. You remove a very large logfile that you don't need with the rm
command:

rajat@root-centos## df
-Ph /var

Filesystem Size
Used Avail Use% Mounted on

/dev/mapper/root-var 7.1G 7.0G 100M 99%
/var

rajat@root-centos## ls
-l /var/log/myapp/userlog

rajat@root-centos## rm
/var/log/myapp/userlog

rajat@root-centos## df
-Ph /var

Filesystem Size
Used Avail Use% Mounted on

/dev/mapper/root-var 7.1G 7.0G 100M 99%
/var

But what's that? The filesystem is still full. With lsof you can
see, that the logfile is still opened in write mode:

To actually free up the space you would have to stop the logging
process. But since this might be a mission critical application this is not
always an option. Is there any way to get rid of the file without stopping the
logging process?

Find the deleted file's representation under /proc

Actually we cannot remove the file as long as the file is still
in use by a process. But what we can do is: Getting the size down to 0. Thanks
to Linux' enhanced /proc filesystem.

And that's how you do it:

First find the process that still uses the file (we already did
that - see above):

We can do almost everything with this file (called 4 here) what
we can do with a real file: we can less it, copy it, and we can change its
contents!

Free up the Space

As already said, we cannot remove the file, but what we can do
is getting the size down to zero. And that's done as with every other file,
e.g. if you use bash (or ksh) - what is most likely under Linux:

rajat@root-centos##
> /proc/25139/fd/4

rajat@root-centos## df
-Ph /var

Filesystem Size
Used Avail Use% Mounted on

/dev/mapper/root-var 7.1G 4.9G 1.2G 69%
/var

As we see there is again free space under /var, and the process
is still running:

rajat@root-centos## lsof | grep var/log/myapp/userlog

myapp 25139 root 4w
REG 3,12 0
0 /var/log/myapp/userlog (deleted)

What more could be done...?

As said before you can work on this file as you work on real
files. That means, you could even save this file and compress it before getting
its size down to 0:

rajat@root-centos## cp
/proc/25139/fd/4 /tmp/userlog

rajat@root-centos##
gzip -9 /tmp/userlog

rajat@root-centos## mv
/tmp/userlog.gz /var/log/myapp/userlog.1.gz

rajat@root-centos##
> /proc/25139/fd/4

Final Remarks

This procedure helps you if you are in a pinch - but basically
you should never remove such an open file, because you still have an issue
here: The process still writes to this file - and the only way to see what it
logs is to use the procedure to save the file as shown above.

So remember to bring the size down to 0 in the first place
instead:

rajat@root-centos##
> /var/log/myapp/userlog

This way the space in
the filesystem is freed immediately - and you still see what your application
is writing to this file.