Fast delete of many files in LINUX and WINDOWS

One of my customers ( Oracle Database 11gR2 on LINUX ) never noticed the Oracle software LUN filling up until a “df -h” reported 97% used, wondered why, and of course wanted it to be solved asap. Well, after some investigation I found the adump directory to be the problem. At this site, database auditing is active, and although a proper cleanup and archiving of the audit log files is in place, this process stalls from time to time for reasons unknown. In this case the result was an adump directory with just under 700.000 files!

Removing the audit files proved to be a problem with LINUX command “rm”, because of the Error: “Argument list too long”. This can be solved with a combination of find and exec, for instance, with adump as current directory, “[ find ./ -type f -exec rm {} \;” or “[find ./ -type f –delete ]”. But I wasn’t very impressed by the performance, and looked for better and faster… and this is what I came to use:

As you can see, this “rsync” statement is really fast, it takes a mere 20 seconds to remove all files, and can be configured to delete files with certain extensions, leaving any other file in the directory. The trick is that you synchronize an empty directory ( /tmp/empty ) with another directory ( /u01/…./adump ), and telling rsync with “–delete” to remove certain files from adump ( see the include list ), not present in the empty directory. Result is the delete of all files, configured in the include list. Rather surprising is the fact that you can do the same trick with robocopy under Windows, as demonstrated:

Use the provided code at your own discretion, and be careful. Rsync and robocopy are both very powerful commands, and should be tested properly before applied in production environments. For both rsync and robocopy this delete|purge trick is a nice but not well-known feature, because it is more of an option in synchronizing or backing up directories or even complete file systems. But there’s no harm in picking up such an option, and putting it to good use where it’s clearly very useful!

1 Comment

That’s an awesome technique Harry. I am about to revisit a script that deal with deleting or moving large number of files on WIndows (which is a notoriously slow filesystem), xcopy and rmdir are too slow, so your suggestions will be put to good use. Thanks.

meta

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 236 other subscribers

Email Address

About

AMIS is internationally recognized for its deep technological insight in Oracle technology. This knowledge is reflected in the presentations we deliver at international conferences such as Oracle OpenWorld, Hotsos and many user conferences around the world. Our AMIS Technology Blog, the most referred Oracle technology knowledge base outside the oracle.com domain. However you arrived here, we appreciate your interest in AMIS. Link to our Google+ Profile AMIS