I have a problem. I run out of disk space all the time. I don't know exactly what files consume all the space. It seems that there are bunch of little files, so it's hard to find with du command. Every time I find something to get rid of to gain some free space (mails, logs, old archives and rpms), disk is full again after some time. Space is stolen from system partition (/). Results of df command:

5 Answers
5

du -hx --max-depth=1 / -- look for what's eating the space. Examine the largest subdirectory (say du -hx --max-depth=1 /var) until you find some space hogs. Logs (in /var/log) are common culprits (which you should deal with using logrotate), as are the cruft that yum likes to keep around (repackaged RPMs). If you can identify any repeat offenders, work out a way to automate whatever maintenance work you're doing to clean up repeatedly.

lsof -n |grep deleted -- look for large files that have been deleted. Identify which process is holding them open, and kill it or tell it to recycle it's handles (if they're log files, kill -HUP <pid> will often do the trick) if you can.

My third line of attack is to resize partitions or allocate new partitions to various mounts, although in your case, since you're not using LVM, that's tricky. Consider taking a maintenance to turn /dev/sdb1 into a VG and allocate from there.

Upgrade. Disk space is so cheap, that if you spend an hour of your time hunting down what's causing space consumption, you just "spent" the time required to pay for a new hard drive, so buy new drives rather than spend too much time hunting down problems.

@Karolis: You can save yourself all that typing by just clicking the little arrow... @David: D'oh, thanks for catching. That's what I get for SFing before my first tea of the evening.
–
wombleNov 18 '09 at 15:39

One addition to womble's answer - if you find you're getting a lot of large log files, consider using logrotate (which is installed by default on most distros) to keep them down to size. If there is one particula log that's taking up most of the space, you should also check it for repeated errors and make sure that the service which produces the log isn't set to too high a verbosity level.

If its not already, move the /var mount point to another partition. File system /var exists to prevent the root FS from filling up.

Same for /tmp, if not already a seperate partition.

You may have a process that deletes files but holds them open. Space will not be reclaimed until closed/process exit. You can use the lsof command to find out what processes are holding what files open. Command lsof may not be in your standard Centos install, you may have to add the package to get it.