I have a very heavy I/O application that is constantly receiving data through the network and writing to a specific directory on the HDD. Then, the application need to load the files from that specific place. One of my clients has been experiencing slowness and when I try to access the directory it takes quite long before I can see the content.

My gut feeling is the HDD is degrading due to high I/O for a couple of years and I'm thinking of changing the HDD. Is there a benefit to write to multiple directories instead of using always the same?

3 Answers
3

Not likely. A directory is not a physical location on the disk. New contents for the directory are placed on disk based on their size and the availability of free space on the disk. The actual file for the directory is just a list of those locations.

However, it is very possible to have too many items in a directory. Once you get above a few hundred or thousand items (depending on the OS and type of access), access to the directory can cause a huge slowdown. If your directory has more than a few hundred items consider adding sub-folders to minimize the number in any given folder.

One final thought is that read errors could cause a large slow down if they are not so bad as to cause total failure. If you suspect this, run a diagnostic on the drive and get your data off ASAP if confirmed.

[Edit]
You mentioned above that these are radiology images. I assume with some form of patient ID or serial number/datestamp as the filename. Suggest you create folders for the first 2-3 letters of whatever denotes the ID and move the items into those. You can do this multiple folders deep and it will make finding the items no harder than it is now and greatly speed up directory indexing.

Note that the same number of items will sit happily in a tree structure with a few dozen sub-folders in the first few branches.
–
Chris NavaDec 3 '12 at 16:48

1

A directory is not a physical location on the disk - The data within a directory doesn't have a fixed location, but the blocks used to store the information about what is within the directory doesn't move around.
–
ZoredacheDec 3 '12 at 17:34

The hard drive does not care. What you are probably experiencing is the filesystem letting the large directory become badly fragmented, as Windows is apt to do. You might try creating a new directory, moving all of the files there, and replacing the original directory with the new one.

moving files between locations on the same drive does nothing regarding fragmentation, because the file does not get copied or written. the filesystem just updates the index to show the file being part of the new directory.
–
Frank ThomasDec 3 '12 at 16:40

Degrag would be ok then, it is a dedicated HDD, so I wouldn't mind to launch a full defrag. They are radiology images so there are literally hundred of thousand of small files.
–
code-gijoeDec 3 '12 at 16:42

@FrankThomas, it isn't the files but the directory that is the problem. Starting with a clean, fresh directory eliminates the fragmentation.
–
psusiDec 3 '12 at 16:44

1

keep in mind, if files never get changed (once they are created they are only ever read) and the disk has sufficient free space, there shouldn't be much fragmentation regarding many small files. fragmentation occurs when a file is updated (changed and saved) but there is not enough contiguous space following the previous end of the file to hold the added data and place it in one contiguous block. if can also occure at file creation if the drive is already so populated that there isn't a contigious block large enough for the file, but if they are all small, that is unlikely.
–
Frank ThomasDec 3 '12 at 16:46

1

@Psusi, I think we must be talking about different phenomena when using the word 'fragmentation'. directories do not become fragmented (or more precisely file scattering algorithms cause files stored on disk to have no direct relation to the file system index structure). Free space fragmentation can cause file fragmentation at file create, but thats solved by increasing volume size or doing a free-space optimization.
–
Frank ThomasDec 3 '12 at 16:55

Short answer no, or at least not because of the folder. File locations on disk have little or no relationship the the file system heirarchy, so two files in the same directory may be in consecutive addresses or may be on opposite ends of the disk.

The non-stop IO may be slowly damaging your drive (but HDDs are constantly dying every second of their lives) but spreading out over multiple directories will make no difference.