This post shares robust wrappers around the standard FileSystemWatcher (FSW) fixing problems commonly encountered when using it to monitor the file system in real-world applications.

Buffering and Recovering FSW

Simply replace the standard FSW with my BufferingFileSystemWatcherand you no longer need to worry about InternalBufferOverflowExceptions. Use my RecoveringFileSystemWatcherto automatically recover from typical transient watch path accessibility problems. Download complete code. For a file system watcher using polling instead of file system events see my FilePoller. To process files detected in either way I recommend using TPL DataFlow ActionBlocks. They allow you to easily process files without having to spawn a Thread or create a Task yourself and allow to configure the degree of parallelism desired. For tips about handling lots of files and using contig.exe to defragment NTFS indexes see NTFS performance and large volumes of files and directories.

Typical FileSystemWatcher Problems

If used properly the standard FileSystemWatcher (FSW) is way better than its reputation. However, there are typical problems one may encounter when first using the FSW:

Unexpected events.

Lost events.

InternalBufferOverflowExceptions.

No option to report files existing before the FSW started.

The standard FileSystemWatcher:

Reports exceptions via its Error event. Not via raising exceptions!

Does not report files that existed before .EnableRaisingEvents =True.

Does detect network disruptions, but does not automatically recover from them.

Does automatically handle renames of its watch path.

Unexpected Event Floods

Some applications trigger lots of file system events for a single action. The FSW simply reports these events. Ex: Excel triggers 15 NTFS events for 4 different files when creating a single new .xlsx file and triggers 8 events for 3 different files of which none is changed event for the file changed one would naively expect:

File system event flood triggered by Excel for single actions like “Save”

If you have control over the file producer you can easily tame this event flood by renaming the files to the watched directory or watched extension only when they are totally complete. For .NET applications a file rename is an atomic operation. Your watcher now only needs to watch for a single renamed event per file. An easy way to create lots of files and file system events is repeatedly copying and pasting all files in a directory via CTRL+A, CTRL-C, CTRL-V.

Lost Events

The FSW only throws exceptions on problems when setting its properties. While watching for changes the FSW reports exceptions via its Error event only and does does not raise them. This is typical for the Event-based Asynchronous Pattern (EAP). To prevent exceptions going unnoticed one must handle the Error event. Strangely this so important FSW Error event does not show up in the Win Forms designer, my wrappers fix this. Maybe not discovering the need to implement an OnError handler and being misled by the FSW throwing exception until started is the reason for the wrongly perceived bad reliability of the FSW. The FSW minimizes usage of precious non-paged memory via its InternalBufferSize property and throws an InternalBufferOverflowException “Too many changes at once in directory …”when exceeded.

Robust File Operations

For file operations one should consider using those from the Microsoft.VisualBasic namespace because they are more robust than those in System.IO and offer more features like automatically overwriting existing files.

When working with files this is typically done in a situation where producers and consumers work concurrently. Here if/then constructs are not robust because changes can happen in between. Thus one should rely on exceptions instead.

With IOException.HResult no longer being protected since .NET 4.5 and C# 6.0 finally supporting exception filters (VB.NET always had this feature) it is now easy to handle typical exceptions like FileNotFound, FileInUse or NetworkNameNoLongerAvailable.

BufferingFileSystemWatcher

My BufferingFileSystemWatcher wraps the standard FSW:

Buffers FSW events in a BlockingCollection. It is better to buffer in a BlockingCollection than consuming precious non-paged memory by increasing InternalBufferSize.

Supports limiting the BlockingCollection.BoundedCapacity via the EventQueueSize property. Must be set before EnableRaisingEvents=True!

Offers reporting existing files via a new event Existed. Existing files are reported before any ones detected by NTFS events.

Offers a new event All reporting all FSW events. Real-world apps typically subscribe to all change types because the FSW change types triggered often do not correspond to the action of the producer. Ex: On saving changes Excel triggers 8 events for 3 different files with no(!) change event for the changed file, see picture above.

Offers the Error event in Win Forms designer.

Wraps FSW via composition not breaking its API. Thus you can simply replace your FileSystemWatcher instances with BufferingFileSystemWatcher and your InternalBufferOverflowException are gone without increasing InternalBufferSize.

The following listing shows key parts of the BufferingFileSystemWatcher in C# 6.0:

RecoveringFileSystemWatcher

My RecoveringFileSystemWatcher wraps the BufferingFileSystemWatcher:

Detects and reports watch path accessibility problems. Using a poll timer monitoring the watch path and the FSW Error event. For Robustness restarting from the Error event is not done directly but also done via the timer!

Automatically recovers from watch path accessibility problems. By restarting the BufferingFileSysteWatcher. New files created during the outage are reported via the Existed event.

Hi Peter,
Thanks for sharing this very useful application.
I have created a filewatcher service which was failing when there were more then 40 files. This has been resolved by using your BufferingFileSystemWatcher API in my application. I have tested with more than 200 files and they all processed! But the processing time increased significantly. Is there any place in you code I may look into to improve the performance?