ParallelProcessingAllowed in CallbackFilter

Any plans to include ParallelProcessingAllowed feature in CallbackFilter?

We are trying to build a service that upon detection of an even on a particular root folder, it launches a separate process build on top of CallbackFileSystem to mount a virtual drive under this root folder.
Trying to do this causes a deadlock situation, because the mounting operation itself causes events to be sent to CallbackFilter application, and this events are serialized with the first event that initiated the need to launch this mounter application.
If ParallelProcessingAllowed were exposed to CallbackFilter we could simply ignore these events that we know are coming from the mounter application.

Can't you just postpone the mounting procedure to after the callback is complete? This is done easily by eg. posting a Windows message to the main thread (or some worker thread) from the callback AND using a mutex both in the callback and the message handler code to ensure that while the callback handler is executed, the worker thread doesn't start the work.

ParallelProcessingAllowed doesn't have a direct equivalent in CallbackFilter and it would be hard (if at all possible) to do something like that in CallbackFilter.

Eugene, that would be an option that will make this work, but it's not an option for us given what we want to do to.

We are trying to present a virtual drive to Sterling Commerce Connect:Direct (also called NDM) product in an on-demand fashion, that why we make our filter call "block" until the mounter application confirms mounting has been successful. We perform this confirmation using a global windows event to avoid extra calls trying to check for the existence of the newly mounted junction.

So, we must block, we can't do the mounting asynchronously because NDM will try to enumerate the files when they are not ready yet. Unless somehow we can detect some subsequent even that will succeed only when mounter completes mounting, hopefully this event will get serialized AFTER all of the events generated by Mounter application.

Now, an idea we were going to prototype tonight:

What if we mount the virtual drive into another folder, and create a junction into the folder that NDM uses to access the files?
If an even happens in another location other than the target junction location, are these events posted to all junctions? Or only the target Junction gets the event?

Jose Battig wrote:
What if we mount the virtual drive into another folder, and create a junction into the folder that NDM uses to access the files? If an even happens in another location other than the target junction location, are these events posted to all junctions? Or only the target Junction gets the event?

If you set filter to junction point, you will get OnOpenFileC callbacks only for the junction point path. Junction point target will raise an events only if you set a filter to the junction point target path.

Quote

Jose Battig wrote:
Trying to do this causes a deadlock situation, because the mounting operation itself causes events to be sent to CallbackFilter ...

We were thinking that by having the ability to process *any* callback to a given object in parallel, we could make the first access attempt block until the drive is mounted, and by identifying the process that issued the subsequent coming callbacks (from the mounter operation) we can safely ignore them knowing they are because of the mounting operation.

Then, when mounting is complete, we resume processing on the initial callback releasing the thread from the calling process (NDM application) that initiated the first callback.

Now, there's a question even of how threads operate callbackfilter. Do threads dequeue from a central events queue? Or there's a scheduler assigning events to be processed by a given thread?

I ask because if the second is true, we can even end up in deadlock situation bu using the Junction strategy, because the junction events may get assigned to be pre-processed by our blocked thread... unless filtering happens on a separate "system" thread that can't be blocked.

The developer working on this is still prototyping, but she could not reach final conclusion. She was even having issue getting the drive mounted on the junction. For some reason the mounter application seemed to be waiting for something to happen, and only completed mounting operation when callbackfilter application closed. She was using CreateProcess() API but I think she even tried with shellExecute with same results.

After a lot of testing, the developer found that apparently something that gets initialized when using CallbackFilter, makes this application that uses CallbackFileSystem block, until the app that uses CallbackFilter is shutdown.

That could be some kind of recursion in callbacks. If you provide a modified sample code to reproduce the problem, I will create a new ticket to check the issue in helpdesk.

Quote

Jose Battig wrote:
Now, there's a question even of how threads operate callbackfilter. Do threads dequeue from a central events queue? Or there's a scheduler assigning events to be processed by a given thread?

There is a queue in kernel mode, processed in user mode by number of callback threads running in circle.

Vladimir, the question is if threads pull from queue or kernel mode "processor" pushes using a ring approach to threads.

If push with ring approach is used, then there chance of deadlock if number of threads is low and if operation on event produces new event(s) that eventually may be pushed into the blocking thread. Pull approach would not cause this deadlock, only idle threads will have the ability to pull from queue.

On the other side, the latest test that our developer did, she didn't even start filtering code, just linked most units from CallbackFilter and that caused CallbackFileSystem application to hang.

Some time today she will try to narrow down the issue and I'll post here the findings (hopefully with some sample code).

Vladimir, the question is if threads pull from queue or kernel mode "processor" pushes using a ring approach to threads.

User mode threads pull requests from kernel queue, using a custom form of request with DeviceIoControl() win32 api.

Quote

Jose Battig wrote:
...and if operation on event produces new event(s) that eventually may be pushed into the blocking thread...

One of the possible deadlock may be reproduced easy. In the case when user callback handler performs some actions that produce another callback (explicitly or implicitly ) - that will result in a 100% deadlock.

Final solution we implemented uses junctions. So we will monitor activity on a given folder, upon activity we mount on another folder that happens to be junctioned to the monitored folder, but the mounting operation won't trigger an even on the monitored folder.
We have a prototype working. So far is looking promising, but we have not tested yet in the real production scenario.