We have found that in rare cases an attempt to create a file (CreateFile Win32 API call) fails (returns an invalid handle) and GetLastError returns error 0x32 (decimal 50, ERROR_NOT_SUPPORTED). If you attempt to call CreateFile with exactly the same arguments, the next API call will succeed.

We have written a short test program that reproduces the problem. You first install Windows into a guest VM, install the guest tools, and then run the test program. The test program will enter an infinite loop where it attempts to create/delete the same file 2000 times and then pause for 5 seconds. The test program source/binary is attached. You run the test program like:

C:\TestVBoxFS.exe "\VBOXSVR\data\temp2.chk" > c:\test.log.txt

Also, we have found that if nothing accesses the \vboxsvr filesystem for about 10-15 minutes, and then we run our test application, the very first CreateFile call fails, but the remaining call fails. We have found this is the best way to reproduce this problem. However, if the filesystem is being accessed 24/7 we still see the CreateFile failure in about 1 out of every million CreateFile calls.

This could be potentially the cause of other tickets that mention random problems with shared folders (e.g., #7160 or #4494) [or it might not be related].

We've tried this on both Windows hosts and Solaris hosts and the error seen by the guest is the same. The ticket mentions 4.1.2 I believe we have reproduced this on 4.1.4 as well.

In general we've found the shared folder mechanism to be both reliable and fast for basic file I/O (e.g., 500 MB/sec), but the random CreateFile failures is impacting the reliability of applications using the shared folder filesystem.

We were able to reproduce this 100% of the time on several different host systems (both Windows and Solaris), the guest in every case was Windows Server 2008 R2 SP1 x64. You had to let the guest system idle for about 10 to 20 minutes without touching the host share (e.g., \vboxsvr\myshare) and then at the command prompt run:

C:
C:\TestVBoxFS.exe \vboxsvr\myshare\myfile.txt > test.log.txt

We also had another company's tech team reproduce this issue in-house -- we originally thought it was a problem with their software, but it turns out the problem was reproducible with the test program we created.

Additionally, 'helloworld' (another party) was able to reproduce the issue reliably enough to debug the issue and provide a fix. We have tested the fix on all of the systems where we saw the issue before, and the patch does fully resolve the problem.

Please consider including the patched changes into mainline trunk. Thanks.

Updated issue solution.
There are some changes to file 'netroot.c' (it fix some artifacts from issue 7160 now).
It placing at src\VBox\Additions\WINNT\SharedFolders\redirector\sys\netroot.c in VBox sources.

Debugging showed next behavior: after long standby system 'forget' file share and when we calling CreateFile it try to reconnect. First connection received as pipe, then - as file share. On first connection VBoxSF.sys returned 'ERROR_NOT_SUPPORTED' and this code sending to calling context. Next connection system make as network FS - and all return successing.
When we manually change request type to NET_ROOT_WILD (network FS) for rdbss.sys when receiving NET_ROOT_PIPE - calling context understands that response.

Also we must return STATUS_NOT_SUPPORTED when system try to access VBoxSF device as mailslot. If we do not make this there are some problems with access to shared folders when it connected as network drive

helloworld, thanks for your findings. When the net root type is set to NET_ROOT_WILD then the VBoxMRxCreateVNetRoot function returns STATUS_BAD_NETWORK_NAME, which seems to be a "good" status code. Simply returning STATUS_BAD_NETWORK_NAME for a pipe open also fixed the problem.

This seems to work because the pipe open request is related to DSF and
MSDN says:

The Windows client returns STATUS_MORE_PROCESSING_REQUIRED to the calling application
to indicate that the path does not correspond to a DFS Namespace or a SYSVOL/NETLOGON
share, and that the I/O operation is not complete.