Many times something seems simple on the surface, but once you start you find it is a pain in the neck. Then, once in awhile, you find it is beyond a pain in the neck. For example, spawning a process and capturing stdout/stderr in .NET. Sure, it doesn't seem hard, create an instance of the Process class, fill in some members, kick it off and wait until it finishes. Works great! On to the next problem...

Until the day it locks up on you unexpectedly. Hmm.

The next step is then to ask the modern Oracle - Google, to see if others have this problem. Sure enough, you find out that if the child process' buffer becomes full, it blocks, and that is causing your deadlock. Even better, there are tons of helpful people on the Internet willing to share the solution to this problem, along with source code snippets. Score! Asynchronous reads for the win. Hook it up, and on to the next problem...

Until the day it locks up on you unexpectedly. Hmm.

I got to this point, and Google began to fail me. It seems either no one ran into this problem, or, the only solution people talk about is the method I'm already using. We write all our crunching tools using C/C++ as command line tools, and then a GUI in C# that spawns these tools. Most of the time, things ran fine for me - but, under heavy stress conditions, such as the child writing a lot of output, the whole thing would lock up. I even had a reproducible case, where it would lock up 100% of the time for a given command line tool with a certain set of arguments. We debugged our hearts out, trying every which way we could to keep things moving on, but we kept coming up short.

I can't say with 100% confidence, only 99%, that the issue was not mine, but Microsoft's. If you try to spawn processes that [may] write to both stdout and stderr (we'll ignore reading from stdin for now), you may find yourself in a world of hurt if one of those buffers fills up. This was a few months ago, and I'm having a hard time remembering the exact details, but it may have been related to the asynchronous call backs for the redirected stdout/stderr being called on the main thread, and, in my case, I was using threadpools for tasks that spawned the processes. My main thread of my application would wait until it got some sort of event from a worker thread that it was done, or there was an error. But, with Microsoft trying to invoke the callback on the main thread for the IO redirection, it never would call to process the buffer, which left the child process waiting for the buffer to clear, and my application's main thread waiting on the worker thread to finish...deadlock.

All I wanted to do was spawn a process and capture all of its stdout and all of its stderr. I did not have to interact with it via stdin at all. You would think this is a simple request. If I only wanted one or the other, I could have just used p.StandardOutput.ReadToEnd(); and just executed the process synchronously. You do not have that luxury if you want both stdout and stderr - you must use asynchronous reads, with the callbacks coming only on the main thread. Throw my need for threadpools into the mix and you find things don't work out so well.

All is not lost, I found a solution. The solution is to not use System.Diagnostics.Process and use raw Win32 API to get the same results. I could of just wrote a C/C++ DLL, or a C++/CLI assembly, but I hate having a mess of DLLs laying around. I wanted a solution that stayed in C#, and so P/Invoke to the rescue. First I got it all working using straight C/C++ and then moved it all to C# using P/Invoke calls to the Win32 API. I'll save you the work, you can find it here: