Introduction

There are several articles on CodeProject and on MSDN that deal with the redirection of Console process' input/output using pipes. Unfortunately, you might have realized that most of the time, you will receive the output of the program as a giant block of data only when the child process terminates. This is a big issue because usually, you spawn a console process that will perform a task for you, and during the execution time, you want to get a feedback.

This article provides background information on why this problem happens, and a nice solution, easy to integrate in your existing programs.

Background

So you have a nice (third-party) console program that performs a long task and prints out progression messages? Everything seems to work normally when you run that program from the command prompt... But as soon as you want to encapsulate it in a nice GUI program that will present the results to the user, things get worse, and you don't get those progression messages until the end of the subprocess.

Well, I have a good news and a bad news:

the good news is: it's not the fault of your GUI program.

the bad news is: most console programs behave differently when their output is redirected to pipes!

Why is that?

Let's dig in Microsoft C Runtime (CRT) Library....

The printf function has an immediate effect when the program is using a real console, and seems delayed when the program is redirected to a pipe.. So, let's take a look at its source code... You will quickly find out that there is a buffering system around the stdout stream, and in order to have _ftbuf flush that stream (= output immediately the result of your printf), you have to have _stbuf reach the last return(1);. Unfortunately, when stdout is redirected to a pipe, you will discover that the if (!_isatty(_fileno(stream))) return(0); prevents this from happening.

The Microsoft CRT considers that stdout is not a TTY when it's a pipe, and changes the buffering behaviour !

So, now there are two options:

You can add a fflush(stdout); after each output instruction. This works, but that requires you to have the source code of the console program and to modify/recompile it

You want a generic solution that works with any console executable. Then follow with me a little further...

But before continuing, it's time to state these:

This analysis is valid for programs compiled with Microsoft C Runtime Library only.

It might not be valid with other runtime libraries, but if you see the same symptoms, that means there must be a similar buffering system.

And finally, the vast majority of console programs out there have been developed with the Microsoft C Runtime Library.

A little deeper in Microsoft C Runtime (CRT) Library....

OK, so, what would be nice is that we cheat the CRT into thinking that stdout is still a TTY when it's a pipe.

Looking at the source code of _isatty, we need the FDEV flag on our file, and this only happens if a call to GetFileType returns FILE_TYPE_CHAR.

Doh'! For a pipe, GetFileType returns FILE_TYPE_PIPE. And MSDN tells us that FILE_TYPE_CHAR is only returned for printers and the console...

My solution is to really use a console buffer that the father process will create, share, and monitor while the child process writes into it.

But there are two drawbacks that we have to solve:

Output console buffers can only be written to, not read.

The only way to have a console buffer is to either be a console process or to call AllocConsole. But we don't want a console window to appear!

The first one is solved by using ReadConsoleOutputCharacter and other console-specific functions that allow us to read information from the console buffer as if we were reading it on the screen.

The second one will be solved in a very elegant way: Instead of calling AllocConsole from our GUI program and quickly find the window in order to hide it (like some articles suggest), we will create an intermediate stub program that our GUI program will spawn instead of spawning the target program.

This little stub program will be a real console process that will be in charge of monitoring the console buffer and flushing the data read onto its own output stream (that our GUI program will redirect to a pipe)

This solution brings two nice advantages:

The stub program can be run hidden with the SW_HIDE startup window option, so no console window will be visible.

If you have already written your GUI program using redirection pipes, you can keep it! The only thing you will have to change is to insert "RTconsole.exe" at the beginning of your CreateProcess command-line.

A look at the RTconsole source code

The arguments to RTconsole.exe will be the original command-line, including the path to the target console program.

We build an inheritable console screen buffer and fill it with zeroes:

We could have used the standard console that comes with any console program, but this is cleaner and it avoids mixing the subprocess output and our own output in the same console screen buffer. The zeroes will allow us to differentiate with space character outputs. Please note also that CreateConsoleScreenBuffer is possible only because we are a console application. In a GUI application, this would require calling AllocConsole which would display a console popup window.

Now, we start the target process normally, sharing the same console. (RTconsole.exe itself must be started with SW_HIDE to hide the shared console.)

There seems to be no way to be notified on new characters arriving in the console screen buffer, that's why we now need a monitoring loop:

do {
if (WaitForSingleObject(pi.hProcess, 0) != WAIT_TIMEOUT)
exitNow = true; // exit after this last iteration
// get screen buffer state
...
// loop until end of subprocess
} while (!exitNow);

We exit that loop when the child process has exited, after doing an additional iteration of the loop, for the very last characters output to be taken in account. Monitoring the child process this way also should solve the problem encountered with blocking ReadFile on 16-bit subprocesses.

In the loop, we monitor if the text cursor has moved since the last monitoring, we read the characters on screen from the last known cursor position up to the current position, we fill back with zeroes the portion we have read, and reset the text cursor to its home position.

Then, we analyze the characters read from the screen buffer, and convert them to lines that are written (flushed) to RTconsole's own original output stream.

Weak points

These have been tested to ensure no problem is happening most of the time, but it's always good to know your weak points

Synchronization issues

There is no atomic way of reading the screen buffer and resetting it for the ongoing incoming data. So while we are reading and clearing the screen buffer, there might have been additional characters written by the child process. That's why I'm switching temporarily to THREAD_PRIORITY_TIME_CRITICAL for a quick check to see if the text cursor has moved since then. Characters are not lost in this case because we only clear the characters we have read

Scrolling screen buffer

If the text cursor has not moved, we reset it back to its home position (0,0) to avoid letting it go down the default 25 lines or so of the screen buffer. Otherwise, the screen would start scrolling, and if it happens, then we might lose some text. Note that you could probably use SetConsoleScreenBufferSize to enlarge the screen buffer if the target console program outputs characters too quickly.

Comments and Discussions

First, I want to say thank you for this article. I'm just wondering if the synchronization issue
could be solved by keeping the handle of the main thread of the spawned child process, and then just before you call ReadConsoleOutputCharacter in the main loop, you can call SuspendThread passing the handle of the main thread. The handle of the main thread is pi.hThread. This is the same one that you close in your code:

CloseHandle(pi.hThread); // always close the hThread after a CreateProcess

Instead of closing it, you could just keep it, and just before you figure out the count of characters written, you can just check the status of the thread using GetExitCodeThread(pi.hThread). If the result is STILL_ACTIVE, you call SuspendThread(pi.hThread), and proceed to read the buffer. And after you're done reading the buffer, you just call ResumeThread(pi.hThread).

But, even if the console process spawns other threads, the output screen buffer is the same for all of them. Somehow, I have a feeling that the main thread is the one that does the writing to the buffer in a manner that is very similar to windows message queues. Every window is owned by just one thread, and other threads just post or send messages to the message queue. Windows will guarantee that the messages are serialized...this is just a gut feeling. I will test this idea soon...
Here is another thing, even if there are multiple threads in the spawned process, there are several ways to enumerate them and suspend them one by one. BUT, I think that there is no need to go that far...
Out of curiosity, I would be very surprised if there is any real-world program out there where different threads write to the same buffer at the same time!!!!

I don't think there is such thing like "the console window is owned by the main thread and all writings are serialized through it", especially as console process are usually not pumping windows messages. Moreover, console windows are handled by a separate process (csrss.exe) which quite confirms no thread within the console process is "responsible" for the console window.

I agree that there must not be many program where different threads write to the same output, however there can be multi-thread program where the thread responsible for writing out the computation of other threads is not necessarily the main thread (even is there is a good chance the developer chose the main thread).

Of course, the ultimate solution could be indeed to enumerate all threads and suspend them all temporarily, but this seems overkill, and may have side effects (suspending threads may lead to the program miss some data acquisition or synchronization issues)

Anyway, the source of RTconsole is available for you to modify it in order to suits your need...
So if your original suggestion of suspending the main thread fits the problem you have with a specific program, feel free to apply this modification to your version of RTconsole

This code will work for small enough tasks, but for a larger continuous task I am getting "Unhandled exception at 0x01361230 in RTconsole.exe: 0xC0000005: Access violation reading location 0x00000000."

All I'm trying to do is invoke Robocopy via RTConsole so that the output may be redirected to a GUI output

greetings.. I have been using RTconsole from a long time . all thanks to u.
I am also experiencing the same error.. [more on fast cpu's)..Is there a way to show progression (of long background process) in real time..
thanks again..

What if a console program outputs a data stream (infinite line of strings) continuously? Can this MFC GUI operate simultaneously while displaying this inifinite data comming out from the console program? Thank you

If your question was about running Windows' ftp.exe
I tested and it works very good with RTconsole.

I changed the demo project to run "RTconsole.exe" "ftp.exe" -s:ftp.txt
with ftp.txt a text file containing a sequence like this one:

open ftp.free.fr
anonymous
user@domain.com
bin
cd /pub/linux/kernel/v2.6
get ChangeLog-2.6.10
get ChangeLog-2.6.11
get ChangeLog-2.6.12
get ChangeLog-2.6.13
get ChangeLog-2.6.14
close
quit

and you can see progressively the output of your ftp session.

If you can't use a pregenerated ftp.txt and want to programmaticaly react to ftp.exe output, you will have to code a handler in the loop that read the output of the subprocess and send custom commands back to the input of the ftp subprocess. But this is another story, unrelated to RTconsole.

Thanks for posting this useful piece of code. I have spent 2 days trying to deal with the problem (I needed to monitor a 3rd party app with no source code available) but always crashing against Windows buffering wall. Your code works perfect for my application and I will credit you for the unvaluable help.

Yes, thank you very much for this great like piece of code, allready compiled and ready to use. It simply worked - that's how software should be.

I also spend two days in finding the root cause (and finally coming to this web page). I started with python pp / subprocess, then used a pywin32 based Popen and tons of other things (Unix behaves so much smoother than Windows - that issue showed it again).

This is an interesting article.
If you really need the Console Output in realtime try this code!
But it is a weak code.
It may happen that you lose characters when the console application prints faster than you read the buffer. For high reliability I cannot recommend this.

But if you came here because you are searching an easy way to capture the output of a Console application and it does not matter to wait until the Console application has exited, there is an easier and safer way for you:

The advantages over this project are the following:
1.) You get stdout and stderr separated (optional)
2.) There is no risk of losing characters.
3.) The coding is much easier for you: You don't have to implement a receiving Pipe in your main application.
4.) With two lines of code you call a console application invisibly, wait until it exits and get all its output.
5.) You can pass Environment variables to the console application.
6.) The codepage is converted between ANSI / OEM.
7.) All the funcionality is in a DLL, so it can be used in ANY project independent of the compiler: VB6, C++, .NET, Java, Delphi....

You are wrong.
Many people come to this article by googling.
Maybe they search any code to read the console output - no matter if does that in real time or not.
They come to this article but it is not exactly what they are looking for.
So my link may redirect them to what they need.

Apart from that the console is not made for capturing the output in real time.
The code here is very weak.
It is very probable that you get corrupt data.

Your solution does not really attempt to fix the synchronization issue...
New characters might have been output by the subprocess between your SetConsoleCursorPosition and ReadConsoleOutputCharacter, overwriting the previous characters (unread yet)
And for characters that might have been output between GetConsoleScreenBufferInfo and SetConsoleCursorPosition, your solution might work at the expense of reading all characters up to the end of the screen, and no longer being able to generate empty lines (\r\n) correctly.

Your calculations on "count" appears complex for a result that is equivalent to my single "DWORD count = ..." line.
All I see you add is a latency up to 2/3 of the screen buffer before resetting the cursor position, which is worse for subprogram that output quickly a lot of data (because the screen buffer might scroll)

One positive thing in your solution however:
The GetLargestConsoleWindowSize seems a good idea to reduce the risk of scrolling.