non-blocking PIPE read on Windows

I have been looking into non-blocking read (readline) operations on
PIPES on windows XP and there seems to be no way of doing this. Ive
read that you could use a Thread to read from the pipe, but if you
still use readline() wouldnt the Thread block too?

What i need to do is, create a process using subprocess.Popen, where
the subprocess outputs information on one line (but the info
continuesly changes and its always on the same line) and read this
information without blocking, so i can retrieve other data from the
line i read in then put this in a GUI interface.

readline() blocks until the newline character is read, but when i use
read(X) where X is a number of bytes then it doesnt block(expected
functionality) but i dont know how many bytes the line will be and its
not constant so i cant use this too.

Advertisements

placid wrote:
> Hi all,
>
> I have been looking into non-blocking read (readline) operations on
> PIPES on windows XP and there seems to be no way of doing this. Ive
> read that you could use a Thread to read from the pipe, but if you
> still use readline() wouldnt the Thread block too?

Yes it will, but that's ok. In this case that's what it's for. While
the thread waits for the readline(), the rest of your program continues
to carry on.
>
> What i need to do is, create a process using subprocess.Popen, where
> the subprocess outputs information on one line (but the info
> continuesly changes and its always on the same line) and read this
> information without blocking, so i can retrieve other data from the
> line i read in then put this in a GUI interface.
>
>
> readline() blocks until the newline character is read, but when i use
> read(X) where X is a number of bytes then it doesnt block(expected
> functionality) but i dont know how many bytes the line will be and its
> not constant so i cant use this too.
>
> Any ideas of solving this problem?
>
>
> Cheers

Advertisements

On 27 Jul 2006 22:26:25 -0700, "placid" <> declaimed the
following in comp.lang.python:
>
> readline() blocks until the newline character is read, but when i use
> read(X) where X is a number of bytes then it doesnt block(expected
> functionality) but i dont know how many bytes the line will be and its
> not constant so i cant use this too.
>
> Any ideas of solving this problem?
>
Use a thread that reads one character at a time; when it sees
whatever signals "end of line" (it sounds like you're reading a progress
bar implemented via <cr>overwrite). Combine the characters into a
string, return the string to the main program via a queue.

If there is no such "end of line" character, but there IS a
noticeable delay between "writes", a more complex method might suffice
-- in which one thread does the byte reads, setting a time value on each
read; a related thread then does a sleep() loop, checking the "last read
time" against the pause length -- if close enough to the pause duration,
combine and return...

Alternatively, take a good old style terminal keyboard (a VT100
Tempest-rated model should be ideal), and use it to beat Bill Gates over
the head until he agrees to push a high-priority upgrade to the command
line I/O system... or makes files work with select() (so you can combine
the time-out with the byte read)

"placid" <> wrote in message
news:
> Hi all,
>
> I have been looking into non-blocking read (readline) operations on
> PIPES on windows XP and there seems to be no way of doing this. Ive
> read that you could use a Thread to read from the pipe, but if you
> still use readline() wouldnt the Thread block too?

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
> What i need to do is, create a process using subprocess.Popen, where
> the subprocess outputs information on one line (but the info
> continuesly changes and its always on the same line) and read this
> information without blocking, so i can retrieve other data from the
> line i read in then put this in a GUI interface.
>
>
> readline() blocks until the newline character is read, but when i use
> read(X) where X is a number of bytes then it doesnt block(expected
> functionality) but i dont know how many bytes the line will be and its
> not constant so i cant use this too.
>
> Any ideas of solving this problem?
>
>
> Cheers

- the subprocess2.py module realizes the non blocking I/O
- the outputparser.py module processes the output from the controlled
process and updates the progress-bar, the status-bar and the log
messages in the GUI. Incomplete lines are stored in a buffer and
processed at the next read.

Dennis Lee Bieber wrote:
> On 27 Jul 2006 22:26:25 -0700, "placid" <> declaimed the
> following in comp.lang.python:
>
> >
> > readline() blocks until the newline character is read, but when i use
> > read(X) where X is a number of bytes then it doesnt block(expected
> > functionality) but i dont know how many bytes the line will be and its
> > not constant so i cant use this too.
> >
> > Any ideas of solving this problem?
> >
> Use a thread that reads one character at a time; when it sees
> whatever signals "end of line" (it sounds like you're reading a progress
> bar implemented via <cr>overwrite). Combine the characters into a
> string, return the string to the main program via a queue.
>

Yes it is a progress bar implemented via <cr> overwrite. I will try
this.
> If there is no such "end of line" character, but there IS a
> noticeable delay between "writes", a more complex method might suffice
> -- in which one thread does the byte reads, setting a time value on each
> read; a related thread then does a sleep() loop, checking the "last read
> time" against the pause length -- if close enough to the pause duration,
> combine and return...

i dont think there is a noticeable delay between "writes".

> Alternatively, take a good old style terminal keyboard (a VT100
> Tempest-rated model should be ideal), and use it to beat Bill Gates over
> the head until he agrees to push a high-priority upgrade to the command
> line I/O system... or makes files work with select() (so you can combine
> the time-out with the byte read)

Have you ever seen a Tempest VT-100? Lead shielding on the monitor
FACE... Turn the brightness all the way up and it still looked dim.
Fiber optic to the computer. And a shield keyboard about three inches
thick and weighing 5 lbs.

All to keep "them" from using radio equipment to pick up key strokes
or the display scan line radiation (use an external vertical/horizontal
sweep generator to an oscilloscope, and a high-gain amplifier to detect
the brightness variation of the CRT).
--
Wulfraed Dennis Lee Bieber KD6MOGHTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: )HTTP://www.bestiaria.com/

Dennis Lee Bieber wrote:
> On 30 Jul 2006 16:22:34 -0700, "placid" <> declaimed the
> following in comp.lang.python:
>
> >
> > Tsk Tsk
>
> Have you ever seen a Tempest VT-100? Lead shielding on the monitor
> FACE... Turn the brightness all the way up and it still looked dim.
> Fiber optic to the computer. And a shield keyboard about three inches
> thick and weighing 5 lbs.
>
> All to keep "them" from using radio equipment to pick up key strokes
> or the display scan line radiation (use an external vertical/horizontal
> sweep generator to an oscilloscope, and a high-gain amplifier to detect
> the brightness variation of the CRT).

placid wrote:
> What i need to do is, create a process using subprocess.Popen, where
> the subprocess outputs information on one line (but the info
> continuesly changes and its always on the same line) and read this
> information without blocking, so i can retrieve other data from the
> line i read in then put this in a GUI interface.
> readline() blocks until the newline character is read, but when i use
> read(X) where X is a number of bytes then it doesnt block(expected
> functionality) but i dont know how many bytes the line will be and its
> not constant so i cant use this too.

I wrote something for this the other day. The class has a getline()
method, which either returns a line or raises an exception instead of
blocking. It also raises an exception instead of returning EOF.

My use case had me reading from multiple processes at once; since
select() doesn't work on files in win32, I had to get a little cheesy.
I've appended the function that implements that use case, for
reference.

The central idea is to use PeekNamedPipe to figure out what's in the
pipe. You can then read that data without fear of blocking. I wrote it
quickly, therefore the code is a little embarassing, but... hopefully
useful all the same.

I think that specialized MSWindows based services too complicated. They have
to many bug possibilites.
So I trying with normal, "in python accessable" pipes. I see that with
flush(), and some of the bintotext tricks I can use the
subprocess/masterprocess communication.
Ok, this is not asynchronous. I can send some job to sp(s), and I can
receive the report from sp(s).
But with threading I can create non-blocking communication.

You can see in the example: the PipeBPPThr define a pipe based process-pool
thread.
This can communicate with a subprocess, can send/receive jobs, etc.

If you collect these threads, and write a process pool object, you can
handle all of the communications with one object.

Share This Page

Welcome to The Coding Forums!

Welcome to the Coding Forums, the place to chat about anything related to programming and coding languages.

Please join our friendly community by clicking the button below - it only takes a few seconds and is totally free. You'll be able to ask questions about coding or chat with the community and help others.
Sign up now!