I come from web programming background, and find myself interested in one peculiarity of using a local shell. I understand that when a program reads from file, it can read at whatever rate's necessary. But I'm curious to know, how does it work when a program gets other program's input piped in and can't process it in realtime?

A good example would be video encoding. Suppose I point a decoder to video file, then it's output gets piped in as encoder's input. The total size of decoded video is more than ram+swap, so I guess there's no way to have it buffered altogether. I found about read and write calls to stdin and stdout, but I'm interested to know what actually happens when this example's encoder can't cope with all the data at once. Does it somehow informs the decoder about desired rate? Does decoder program needs to be specifically prepared for such a signal and modify it's processing speed accordingly? If not, how does it all gets balanced in the end?

2 Answers
2

When a writer writes to a pipe and the pipe is full (its size is limited to a few kilobytes), its process blocks until one of the readers frees some space. Similarly, when a reader reads from a pipe, its process blocks until there is something there.

There are also asynchronous writes and reads that a programmer can use to queue up these reads and writes.

Pipes are usual unix descriptors at the programmatic level. When you setup two programs to communicate through a pipe, all they see at first is the standard output and standard input descriptors, which they interact with, because the shell did setup them that way. There are some peculiarities with these descriptors when setup this way, but processes interact with them as with any other file descriptor (it's the whole unix philosophy, you know).

Basically, the writer process may write as much as he wants in the pipe, but, because the pipe has a maximum storage limit (a buffer), it may be blocked if the pipe is full, or the request may be answered as "try again later" if the writer didn't want it to block.

Conversely, the reader may read as much as he wants, but it may also be blocked (or answered with "try again later" as well) if the pipe is empty.