LIBPIPELINE

NAME

libpipeline
- pipeline manipulation library

SYNOPSIS

In pipeline.h

DESCRIPTION

is a C library for setting up and running pipelines of processes, without
needing to involve shell command-line parsing which is often error-prone and
insecure.
This relieves programmers of the need to laboriously construct pipelines
using lower-level primitives such as
fork
and
execve

The general way to use
involves constructing a
Vt pipeline
structure and adding one or more
Vt pipecmd
structures to it.
A
Vt pipecmd
represents a subprocess (or
``command )''
while a
Vt pipeline
represents a sequence of subprocesses each of whose outputs is connected to
the next one's input, as in the example
ls | grep pattern | less
The calling program may adjust certain properties of each command
independently, such as its environment and
nice(3)
priority, as well as properties of the entire pipeline such as its input and
output and the way signals are handled while executing it.
The calling program may then start the pipeline, read output from it, wait
for it to complete, and gather its exit status.

Strings passed as
Vt const char *
function arguments will be copied by the library.

Functions to build individual commands

Ft pipecmd * Fn pipecmd_new const char *name

Construct a new command representing execution of a program called
name

Ft pipecmd * Fn pipecmd_new_argv const char *name va_list argv

Ft pipecmd * Fn pipecmd_new_args const char *name ...

Convenience constructors wrapping
Fn pipecmd_new
and
Fn pipecmd_arg .
Construct a new command representing execution of a program called
name
with arguments.
Terminate arguments with
NULL

Ft pipecmd * Fn pipecmd_new_argstr const char *argstr

Split
argstr
on whitespace to construct a command and arguments, honouring shell-style
single-quoting, double-quoting, and backslashes, but not other shell
evilness like wildcards, semicolons, or backquotes.
This is included only to support situations where command arguments are
encoded into configuration files and the like.
While it is safer than
system(3),
it still involves significant string parsing which is inherently riskier
than avoiding it altogether.
Please try to avoid using it in new code.

Construct a new command that calls a given function rather than executing a
process.

The data argument is passed as the function's only argument, and will be
freed before returning using free_func (if
non- NULL ).

pipecmd_*
functions that deal with arguments cannot be used with the command returned
by this function.

Ft pipecmd * Fn pipecmd_new_sequencev const char *name va_list cmdv

Ft pipecmd * Fn pipecmd_new_sequence const char *name ...

Construct a new command that itself runs a sequence of commands, supplied as
Vt command *
arguments following
name
and terminated by
NULL
The commands will be executed in forked children; if any exits non-zero then
it will terminate the sequence, as with "&&" in shell.

pipecmd_*
functions that deal with arguments cannot be used with the command returned
by this function.

Ft pipecmd * Fn pipecmd_new_passthrough void

Return a new command that just passes data from its input to its output.

Split
argstr
on whitespace to add a list of arguments, honouring shell-style
single-quoting, double-quoting, and backslashes, but not other shell
evilness like wildcards, semicolons, or backquotes.
This is included only to support situations where command arguments are
encoded into configuration files and the like.
While it is safer than
system(3),
it still involves significant string parsing which is inherently riskier
than avoiding it altogether.
Please try to avoid using it in new code.

Ft void Fn pipecmd_get_nargs pipecmd *cmd

Return the number of arguments to this command.
Note that this includes the command name as the first argument, so the
command
`echo foo bar'
is counted as having three arguments.

Ft void Fn pipecmd_nice pipecmd *cmd int value

Set the
nice(3)
value for this command.
Defaults to 0.
Errors while attempting to set the nice value are ignored, aside from
emitting a debug message.

Ft void Fn pipecmd_discard_err pipecmd *cmd int discard_err

If
discard_err
is non-zero, redirect this command's standard error to
/dev/null
Otherwise, and by default, pass it through.
This is usually a bad idea.

Clear the environment while running this command.
(Note that environment operations work in sequence; pipecmd_clearenv
followed by pipecmd_setenv causes the command to have just a single
environment variable set.)

Ft void Fn pipecmd_sequence_command pipecmd *cmd pipecmd *child

Add a command to a sequence created using
Fn pipecmd_new_sequence .

Ft void Fn pipecmd_dump pipecmd *cmd FILE *stream

Dump a string representation of a command to stream.

Ft char * Fn pipecmd_tostring pipecmd *cmd

Return a string representation of a command.
The caller should free the result.

Ft void Fn pipecmd_exec pipecmd *cmd

Execute a single command, replacing the current process.
Never returns, instead exiting non-zero on failure.

Ft void Fn pipecmd_free pipecmd *cmd

Destroy a command.
Safely does nothing if
cmd
is
NULL

Functions to build pipelines

Ft pipeline * Fn pipeline_new void

Construct a new pipeline.

Ft pipeline * Fn pipeline_new_commandv pipecmd *cmd1 va_list cmdv

Ft pipeline * Fn pipeline_new_commands pipecmd *cmd1 ...

Convenience constructors wrapping
Fn pipeline_new
and
Fn pipeline_command .
Construct a new pipeline consisting of the given list of commands.
Terminate commands with
NULL

Joins two pipelines, neither of which are allowed to be started.
Discards
Fa want_out ,
Fa want_outfile ,
and
Fa outfd
from
p1
and
Fa want_in ,
Fa want_infile ,
and
Fa infd
from
p2

Ft void Fn pipeline_connect pipeline *source pipeline *sink ...

Connect the input of one or more sink pipelines to the output of a source
pipeline.
The source pipeline may be started, but in that case
Fn pipeline_want_out
must have been called with a negative
Fa fd ;
otherwise, calls
Fn pipeline_want_out source -1 .
In any event, calls
Fn pipeline_want_in sink -1
on all sinks, none of which are allowed to be started.
Terminate arguments with
NULL

This is an application-level connection; data may be intercepted between the
pipelines by the program before calling
Fn pipeline_pump ,
which sets data flowing from the source to the sinks.
It is primarily useful when more than one sink pipeline is involved, in
which case the pipelines cannot simply be concatenated into one.

The result is similar to
tee(1),
except that output can be sent to more than two places and can easily be
sent to multiple processes.

Set file descriptors to use as the input and output of the whole pipeline.
If non-negative,
fd
is used directly as a file descriptor.
If negative,
Fn pipeline_start
will create pipes and store the input writing half and the output reading
half in the pipeline's
Fa infd
or
Fa outfd
field as appropriate.
The default is to leave input and output as stdin and stdout unless
Fn pipeline_want_infile
or
Fn pipeline_want_outfile
respectively has been called.

Set file names to open and use as the input and output of the whole
pipeline.
This may be more convenient than supplying file descriptors, and guarantees
that the files are opened with the same privileges under which the pipeline
is run.

Calling these functions (even with
NULL
which returns to the default of leaving input and output as stdin and
stdout) supersedes any previous call to
Fn pipeline_want_in
or
Fn pipeline_want_outfile
respectively.

Ft void Fn pipeline_ignore_signals pipeline *p int ignore_signals

If
ignore_signals
is non-zero, ignore
SIGINT
and
SIGQUIT
in the calling process while the pipeline is running, like
system(3).
Otherwise, and by default, leave their dispositions unchanged.

Ft int Fn pipeline_get_ncommands pipeline *p

Return the number of commands in this pipeline.

Ft pipecmd * Fn pipeline_get_command pipeline *p int n

Return command number
n
from this pipeline, counting from zero, or
NULL
if
n
is out of range.

Ft pipecmd * Fn pipeline_set_command pipeline *p int n pipecmd *cmd

Set command number
n
in this pipeline, counting from zero, to
cmd
and return the previous command in that position.
Do nothing and return
NULL
if
n
is out of range.

Ft pid_t Fn pipeline_get_pid pipeline *p int n

Return the process ID of command number
n
from this pipeline, counting from zero.
The pipeline must be started.
Return
-1
if
n
is out of range or if the command has already exited and been reaped.

Ft FILE * Fn pipeline_get_infile pipeline *p

Ft FILE * Fn pipeline_get_outfile pipeline *p

Get streams corresponding to
Fa infd
and
Fa outfd
respectively.
The pipeline must be started.

Ft void Fn pipeline_dump pipeline *p FILE *stream

Dump a string representation of
p
to stream.

Ft char * Fn pipeline_tostring pipeline *p

Return a string representation of
p
The caller should free the result.

Ft void Fn pipeline_free pipeline *p

Destroy a pipeline and all its commands.
Safely does nothing if
p
is
NULL
May wait for the pipeline to complete if it has not already done so.

Functions to run pipelines and handle signals

Vt typedef void pipeline_post_fork_fn (void)

Ft void Fn pipeline_install_post_fork pipeline_post_fork_fn *fn

Install a post-fork handler.
This will be run in any child process immediately after it is forked.
For instance, this may be used for cleaning up application-specific signal
handlers.
Pass
NULL
to clear any existing post-fork handler.

Ft void Fn pipeline_start pipeline *p

Start the processes in a pipeline.
Installs this library's
SIGCHLD
handler if not already installed.
Calls
error (FATAL)
on error.

Wait for a pipeline to complete.
Set
* statuses
to a newly-allocated array of wait statuses, as returned by
waitpid(2),
and
* n_statuses
to the length of that array.
The return value is similar to the exit status that a shell would return,
with some modifications.
If the last command exits with a signal (other than
SIGPIPE
which is considered equivalent to exiting zero), then the return value is
128 plus the signal number; if the last command exits normally but non-zero,
then the return value is its exit status; if any other command exits
non-zero, then the return value is 127; otherwise, the return value is 0.
This means that the return value is only 0 if all commands in the pipeline
exit successfully.

Ft int Fn pipeline_wait pipeline *p

Wait for a pipeline to complete and return the exit status.

Ft int Fn pipeline_run pipeline *p

Start a pipeline, wait for it to complete, and free it, all in one go.

Ft void Fn pipeline_pump pipeline *p ...

Pump data among one or more pipelines connected using
Fn pipeline_connect
until all source pipelines have reached end-of-file and all data has been
written to all sinks (or failed).
All relevant pipelines must be supplied: that is, no pipeline that has been
connected to a source pipeline may be supplied unless that source pipeline
is also supplied.
Automatically starts all pipelines if they are not already started, but does
not wait for them.
Terminate arguments with
NULL

Functions to read output from pipelines

In general, output is returned as a pointer into a buffer owned by the
pipeline, which is automatically freed when
Fn pipeline_free
is called.
This saves the caller from having to explicitly free individual blocks of
output data.

Ft const char * Fn pipeline_read pipeline *p size_t *len

Read
len
bytes of data from the pipeline, returning the data block.
len
is updated with the number of bytes read.

Ft const char * Fn pipeline_peek pipeline *p size_t *len

Look ahead in the pipeline's output for
len
bytes of data, returning the data block.
len
is updated with the number of bytes read.
The starting position of the next read or peek is not affected by this call.

Ft size_t Fn pipeline_peek_size pipeline *p

Return the number of bytes of data that can be read using
Fn pipeline_read
or
Fn pipeline_peek
solely from the peek cache, without having to read from the pipeline itself
(and thus potentially block).

Ft void Fn pipeline_peek_skip pipeline *p size_t len

Skip over and discard
len
bytes of data from the peek cache.
Asserts that enough data is available to skip, so you may want to check
using
Fn pipeline_peek_size
first.

Ft const char * Fn pipeline_readline pipeline *p

Read a line of data from the pipeline, returning it.

Ft const char * Fn pipeline_peekline pipeline *p

Look ahead in the pipeline's output for a line of data, returning it.
The starting position of the next read or peek is not affected by this call.

Signal handling

installs a signal handler for
SIGCHLD
and collects the exit status of child processes in
Fn pipeline_wait .
Applications using this library must either refrain from changing the
disposition of
SIGCHLD
(in other words, must rely on
for all child process handling) or else must make sure to restore
'sSIGCHLD
handler before calling any of its functions.

If the
Fa ignore_signals
flag is set in a pipeline (which is the default), then the
SIGINT
and
SIGQUIT
signals will be ignored in the parent process while child processes are
running.
This mirrors the behaviour of
system(3).

leaves child processes with the default disposition of
SIGPIPE
namely to terminate the process.
It ignores
SIGPIPE
in the parent process while running
Fn pipeline_pump .

Reaping of child processes

installs a
SIGCHLD
handler that will attempt to reap child processes which have exited.
This calls
waitpid(2)
with
-1
so it will reap any child process, not merely those created by way of this
library.
At present, this means that if the calling program which forks other child
processes which may exit while a pipeline is running, the program is not
guaranteed to be able to collect exit statuses of those processes.

You should not rely on this behaviour, and in future it may be modified
either to reap only child processes created by this library or to provide a
way to return foreign statuses to the application.
Please contact the author if you have an example application and would like
to help design such an interface.

ENVIRONMENT

If the
PIPELINE_DEBUG
environment variable is set to
``1''
then
will emit debugging messages on standard error.

EXAMPLES

In the following examples, function names starting with
pipecmd_
or
pipeline_
are real
functions, while any other function names are pseudocode.

The simplest case is simple.
To run a single command, such as
mv
source
dest

SEE ALSO

AUTHORS

An -nosplit
Most of
was written by
An Colin Watson Aq cjwatson@debian.org ,
originally for use in man-db.
The initial version was based very loosely on the
Fn run_pipeline
function in GNU groff, written by
An James Clark Aq jjc@jclark.com .
It also contains library code by
An Markus Armbruster ,
and by various contributors to Gnulib.

is licensed under the GNU General Public License, version 3 or later.
See the README file for full details.

BUGS

Using this library in a program which runs any other child processes and/or
installs its own
SIGCHLD
handler is unlikely to work.