DESCRIPTION

Sendbatches administers batched transmission of news to other sites.
It should be run periodically, as the owner of the news database, by
cron(8) or similar means. It prepares and sends batches of news,
subject to restrictions on available space and length of outbound
queues.
Normally, sendbatches does locking to ensure that only one sendbatches
is running at a time. The -p option suppresses the locking and permits
parallel sendbatches runs, although lower-level locking is done to
ensure that only one is trying to prepare batches for any particular
site at a given time. Parallel sendbatches runs impose very heavy
system loads but may be useful to systems with extensive hardware
parallelism and many outbound news feeds.
Each site that can have batches sent to it needs a site directory under
/var/spool/news/out.going. If sendbatches is invoked with specific
sites given, it considers batching for those sites, only, in that
order. By default, sendbatches consults the batchparms file (see
below) to determine what to do: If there is a /default/ entry in
batchparms, sendbatches will consider batching for all sites that have
directories in /var/spool/news/out.going, in oldest-first order by
modification time of the directory. If there is no /default/ entry,
sendbatches considers batching for those sites named in batchparms, in
the order named.
To use the batcher, names of files to be sent to a specific site should
be appended to a togo file in its site directory. The batcher expects
the lines in togo to have two fields, a filename (as a full pathname,
or relative to /var/spool/news) of an article and its size in bytes. A
missing size field is arbitrarily assumed to be a default average.
Sendbatches uses a number of auxiliary programs to do the real work.
The search path it uses to find them includes, in order, the site
directory for the site in question, /var/lib/news/bin, and
/usr/lib/news/batch. This permits per-site and per-news-database
overrides of the default behaviors. Sendbatches provides all these
programs with environment variables NEWSSITE, containing the name of
the site that batches are being prepared for, and NEWSSITEDIR,
containing the full pathname of the site directory, in case these are
useful in customization.
The names of most of the auxiliary programs, and some other parameters,
are taken from the file /var/lib/news/batchparms, an ASCII text file.
Empty lines and lines starting with ‘#’ are ignored. Other lines
specify the behavior for sites, one line per site. A site line is four
fields, separated by white space.
The first field is the site name. A line whose site name is /default/
specifies what parameters should be used for sites not explicitly
mentioned. (The presence or absence of such a line also influences the
behavior of sendbatches when invoked without arguments; see above.)
The second field is the class of the site. If sendbatches is invoked
with the -c option, it attempts batching only for sites of the
specified class. A class is a single letter, by convention ‘u’ for
UUCP feeds and ‘n’ for NNTP feeds; user-defined classes should be
uppercase letters.
If the character ‘!’ appears anywhere in the class field of a
batchparms line, that disables the line. Any site whose batching would
have been controlled by that line will have no batching attempted for
it under any circumstances. This provides a way to turn a site off
temporarily.
The third field is the size of batches to be prepared (before
compression), in bytes. It may optionally be two sizes separated by
‘-’, in which case the first is the nominal size and the second is the
absolute maximum. If only one size is given, that is the nominal size,
and the absolute maximum is three times that.
The fourth field is the maximum length of the output queue for
transmission to that site. If it is ‘-’, no queue-length limiting is
done.
The fifth field, which may contain white space, is the command line
(normally a pipeline of three-four programs, possibly with options) to
be used to build, compress, and transmit batches to that site. It
receives the contents of the togo file on standard input. It may not
contain any single quotes (’).
For each site being considered for batches, sendbatches first
determines whether there are in fact any articles to be batched.
Assuming there are, sendbatches then finds the batchparms line for that
site and (if queue-length limiting is done for that site) invokes
queuelen (see newsaux(8CN)) to find out the size of the outbound queue
for the site. Sendbatches limits the number of batches prepared to the
minimum of the limits implied by queue lengths and available space.
Sendbatches uses batchsplit as necessary to slice chunks out of the
togo file, each chunk containing the togo lines for a batch limited to
the nominal size. Exception: a single article bigger than the nominal
size will still go out as one batch, provided it does not exceed the
absolute-maximum size.
Each chunk is then processed through the command line, which usually
consists of a batch preparer (typically batcher), which assembles the
articles into a batch, a batch compressor (typically compcun), which
performs compression, possibly a batch encoder, which encodes the
compressed batch against the vagaries of ill-designed transmission
channels, and a batch transmitter (typically viauux), which sends the
batch on its way (e.g. enqueues it for transmission). All are run with
/var/spool/news as the current directory, so non-absolute pathnames in
the chunk are valid filenames; NOTE that this represents a change from
earlier releases, in which only the preparer ran there.
If there is anything left in the file togo.leftover after a chunk is
processed, sendbatches assumes this is the portion of the chunk which
could not be processed at this time. If/when this happens, sendbatches
replaces the chunk file with the contents of togo.leftover and attempts
no further batching to that site on this run.
Batch preparers in the standard distribution are:
batcher Normal batching.
batchih Ihave-sending part of uucp ihave/sendme (not to be
confused with NNTP).
batchsm Sendme-sending part of ihave/sendme.
batchra Requested-article-sending part of ihave/sendme.
Batchih, batchsm, and batchra have to map from the phony ‘‘site name’’
given in their batchparms line to the name of the site they should
actually send to; they do this by stripping off the last ‘.’ and
everything that follows (usually ‘.ihave’ or ‘.sendme’ respectively,
but on machines which limit the size of filenames these may have to be
shortened).
Caution: batchih and batchsm do their work by constructing an article
and feeding it to inews, which means that the batch size must be within
what inews will accept, if it imposes limits (see inews(1CN)).
Batch compressors can include ordinary programs like compress and gzip,
but one special compressor is provided by the distribution:
compcun Compression with compress plus the silly B-news-
compatible ‘#! cunbatch’ header. Options, if any, are
passed to compress.
Batch encoders supplied with the distribution are:
bencode Encodes 8-bit data using only the ASCII characters
‘‘A’’ - ‘‘Z’’, ‘‘a’’ - ‘‘z’’, ‘‘0’’ - ‘‘9’’, ‘‘+’’, and
‘‘-’’. The ASCII characters blank, newline, and ‘‘/’’
also appear in the encoded file, but do not represent
encoded bits. The encoded data is terminated with a
byte count and cyclic redundancy check for detecting
corruption. This ought to suffice to get data through
almost any network.
c7encode Encodes 8-bit data into a 7-bit form optimized for
transmission by uucp ‘f’ protocol. The encoding is
complex and bizarre. Obsolete; use bencode instead.
nencode Feebly attempts to protect text (not compressed!) data
by prepending a blank line to the input and an ‘N’ to
each line. Obsolete and not recommended; use bencode
instead.
Most transmitters take an optional site argument, using the value of
the NEWSSITE variable if no argument is supplied. Batch transmitters
in the standard distribution are:
viauux Normal transmission via UUCP. -z or -n may be used to
feed the corresponding option to uux. A -g option may
be used to feed the grade to uux (the default is grade
‘d’). A -d option may be used to specify a
decompressor program (e.g., gunzip) to the site’s
rnews; beware that old rnewses may not recognize this.
-c may be used to request execution of the site’s
cunbatch program rather than rnews.
viamail Mail the batch to ‘site!rnews’. -e specifies mailing
to ‘site!enews’ instead. -@ specifies use of @
syntax, e.g. ‘rnews@site’, instead of ! syntax.
viainews Feed the batch back to inews (arguments are ignored)
(normally useful only for ihave/sendme).
viarsh Use rsh to run rnews on the site via Ethernet,
Internet, etc. (the directory containing rnews must
be in the default PATH on site).
Sendbatches logs some information about sites with backlogs in
/var/lib/news/batchlog (see also newsdaily(8CN)). This is intended to
help detection and diagnosis of flow problems.

SEEALSO

DIAGNOSTICS

Complaints, if any, from sendbatches and its auxiliaries are mailed via
report.

HISTORY

Written at University of Toronto as part of the C News project. A
number of the ideas came from Chris Lewis, who was then with Motorola.
Bencode written at University of Waterloo by Reg Quinton and Ken
Lalonde.

BUGS

Sendbatches estimates available space without considering the effects
of compression; this is usually too conservative.
Viarsh does not incorporate a spooling subsystem, so a slow site stalls
the entire batching system and a non-responding site loses news. It is
not recommended for bulk transmission or where high reliability is
essential.
There ought to be an encoder using MIME’s base64 encoding and suitable
descriptive headers.
The logging is overly simplistic and doesn’t work well when selective
batching (controlled by site arguments, classes, or disabling) is being
done.
10 March 1995 NEWSBATCH(8cn)