Behind the Scenes

Beginning with Pine 4.00 there are two types of address book storage. There
are local address books, which are the address books that are
stored in a local file (the address books Pine has had all along); and there
are remote address books, which are stored on an IMAP server.

NOTE: The remote address book capability does not allow you to access an
existing local address book from a remote system! That is, you can't set
the remote address book to something like {remote.host}.addressbook
and expect to access the existing .addressbookfile
on remote.host. Instead, you need to create a new remote address book in
a new, previously unused remote mail folder. Then you can use the
Select and Apply Save commands in the address book screen to
Save all of the entries from an existing local address book to
the new remote address book.

Beginning with Pine 4.00 there is a new type of address book called a
remote address book.
A remote address book is stored in a mail folder on an
IMAP server. A Pine remote address book is just like a Pine local address book
in that it is not interoperable with other email clients.
The folder is a regular folder containing mail messages but
those messages are special. The first message must be a pine remote
address book header message which contains the header x-pine-addrbook.
The last message in the folder contains the address book data.
In between the first
and the last message are old versions of the address book data. The address
book data is simply stored in the message as it would be on disk, with no MIME
encoding. When it is used the data from the last message in the folder is
copied to a local file and then that file is used exactly like a local
address book file is used. When a change is made the modified local file
is appended to the remote folder in a new message. In other words, the
local file is just a cache copy of the data in the remote folder. Each client
which uses the remote address book will have its own cache copy of the data.
Whenever a copy is done the entire address book is copied, not just the
entries which have changed.

Pine can tell that the remote data has changed
by one of several methods.
If the date contained in the Date header of the last message has changed then
it knows it has changed. If the UID of the last message has changed, or the
number of messages in the folder has changed, it knows that it has changed.
When Pine discovers the folder has changed it gets
a new copy and puts it in the local cache file.

There is a new configuration file variable for remote address books called
remote-abook-metafile.
The variable is the name of a file in which
information about remote address books is stored. There is one line in the
metafile for each remote address book. The information stored there is the
name of the cache file and information to help figure out when the remote
folder has changed. If the metafile or any of the cache files is deleted
then Pine will rebuild them the next time it runs.

Remote address books have names that look just like regular
remote mail folder names.
For example:

{host.domain}foldername

Pine decides whether or not an address book is remote simply by
looking at the first character of the address book name and comparing
it to '{'.

Information About All Address Books

The address book is named, by default, .addressbook in the
user's Unix home directory, or in the case of PC-Pine,
ADDRBOOK, in the same directory as the PINERC file.
There may be more than
one address book, and the default name can be overridden via an entry in
any of the Pine configuration files. The two configuration variables
address-book
and global-address-book
are used to specify the names of the address books.
Each of these variables is a list
variable. The total set of address books for a user is the combination of
all the address books specified in these two lists. Each entry in the
list is an optional nickname followed by an address book name. The nickname is
everything up to the last space before the file name. The
global-address-book list will typically be configured in the
system-wide configuration file, though a user may override it like most
other variables. Address books which are listed in the
global-address-book variable are forced read-only, and are
typically shared among multiple users.

Local address books (or local cache files for remote address books) are
simple text files with lines in the format:

<nickname>TAB<fullname>TAB<address>TAB<fcc>TAB<comments>

The last two fields are optional. A "line" may be made up of multiple
actual lines in the file by using continuation lines, which are lines
beginning with SPACE characters. The line breaks may be after TABs or in
between addresses in a distribution list.
Each actual line in the file must
be less than 1000 characters in length.

Nicknames (the first field) are short names that the user types instead of
typing in the full address. There are several characters which aren't
allowed in nicknames in order to avoid ambiguity when parsing the address
(SPACE, COMMA, @, ", ;, :, (, ), [, ], , \).
Nicknames aren't required. In fact,
none of the fields is required.

The fullname field is usually stored as Last_name, First_name, in order
that a sort on the fullname field comes out sorted by Last_name.
If there is an unquoted comma in the fullname,
Pine will flip the first and last name around and get rid of
the comma when using the entry in a composition. It isn't required that
there be a comma, that's only useful if the user wants the entries to sort
on last names.

The address field takes one
of two forms, depending on whether the entry
is a single (simple) address or a distribution list. For a simple entry,
the address field is an RFC 822 address. This could be either the
email-address part of the address, i.e., the part that goes inside the
brackets (<>), or it could be a full RFC 822 address. The phrase part
of the address (the fullname) is used unless there is a fullname present
in the fullname field of the address book entry. In that case, the fullname
of the address book entry replaces the fullname of the address. For a
distribution list, the <address> is in the format:

"(" <address>, <address>, <address>, ... ")"

The only purpose for the parentheses around the list of addresses is to
make it easier for the parsing routines to tell that it is a simple entry
instead of a list. The two are displayed differently and treated slightly
differently in some cases, though most of the distinction has disappeared.
Each of the addresses in a list can be a full RFC 822 address with
fullname included, or it may be just the simple email-address part of the
address. This allows the user to have a list which includes the fullnames
of all the list members. In both the simple and list cases,
addresses may also be other nicknames which appear in this address book or
in one of the other address books. (Those nicknames are searched for by
looking through the address books in the order they appear in the address
book screen, with the first match winning.) Lists may be nested. If
addresses refer to each other in a loop (for example, list A includes
list B which includes list A again) this is detected and flagged. In that
case, the address will be changed to "**** address loop ****".

The optional fcc field is a folder name, just like the fcc field in the
composer headers. If the first address in the To field of a composition
comes from an address book entry with an fcc field, then that fcc is
placed in the fcc header in the composer.

The comments field is just a free
text field for storing comments about an
entry. By default, neither the fcc nor the comments field is shown on the
screen in the address book screen. You may make those fields visible by
configuring the variable
addressbook-formats.
They are also searched when you use the WhereIs
command in the address book screen and are visible when you
View or Update an entry.

The address book is displayed in the order that it is stored.
When the user chooses a different sorting criterion, the data is actually
sorted and stored, as opposed to showing a sorted view of the data.

When the address book is written out, it is first written to a temporary
file and if that write is successful it is renamed. This guards
against errors writing the file that might destroy the whole address book.
The address book is re-written after each change. If the address book
is a remote address book, the file is then appended to the remote mail
folder using IMAP.

The end-of-line character(s) in the address book file are those native to
the system writing it. So it is <LF> on Unix and
<CR><LF> on PC's. However, both Unix and PC versions of Pine
can read either format, so it should be possible to share a read-only
address book among the two populations (using NFS, for example). The
end-of-line character for the LookUp file is always just <LF>, even
on a PC.

Starting in 3.90 there is an additional file for each address book, called
the LookUp file. It usually has the same name as the address book file
with the suffix ".lu" appended. (It might have a different name if a file
name length restriction prohibited that name.) This file is created and
maintained by Pine. If it is deleted, Pine will recreate it next time
it runs. Its purpose is to speed up lookups for large address
books and to reduce memory requirements for large address books.
A fairly detailed description of how it is used is
given in src/pine/adrbklib.h.

The lookup file changes whenever the address book itself is changed.
If it doesn't exist, Pine attempts to create it.
If Pine doesn't have permission to create the lookup file with
the standard name, it will create a temporary version in a temp directory.
You want to avoid this since it would have to be rebuilt every
time Pine was run, and rebuilding
takes a significant time for a large address book.
So, if you're going to have a shared address book in a read-only directory,
it is highly desirable to create the lookup file so that the users sharing
it won't have to each create a copy in a temp directory.
You can do that by running Pine and accessing the address book
under a user id which does have permission to write the
file or by using the -create_lu command line
argument to Pine.
If users may be using a shared address book that needs updating,
it is best to move the old address book to another name rather than
copying over it since the file may be opened by running Pines.
It is also best to make the lookup file for the new addrbook before moving
it and the address book file into place, otherwise users may get stuck
attempting to initialize the new lookup file.
The lookup file contains a timestamp which records the mtime of the address
book file when the lookup file was last updated.
Whenever a user runs Pine the current mtime of the address book
is checked against this timestamp and if they differ, Pine will
want to rebuild the lookup file.
Because of this, it isn't a good idea to build the lookup file and then
copy the address book and lookup file into place.
You should move it or copy it in some way which preserves the address book
file's mtime (e.g., use mv or tar).

Validity Checking of Address Books

There is no file locking done on Pine address books, however, there
is considerable validity checking done to make sure that the address book
hasn't changed unexpectedly. Whenever the address book is about to be changed,
a check is made to see if the file is newer than when we read it or the
remote address book folder has changed since we last copied it. If either
of these is true, the change is aborted.

There is an automatic, behind-the-scene check that happens every so often,
also. For example, if someone else changes one of the address books that
you have configured, your Pine's copy of the address book will usually
be updated automatically without you noticing. This checking happens at the
same time as new mail checking takes place, unless you are actively using
the address book, in which case it happens more frequently.

Another sort of validity check is that the lookup file contains a timestamp
internally that is supposed to match the time that the address book file
itself was last modified. If the lookup file timestamp doesn't match the
date of the address book file, a new lookup file is built. If you are
having trouble, it is always ok to remove the lookup file and restart.
Pine will automatically rebuild the lookup file.

One other validity check happens when looking up an entry in the address
book file. An entry is looked up by first getting an offset into the
address book file from the lookup file. A seek to that location is done
and then the entry is read. An entry should be at the start of a line. If
it isn't, something is wrong. In that case, the lookup file is rebuilt
and the operation is repeated if possible.

Beginning with Pine 4.30, configuration information may be stored
remotely.
Remote configuration information is stored in a folder on an IMAP server.
This should be a folder which is used only for storing the configuration
information.
In other words, it should be a folder which didn't exist before.

Remote configuration folders are very similar to remote address book folders.
They both consist of a header message, which serves to identify the type
of folder; the last message, which contains the data; and intermediate
messages, which contain old versions of the data.
The first message must contain the header x-pine-pinerc.

When a remote configuration is being used, the folder is checked to make
sure it is a remote configuration folder, then the data contained in the
last message is copied to a temporary file.
That file is treated just like any regular local configuration file from
that point on.
Whenever a configuration change is made, the entire file is copied back
to the IMAP server and is appended to the folder as a new message.

Because remote configuration folders are so similar to remote address books,
the configuration variable
remote-abook-metafile
is used by both.

Remote configuration folders have names that look just like regular
remote mail folder names.
For example:

{host.domain}mypinerc

Pine decides whether or not a configuration file is remote simply by
looking at the first character of the name and comparing
it to '{'.

Periodically Pine will save the whole
mail folder to disk to prevent loss
of any mail or mail status in the case that it gets interrupted,
disconnected, or crashes. The period of time Pine waits to do the
checkpoint is calculated to be minimally intrusive. The timing can be
changed (but usually isn't) at compile time. Folder checkpointing happens
for both local folders and those being accessed with IMAP. The delays are
divided into three categories:

Good Time:

This occurs when Pine has been idle for more than 30 seconds. In
this case Pine will checkpoint if 12 changes
to the file have been made or
at least one change has been made and a checkpoint hasn't been done for
five minutes.

Bad Time:

This occurs just after Pine has
executed some command. Pine will
checkpoint if there are 36 outstanding changes to the mail file or at
least one change and no checkpoint for ten minutes.

Very Bad Time:

Done when composing a message. In this case, Pine will only
checkpoint if at least 48 changes have been made or at least one change has been
made in the last twenty minutes with no checkpoint.

If Unix Pine is compiled with the compiler
DEBUG option on (the default),
then Pine will produce debugging output to a file. The file is
normally .pine-debugX in the user's home directory where
X goes from 1 to 4. Number 1 is always the most recent session
and 4 the oldest. Four are saved because often the user has gone in and
out of Pine a few times after a problem has occurred before the expert
actually gets to look at it. The amount of output in the debug files
varies with the debug level set when Pine is
compiled and/or as a command
line flag. The default is level 2. This shows very general things and
records errors. Level 9 produces copious amounts of output for each
keystroke.

PC-Pine creates a single debug file
named PINEDEBG.TXT in the
same directory as the PINERC file.

Pine is not designed to process email messages as they are delivered;
rather Pine depends on the fact that some other program (sendmail, etc)
will deliver messages and Pine simply reads the email folders
which that other program creates.
For this reason, Pine cannot filter incoming
email into different folders. It can, however, work alongside most of the
programs available over the Internet which perform this task.
Pine is known to operate successfully with the Elmfilter program and with procmail.

Pine allows users to specify a set of
incoming-folders.
Pine will separate out all the folders listed
as incoming-folders and offer convenient access to these.
We hope that in the future Pine will be able to offer new message
counts for all of the incoming folders, but we haven't done this so
far because of the performance penalty.

A folder is a group of messages.
The default format used by Unix Pine is
the Berkeley mail format. It is also used by the standard mail
command and by elm.
Unix Pine also understands message folders in
other formats, such as Tenex, MH, MMDF, and Netnews.

PC-Pine reads and writes local (PC) folders in a special
format similar to the Tenex format.
Near as we can tell, PC-Pine is the only program to use
this format.
Beginning with version 3.90, PC-Pine includes a ReadOnly
driver for the Berkeley mailbox format in addition. That means that you
can import Unix mail folders, or mount them via NFS or SMB, and PC-Pine
can read them --but not modify them.

Extensions. In the past, file name extensions have been significant in
both Unix Pine and PC-Pine,
but this has caused more problems than it solved.
Therefore, on Unix Pine extensions no longer have any special
meaning, and this is the trend for PC-Pine as well.

By default, PC-Pine adds ".MTX" to the name of any
local (PC) folders that
are referenced, and suppresses the extension from the "Folder List" display.
Now that PC-Pine can read more than one folder format,
the MTX extension no longer implies a particular format,
and is largely irrelevant.
By using the folder_extension
option, you can change this behavior.
In particular, you may set folder-extension to the "null string"
(a pair of double quotes) which tells PC-Pine to
neither add nor hide-from-view any folder name extension.

The reason you might wish to over-ride the MTX default is that recent
versions of PC-Pine have the ability to open (albeit ReadOnly) normal
Unix mail folders. Since it might be inconvenient to rename all of them to
have an MTX extension, it is possible with this option to switch PC-Pine's
behavior so that such folders can be seen and accessed without changing
their names. However, doing this means that your existing PC-Pine local
folders will have apparently changed their names. For example, if you had
a local folder named "FOO" it will now appear in the "Folder List" as
"FOO.MTX". If you wish to save additional messages to that folder, you
will need to enter the full name, "FOO.MTX" at the Save prompt.
Likewise for GoTo.

If you wish to permanently avoid having to deal with folder name
extensions, you will need to set this option to the null string by
entering two double- quote marks, and you will need to rename your
existing local folders to not have an MTX extension. In DOS this can be
done in one command, once you have changed to your mail directory: RENAME
*.MTX *.

We don't know why you might wish to, but you could also use this option to
tell PC-Pine to use an extension other than MTX. In this case, enter the
three characters you desire to use in lieu of "MTX". Note that your
existing folders will need to be renamed to correspond to this new
extension.

Berkeley Mail Format

This format comes to us from the ancient UNIX mail program,
/bin/mail. (Note that this doesn't have anything to do with
Berkeley, but we call it the Berkeley mail file format anyway.) This
program was actually used to interactively read mail at one time, and is
still used on many systems as the local delivery agent. In the Berkeley
mail format, a folder is a simple text file. Each message (including the
first) must start with a separator line which takes approximately the
form:

From juser@u.example.edu Wed Aug 11 14:32:33 1993

Each message ends with two blank lines. There are actually several
different variations in the date part of the string, twenty at last count.
Because of the format of the separators, lines in the mail message
beginning with "From ", space included, risk being confused as message
separator lines. Some mail programs will interpret any line beginning
with "From " as a message separator,
while others --including Pine-- will
not be confused unless the line really looks like a message separator,
complete with address and date. Such lines will be modified to begin with
">From ". In deference to other mail programs, you may also set the
save-will-quote-leading-froms
feature, in which case any line beginning
with "From " will be modified as above. If you see this occasionally in
incoming mail messages,
the culprit is not Pine but the message delivery
program being used at your site.

You can fool Pine into thinking a file is a mail folder by copying a
suitable message separator from a real folder to the beginning of the file
and wherever you want message boundaries. The vast majority of INBOXes
Pine reads and folders it writes are of this format.

Tenex and MTX Formats

Like the Berkeley format, the Tenex folder format uses a single file
per folder. Historically, the name of Tenex-format folders ended with
.txt, but this rule is no longer enforced. The file format
consists of a header line followed by the message text for each message.
The header is in one of two forms:

and is immediately followed by a newline (and the message text).
The fields in the formats are:

dd two-digit day of month (leading space if a single-digit day)
mmm three-letter English month name (Jan, Feb, etc.)
yy two-digit year in 20th century (obsolete)
yyyy four-digit year
hh two-digit hour in 24-hour clock (leading zero if single-digit)
mm two-digit minute (leading zero)
ss two-digit second (leading zero)
zzz three-letter North American time zone (obsolete)
sssss signed four-digit international time zone as in RFC 822
n one or more digits of the size of the following message in
bytes
ffffffffffff
twelve-digit octal flags value

Punctuation is as given above.

The time in the header is the time that message was written to the
folder. The flags are interpreted as follows: the high order 30 bits are
used to indicate user flags, the next two bits are reserved for future
usage, the low four bits are used for system flags (010 = answered, 04 =
flagged urgent, 02 = deleted, 01 = seen).

If a Tenex-format (or empty) file named mail.txt exists in a
Pine user's home directory, this triggers special processing in Pine. When
INBOX is opened, mail is automatically moved from /usr/spool/mail
into mail.txt in the user's home directory.

The format used by PC-Pine is identical to the Tenex format, with two
exceptions: the folder name ends with .MTX instead of
.txt (this is a requirement in the MTX format), and DOS-style
CR/LF newlines are used instead of UNIX-style LF newlines.

Netnews Format

The netnews format is a ReadOnly format which uses directories under
/usr/spool/news as folders.
The /usr/spool/news/ prefix is
removed and all subsequent ``/'' (slash) characters are changed to ``.''
(period). For example, the netnews folder name comp.mail.misc
refers to the directory name /usr/spool/news/comp/mail/misc.
In addition, the news folder name must appear in the file
/usr/lib/news/active for it to be recognized.
Individual messages are
stored as files in that directory, with file names being the ASCII form of
a number assigned to that message. The default locations above can be
changed with the config variables
news-spool-directory
and
news-active-file-path.

There are two kinds of locking which Pine has to worry about. The first
might be called program-contention locking. This affects the times when a
program is performing actual updates on a folder. An update might be a
message delivery program appending a message (sendmail delivering
a message to an INBOX), status changes (checkpoints by Pine every few
minutes) or deletion of messages (an expunge in Pine). For moderate sized
mail messages, these operations should not last for more than a few
seconds. The second kind of locking has to do with user-contention
situations. This would be the case when one folder is shared by a group
of people or even when one person starts multiple email sessions all of
which access the same folders and INBOX.

There are two standard locking mechanisms which handle program-contention
locking. To be on the safe side, Pine implements both of them. The older
mechanism places a file xxxx.lock (where xxxx is the
name of the file being locked) in the same directory as the file being
locked. This makes use of the fact that directory operations are atomic
in UNIX and mostly works across NFS. There are involved algorithms used
to determine if a lock has been held for an excessive amount of time and
should be broken. The second program-contention locking mechanism uses
the flock() system call on the mailbox. This is much more
efficient and the locks can't get stuck because they go away when the
process that created them dies. This is usually found on 4BSD and related
machines.

In addition to these, Pine--through the c-client library--provides robust
locking which prevents several users (or several instances of the same
user) having a mail file open (for update) at once. This user-contention
lock is held the entire time that the folder is in use.

With IMAPd 7.3(63) and Pine 3.84 and higher, the second Pine session which
attempts to open a particular folder
(usually INBOX) with Pine will
``win''. That is to say, the second session will have read/write access
to the folder. The first user's folder will become read-only. (Note that
this is exactly the opposite of the behavior prior
to Pine 3.84 where the
second open was read-only. Having the latest open be read-write seems to
match more closely with what users would like to have happen in this
situation.)
Pine's additional locking is only effective against multiple
uses of Pine or other programs using the c-client library, such as
MailManager, ms, IMAPd and a few others. Beginning with Pine
3.85, there is a -o command line flag to intentionally open a
mailbox read-only.

Pine locking on UNIX systems works by creating lock files in /tmp
of the form \usr\spool\mail\joe. The system call
flock() is then used on these files; the existence of the file
alone does not constitute a lock. This lock is created when the folder is
opened and destroyed when it is closed. When the folder is actually being
written, the standard UNIX locks are also created.

If a folder is modified by some other program while Pine has it open, Pine
will give up on that mail file, concluding it's best not to do any further
reads or writes. This can happen if another mailer that doesn't observe
Pine's user-contention locks (e.g. elm or mail) is run
while Pine has the mail folder open. Pine checkpoints files every few
minutes, so little data can be lost in these situations.

PC-Pine does not do any folder locking. It depends on IMAP servers to
handle locking of remote folders. It is assumed that only one Pine
session can be running on the PC at a time, so there is no contention
issue around folders on the PC itself.

The INBOX folder is treated specially. It is normally kept open
constantly so that the arrival of new mail can be detected. The name
INBOX refers to wherever new mail is retrieved on the system. If
the inbox-path variable is set,
then INBOX refers to
that. IMAP servers understand the concept of INBOX, so
specifying the folder {imap.u.example.edu}INBOX is meaningful.
The case of the word INBOX is not important,
but Pine tends to display it in all capital letters.

The folders for sent mail
and saved messages folders are
also somewhat special.
They are automatically created if they are absent and recreated
if they are deleted.

The file pine.hlp in the pine subdirectory of the
distribution contains all the help text for Pine.
On UNIX, it is compiled right into the Pine binary as strings.
This is done to simplify installation and configuration.
The pine.hlp file is in a
special format that is documented at the beginning of the file.
It is divided into sections, each with a name that winds up being referenced as
a global variable. This file is processed by two awk scripts and
turned into C files that are compiled into Pine.

PC-Pine, which tries to run on machines with as little as 640k
of memory, leaves the Pine help text out of the executable.
PINE.EXE, PINE.HLP, and PINE.NDX
are all needed for PC-Pine's help system.

While Pine was designed in the U.S.
and used mostly for English-language
correspondence, it is a goal for Pine to handle email in almost any
language. Many sites outside of the U.S. run Pine in their native
language. The default character set for Pine is US-ASCII. That can be
changed in the personal or system-wide configuration file with the
variable character-set.

When reading incoming email, Pine allows all character sets to pass
through. Pine doesn't actually display the characters but
simply passes them through;
it is up to the actual display device to show the characters correctly.
When composing email, Pine will accept input in any language
and tag the message according to the character-set variable.
Again, it is up to the input device to generate the correct sequences for
the character set being used.

With the exception of UNICODE-1-1-UTF-7, the outgoing message is checked
to see if it is all US-ASCII text (and contains no escape characters). In
that case, the text will be labeled as US-ASCII even if the
character-set variable is set to something else. The theory is
that every reasonable character set will have US-ASCII as a subset, and
that it makes sense to label the text with the lowest-common-denominator
label so that more mailers will be able to display it. Text in the
UNICODE-1-1-UTF-7 character set is never re-labeled as US-ASCII.
If the outgoing message is not all US-ASCII text, then it will be labeled with
the character-set variable set by the user. If the user has not
set the character-set variable then it will be labeled as
X-UNKNOWN.

BUG: If you prepare a UNICODE-1-1 document and read it into the
composer with ^R,
Pine may mistreat it. If your document, when misviewed
as 8-bit bytes, does not contain any individual bytes greater than 0x7f
base 16, then Pine will re-label your outgoing
message as US-ASCII, even
if your message is really in Unicode Cyrillic, Arabic, or Thai. On the
other hand, if your UNICODE-1-1, when misviewed as 8-bit bytes, does
contain at least one individual byte greater than 0x7f base 16, as is
likely for Unicode French/German/Spanish, Greek, Japanese, and Chinese,
then Pine will retain the UNICODE-1-1 label.

Earlier versions of Pine made use of the character set tags associated
with text in MIME to decide if the text should be displayed or not.
Depending on the character set tag and the character-set variable
in Pine, the text was either displayed as is, displayed with some
characters filtered out, or not displayed at all. The current version
uses a much simpler algorithm in order to maximize the chance that useful
contents are readable by the user. It simply displays
all messages of type text and makes no attempt to filter
out characters that may be in the wrong character set. If the text is
tagged as something other than US-ASCII and the tag does not match the
character set that the character-set variable is set to,
then an attempt is made to convert the incoming characters into the character
set given in the character-set variable.
Characters which are not representable in the character set of the display
will be converted to question marks.
The feature
disable-charset-conversions
may be used to turn off this automatic conversion.
A warning is printed at the start of the message and the characters are
sent to the display. It is
possible that the text will be displayed incorrectly.

There is special-case code which attempts to handle Japanese text (ISO-2022-JP).
Incoming ISO-2022-JP text will be converted to EUC in
UNIX Pine and to Shift-JIS in PC-Pine
before it is sent to the display.
Conversely, when sending mail UNIX Pine will convert EUC into ISO-2022-JP
and PC-Pine will convert Shift-JIS into ISO-2022-JP.
The feature
disable-2022-jp-conversions
may be used to turn off this automatic conversion.

There used to be a facility in PC-Pine to map between various
DOS Code Pages and standard character sets.
That facility, including the CP_TO_ISO and ISO_TO_CP hooks, was broken
for several versions of Pine and is now entirely removed.
We hope that the character set conversion that happens automatically
before displaying a message will handle most of what the Code Page code used
to do.
For example, a Russian user would probably set the
character-set variable in the PINERC file to
the value KOI8-R.
That would cause outgoing messages to be labeled as KOI8-R.
If an incoming message was labeled with the Windows-1251 character set,
Pine should be able to convert the Windows-1251 characters to KOI8-R before
sending them to the display.

If the user is composing mail and is interrupted by being disconnected
(SIGHUP, SIGTERM or end of file on the standard input), Pine will save the
interrupted composition and allow the user to continue it when he or she
resumes Pine. As the next Pine session starts, a message will be given
that an interrupted message can be continued. To continue the interrupted
message, simply go into the composer. To get rid of the interrupted
message, go into the composer and then cancel the message with
^C.

Composition of half-done messages may be postponed to a later time by
giving the ^O command. Other messages can be composed while
postponed messages wait. All of the postponed messages are kept in a
single folder. Postponing is a good way to quickly reference other
messages while composing.

The c-client library allows for several flags or status marks to be set
for each message. Pine uses four of these flags: UNSEEN, DELETED,
ANSWERED, and FLAGGED. The N in Pine's
FOLDER INDEX means that a
message is unseen-it has not been read from this folder yet. The D
means that a message is marked for deletion.
Messages marked with D are
removed when the user Expunges the folder (which usually happens
when the folder is closed or the user quits Pine).
The A in Pine's
FOLDER INDEX means that the message has been replied-to. The * in
Pine's FOLDER INDEX means that the message has been ``flagged'' as
important. That is, the user used the Flag command to turn the
FLAGGED flag on. This flag can mean whatever the user wants it to mean.
It is just a way to mark some messages as being different from others. It
will usually probably be used to mark a message as somehow being
``important''. For Berkeley format folders, the message status is written
into the email folder itself on the header lines marked Status:
and X-Status.
In Tenex and PC-Pine's MTX folder formats, the
status goes into the 36-bit octal flags.

Pine should be able to handle just about any MIME message. When a MIME
message is received, Pine will display a list of all the parts, their
types and sizes. It will display the attachments when possible and
appropriate and allow users to Save all other attachments.

Starting with version 3.90, Pine honors the "mailcap"
configuration system for specifying external programs for handling attachments.
The mailcap file maps MIME attachment types to the external programs
loaded on your system which can display and/or print the file.
A sample mailcap file
comes bundled with the Pine distribution. It includes comments which
explain the syntax you need to use for mailcap. With the mailcap file,
any program (mail readers, newsreaders, WWW clients) can use the same
configuration for handling MIME-encoded data.

If a MAILCAPSenvironment variable is defined,
Pine will use that to look
for one or more mailcap files, which are combined. In the absence of
MAILCAPS, Unix Pine will look for
a personal mailcap file in ~/.mailcap
and combine that with a system-wide file in /etc/mailcap.
PC-Pine will look for a file named MAILCAP in the
same directory as the PINERC file,
and/or the directory containing the PINE.EXE executable.

Messages which include rich text or enriched text in the
main body will be displayed in a very limited way (it will show bold and
underlining).

If Pine sees a MIME message part tagged as type IMAGE,
and Pine's
image-viewer
configuration variable is set, Pine will attempt to
send that attachment to the named image viewing program. In the case of
UNIX Pine, the DISPLAY environment variable is
checked to see if an X-terminal is being used (which can handle the images).
If the image-viewer variable is not set,
Pine uses the mailcap
system to determine what to do with IMAGE types, just as it does for any
other non-TEXT type, e.g. type APPLICATION. For MIME's generic "catch
all" type, APPLICATION/OCTET-STREAM, the mailcap file will
probably not specify any action, but Pine users may always
Save any MIME attachment to a file.

MIME type "text/plain" is handled a little bit differently than the
other types. If you are viewing the main body part in the MESSAGE TEXT
viewing screen, then Pine will use its internal viewer to display it.
This happens even if there is a mailcap description which matches this
particular type. If it is labeled as having a character set other than
the one you are using, it will still be displayed by the internal viewer
(perhaps incorrectly), though you will get a warning message prepended
to the message in the viewing screen. However, if you view a part of
type "text/plain" from the ATTACHMENT INDEX screen, then Pine will check
the mailcap database for a matching entry and use it in preference to
its internal viewer.

Some text attachments, specifically those which are just other email messages
forwarded as MIME messages, are displayed as part of the main body of the
message. This distinction allows easy display when possible (the forward
as MIME case) and use of an attachment viewer when that is desirable (the
plain text file attachment case).

If the parts of a multipart message are alternate versions of the same
thing Pine will select and display the one best suited.
For parts of type "message/external-body", the parameters showing the
retrieval method will be displayed, and the retrieval process is automated.
Messages of type "message/partial" are not currently supported.

There are two important factors when trying to include an attachment in a
message: encoding and labeling.
Pine has rules for both of these which
try to assure that the message goes out in a form that is robust and can
be handled by other MIME mail readers.

MIME has two ways of encoding data-Quoted-Printable and Base64.
Quoted-Printable leaves the ASCII text alone and only changes 8-bit
characters to "=" followed by the hex digits. For example, "=09" is a
tab. It has the advantage that it is mostly readable and that it allows
for end of line conversions between unlike systems. Base64 encoding is
similar to uuencode or btoa and just encodes a raw bit
stream. This encoding is designed to get text and binary files through
even the most improperly implemented and configured gateways intact, even
those that distort uuencoded data.

All attachments are encoded using Base64 encoding. This
is so that the attachment will arrive at the other end looking exactly
like it did when it was sent. Since Base64 is completely unreadable
except by MIME-capable mailers or programs, there is an obvious tradeoff
being made here. We chose to ensure absolutely reliable transport of
attachments at the cost of requiring a MIME-capable mailer to read them.
If the user doesn't want absolute integrity he or she may always
include text (with the ^R command) in the body of a
message instead of attaching it. With this policy, the only time
quoted-printable encoding is used is when the main body of a message
includes special foreign language characters.

When an attachment is to be sent, Pine sniffs through it to try to set the
right label (content-type and subtype). An attachment with any lines
longer than 500 characters in it or more than 10% of the characters are
8-bit it will be considered binary data. Pine will recognize (and
correctly label) a few special types including GIF, JPEG, PostScript, and
some audio formats. Another method which can be more robust and flexible for
determining the content-type and subtype is to base it on the file extension.
This method uses a MIME.Types File.

If it is not binary data (has only a small proportion of 8-bit characters
in it,) the attachment is considered 8-bit text. 8-bit text attachments
are labeled "text/plain" with charset set to the value of the user's
character-set variable. If an attachment is ASCII (no 8-bit
characters) and contains no ESCAPE, ^N, or ^O characters
(the characters used by some international character sets), then it is
considered plain ASCII text. Such attachments are given the MIME label
"text/plain; charset=US-ASCII", regardless of the setting of the user's
character-set variable.

All other attachments are unrecognized and therefore given the generic
MIME label "application/octet-stream".

Pine checks for new mail in the INBOX and in
the currently open folder every two and a half minutes by default.
It used to be 30 seconds
instead of 150 seconds, but we increased it in order to reduce the load on
large systems with lots of Pine users. The value can be changed at
compile-time in the pine/os.h file. This value can be changed with the
variable mail-check-interval.
A new mail check can be forced by redrawing the screen with a ^L.

When there is new mail, the message(s) will appear in the index, the
screen will beep, and a notice showing the sender and subject will be
displayed. If there has been more than one new message since you last
issued a command to Pine, the notice will show the count of new messages
and the sender of the most recent one.

Questions have arisen about the interaction between Pine and external mail
notification routines (biff, csh, login). Firstly and unfortunately, we
have found no PC based program that will check for email on an IMAP server
when PC-Pine is not running. If you find one, please tell us.

The UNIX case is more complicated. Pine sets the modification and access
time on a file every time it performs a write operation (status change or
expunge). You need to see which of these your email notification program
is looking at to know how it will behave with Pine.

It is possible to access mail folders on NFS mounted volumes with
Pine, but there are some drawbacks to doing this, especially in the case
of incoming-message folders that may be concurrently updated by Pine and
the system's mail delivery agent. One concern is that Pine's
user-contention locks don't work because /tmp is usually not
shared, and even if it was, flock() doesn't work across
NFS.

The implementation of the standard UNIX ".lock" file
locking has been modified to work with NFS as follows. Standard
hitching post locking is used so first a uniquely named file is created,
usually something like xxxx.host.time.pid. Then a link to it is
created named xxxx.lock where the folder being locked is
xxxx. This file constitutes the lock. This is a standard UNIX
locking scheme. After the link returns, a stat(2) is done on the
file. If the file has two links, it is concluded that the lock succeeded
and it is safe to proceed.

In order to minimize the risks of locking failures via NFS, we strongly
recommend using IMAP rather than NFS to access remote incoming message
folders, e.g. your INBOX. However, it is generally safe to access
personal saved-message folders via NFS since it is unlikely that
more than one process will be updating those folders at any given time.
Still, some problems may occur when two Pine sessions try to access the
same mail folder from different hosts without using IMAP. Imagine the
scenario: Pine-A performs a write that changes the folder. Pine-B then
attempts to perform a write on the same folder. Pine-B will get upset
that the file has been changed from underneath it and abort operations on
the folder. Pine-B will continue to display mail from the folder that it
has in its internal cache, but it will not read or write any further data.
The only thing that will be lost out of the Pine-B session when this
happens is the last few status changes.

If other mail readers besides Pine are involved, all bets are off.
Typically, mailers don't take any precautions against a user opening a
mailbox more than once and no special precautions are taken to prevent
NFS problems.

UNIX Pine can print to the standard UNIX line printers or to generic
printers attached to ANSI terminals using the escape sequences to turn the
printer on and off. The user has a choice of three printers in the
configuration.

The first setting, attached-to-ansi, makes use of escape
sequences on ANSI/VT100 terminals. It uses "<ESC>[5i" to begin
directing all output sent to the terminal to the printer and then
"<ESC>[4i" to return to normal. Pine will send these escape
sequences if the printer is set to attached-to-ansi. This works
with most ANSI/VT100 emulators on Macs and PCs such as kermit, NCSA
telnet, VersaTerm Pro, and WinQVT. Various terminal emulators implement
the print feature differently. For example, NCSA telnet requires "capfile
= PRN" in the config.tel file. Attached-to-ansi printing doesn't
work at all with the telnet provided with PC-NFS. There is also a closely
related method called attached-to-ansi-no-formfeed which is
the same except for the lack of formfeed character at the end of the
print job.

Attached-to-wyse and attached-to-wyse-no-formfeed are
very similar to "attached-to-ansi".
The only difference is in the control characters sent to turn the printer
on and off.
The Wyse version uses Ctrl-R for on, and Ctrl-T for off.

The second selection is the standard UNIX print command. The default is
lpr, but it can be changed on a system basis to anything so
desired in /usr/local/lib/pine.conf.

The third selection is
the user's personal choice for a UNIX print command. The text to be
printed is piped into the command. Enscript or lpr with
options are popular choices. The actual command is retained even if one
of the other print selections is used for a while.

Both the second and third sections are actually lists of possible commands
rather than single commands.

If you have a PostScript printer attached to a PC or Macintosh, then you
will need to use a utility called ansiprt to get printouts on
your printer. Ansiprt source code and details can be found in
the ./contrib directory of the Pine distribution.

The three printer choices are for UNIX Pine only.
PC-Pine can only print to the locally attached printer.
All printing on PC-Pine
is done via ROM BIOS Print Services (Int 17h). After verifying the
existence of a local printer via the BIOS Equipment-List Service (Int 11h),
it simply sends the message text, character by character, to the
first printer found using ASCII CR and LF at the end of lines and followed
by an ASCII FF. Note, some system adjustments using the PC's "MODE"
command may be required if the printer is not on the first parallel port.
PC-Pine cannot generate PostScript,
so printing to exclusively PostScript printers does not work.

Pine users get two options for moving
messages in Pine: Save and Export.
Save is used when the message should remain ``in the
Pine realm.'' Saved messages include the complete header
(including header lines normally hidden by Pine),
are placed in a Pine folder collection and
accumulate in a standard folder format which Pine can read.
In contrast, the Export command is used to write the contents
of a message to a file for use outside of Pine.
Messages which have been exported are
placed in the user's home directory (unless the feature
use-current-dir is
turned on), not in a Pine folder collection.
Unless FullHeaderMode is toggled on, all delivery-oriented headers are
stripped from the message. Even with Export, Pine retains
message separators so that multiple messages can accumulate in a single
file and subsequently be accessed as a folder. On UNIX systems, the
Export command pays attention to the standard umask for
the setting of the file permissions.

Pine's default behavior is to keep a copy of each outgoing message in a
special "sent mail" folder. This folder is also called the fcc for "file
carbon copy". The existence, location and name of the sent mail folder
are all configurable. Sent mail archiving can be turned off by setting
the configuration variable
default-fcc="". The sent mail folder
is assumed to be in the default collection for Saves,
which is the first collection named in
folder-collections.
The name of the folder
can be chosen by entering a name in default-fcc.
With PC-Pine,
this can be a bit complicated. If the default collection for Saves is
local (DOS), then the default-fcc needs to be SENTMAIL,
which is syntax for a DOS file. However, if the default collection
for Saves is
remote, then the default-fcc needs to be sent-mail
to match the UNIX syntax.

The configuration variable
fcc-name-rule
also plays a role in selecting the folder to save sent mail in.

A danger here is that the sent mail could grow without bound. For this
reason, we thought it useful to encourage the users to periodically prune
their sent mail folder. The first time Pine is used each month it will
offer to archive all messages sent from the month before. Pine also
offers to delete all the sent mail archive folders which are more than 1
month old. If the user or system has disabled sent mail archiving (by
setting the configuration variable default-fcc="")
there will be no pruning question.

Spell checking is available for UNIX Pine only. We could not find an
appropriate PC based spell checker to hook into PC-Pine.
Even UNIX Pine
depends on the system for its spell checking and dictionary. Pico, the
text editor, uses the same spell checking scheme as Pine.

Lines beginning with ">" (usually messages included in replies) are not
checked. The message text to be checked is on the standard input and the
incorrect words are expected on the standard output.

The default spell checker is UNIX spell. You can replace this
by setting the speller configuration
variable.
Pine also respects the environment variable SPELL.
The spelling checker reads its words from a standard dictionary on the system.
Below is a description, contributed by Bob Hurt, of how you can create
your own personal dictionary with additional ``correct'' words.

Step 1:

Make a file with all the words you want to include in your new
dictionary. I did mine with one word per line in alphabetical order. Caps
don't matter at all, as far as I know.

Step 2:

At the UNIX prompt, type "cat [word file] | spellin /usr/dict/hlista
> [new dict name]" where [word file] is the file you just created and
[new dict name] is the name of the new dictionary that Pine will look at
instead of the standard /usr/dict/hlista. I named my word file
.bobwords and my dictionary .bobspell so I don't have to
see them when I do a ls command (ls doesn't list "dot"
files). I also put the above command into my .alias file as the
command makedict so I can add a word to my word file and easily
recreate my dictionary. NOTE: the new dictionary is in something called a
"hashed" format, and can't be read normally.

Step 3:

Check your new dictionary. At the UNIX prompt, type "cat [word file]
| spellout [new dict name]" If you did everything correctly, it should
just give you another prompt. If it lists any of the words in your file,
something is wrong. I can try to help if all else fails.

Step 4:

Now you have to tell UNIX to use your dictionary instead of the
standard one by setting the environment variable SPELL to access
your dictionary. Go into your .login or .cshrc file in
your home directory (it doesn't seem to make a difference which one you
use) and add the line

setenv SPELL "spell -d [new dict name]"

I also created an alias for SPELL in my .alias file
so I can use the UNIX spell command to spell-check a file outside
of Pine. (The .alias line is: alias spell 'spell -d [new dict
name]')

Step 5:

Now you need to logoff and log back on to let UNIX look at your
.login (or .cshrc) file.

Here is an alternative method suggested by Zachary Leber:

Create a list (e.g. .zachwords) with the upper case followed
by lower case words, sorted alphabetically.

Add this line to .cshrc:

setenv SPELL 'spell +/home/ie/rsa/.zachwords'

The limitation here is that the path must be absolute (e.g.
+~/.zachwords doesn't work).

My man pages for spell show this + flag to be an easy way to do the
exception list. This way you don't have to bother with hash lists or
rehashing, and it seems to work across several platforms.

Pine has been designed to require as little as possible from the terminal.
At the minimum, Pine requires cursor positioning, clear to end of line,
and inverse video. Unfortunately, there are terminals that are missing
some of these such as a vt52. Pine makes no assumptions as to whether the
terminal wraps or doesn't wrap. If the terminal has other capabilities it
may use some of them. Pine won't run well on older terminals that require
a space on the screen to change video attributes, such as the Televideo
925. One can get around this on some terminals by using "protected field"
mode. The terminal can be made to go into protected mode for reverse
video, and then reverse video is assigned to protected mode.

Pine handles screens of most any size and resizing on the fly. It catches
SIGWINCH and does the appropriate thing. A screen one line high will
display only the new mail notification. Screens that are less than ten
columns wide don't format very nicely or work well, but will function fine
again once resized to something large. Pine sets an internal maximum
screen size (currently 170x200) and decides to use either termcap
or terminfo when it is compiled.

On the input side of things, Pine uses all the standard keys, most of the
control keys and (in function-key mode) the function keys. Pine avoids
certain control keys, specifically ^S, ^Q, ^H, and ^\
because they have other meanings outside of Pine (they control data flow,
etc.) ^H is treated the same as the delete key, so the
backspace or delete keys always works regardless of any
configuration. There is a feature compose-maps-delete-key-to-ctrl-d
which makes the delete key behave like ^D rather than ^H (deletes current
character instead of previous character).

Sometimes a communications program or communications server in between you
and the other end will eat certain control characters. There is a
work-around when you need it. If you type two escape characters followed
by a character that will be interpreted as the character with the control
key depressed. For example, ESC ESC T is equivalent to
^T.

When a function key is pressed and Pine is in regular (non-function key)
mode, Pine traps escape sequences for a number of common function keys so
users don't get an error message or have an unexpected command executed
for each character in the function key's escape sequence. Pine expects
the following escape sequences from terminals defined as VT100:

Arrow keys are a special case. Pine has the escape sequences for a number
of conventions for arrow keys hard coded and does not use termcap
to discover them. This is because termcap is sometimes
incorrect, and because many users have PC's running terminal emulators
that don't conform exactly to what they claim to emulate.
In some versions of Pine before 4.00 there was a compile-time macro
called TERMCAP_WINS which could be set to cause the termcap
or terminfo definitions to be used instead of the built in definitions.
Beginning with 4.00 there is a hidden runtime feature which can be turned
on to accomplish the same thing. The feature is called
termdef-takes-precedence
and it can be set in any of the Pine configuration files.
Some arrow keys on old terminals send single control characters
like ^K (one even sends ^\).
These arrow keys will not work with Pine.
The most popular escape sequences for arrow keys are:

Start up Pine from an xterm, and specify a "resource name".
This resource name will allow the user to specify resources for Pine (that
deviate from the defaults). For example, xterm -name Pine -e pine
& (the resource name Pine corresponds to the translations
just added in the .Xdefaults file).