ipsmtp

ipsmtp
This feeds and receives from Internet Mail.
It sends to an SMTP server and can receive either on a local server (Unix only)
or from a POP3 server.
We define OUTGOING as FIP -> SMTP and the reverse (SMTP or POP3 -> FIP)
INCOMING.
Routing Files Outbound
----------------------
Messages for sending are put in the input queue, usually spool/2smtp, with the
FipHdr field DA containing one or more Mail Address (separated by commas if
multiple) - such as wizzo@fingerpost.co.uk.
Messages can be sent directly to the spool/2smtp queue, routed normally by Fip
or as a group client for Syndication by IPPOST.
In the latter case, the client is specified as type:unixmail and there is a
special DU called TOUNIXMAIL in USERS which has the syntax :
TOUNIXMAIL= DP:(hostname) DQ:2smtp DC:(whatever)
where hostname is the name of the UnixBox doing the transfer
and whatever is the character set required.
The From field or sender is the header field SU or, if it exists, the SA
(Source Author) which is formatted as SA@SU. SU is normally the Source system
or type while SA is the normally the logon or sender. If the SA is an Atex
style 'LOGON ;dd/mm,hh/nn' then only the logon is used.
So if the SA:fippo and SU:liverpool-fc.co.uk, the sender becomes
fippo@liverpool-fc.co.uk
If the SA is ALWAYS the full address - like SA:hedgehog@flea-pit.com, then use
the '-F' input switch to NOT add the SU header field.
If neither SU nor SA exist, the default sender is 'fip' unless overridden by
the 'defsender:' in the parameter file. Note that this name must be in the
passwd file as a valid logon.
The data of the outgoing messages will be converted if 8 bit or control chrs to
the normal '=XX' hex format used.
Routing Files Inbound
---------------------
Messages from the Internet are sent to the special DU, FIPMSG which is either
hardcoded to a specific destination, or more usually, directed to IPPOST to
detetmine if the recepient is on the editorial system OR is a mailbox (IPBOX)
customer.
Testing
-------
If things do NOT look like they are working, you can run ipsmtp/ipsmtpssl
manually with the -1 and -D to run once and display t
he handshake.
Parameter File
--------------
The parameter file, normally tables/mail/SMTP, is read for the logon names to
scan on a regular basis plus the destination to send these to. The syntax of
the parameter file is :
; comment
defsender: (default sender) default: fip
Used ONLY if there is NO fipHdr field SU nor SA.
This is used for the 'From:' field for outgoing messages.
forcesender: (Force sender to this logon) default: no
force-sender: (Force sender to this logon) default: no
default-sender: (Force sender to this logon) default: no
This is the default reply-to address. It must be a valid logon and domain. It
is also used for the 'From:' field for ALL outgoing messages if there is NO
'forcefrom' (see below) or FipHdr field 'DJ' in the file to send irrespective
of whether there is a FipHdr SA or SU.
default-sender is used ONLY if there is no force-sender and nothing in DJ
It should be JUST the address :
forcesender: lucy@sky-with-diamonds.com
forcefrom: (Force the From field)
force-from: (Force the From field)
This is the From field in the outgoing file which does not have
to be the same as the sender's address.
If using the '-h' host input switch it can contain a 'pretty'
version :
eg forcefrom: "MP3 king Steinhoff" <dohrayme@singalong.com>
defdest: (default Fip Destination (DU FipHdr field) default: FIPMSG
chrset: (Source character set ie SC header field) default: ascii
newDA: FipHdr field to replace the DA field which default: DA
holds the email address of the person to sent to.
eg newDA:DZ
This can also be an combination field :
combie:QA DA|DZ|NA
newDA:QA
hash-in-address: (FipSeq Chr)
Some old-style email addresses have a leading hash/pound/# sign - but a hash
is a field sepearator in FipHdr. So another chr is used in the DA FipHdr and
mapped at the last moment.
default is \243
hash-in-fiphdr: (FipSeq Chr)
Sometimes a Subject line might have a hash/pound/# sign - but a hash is a
field sepearator in FipHdr. So another chr is used in the DA FipHdr and mapped
at the last moment.
default is '#' (ie a hash is a hash and that is where the FipHdr field ends
!)
eg in tables/route file, map a HASH out of the way : HASH-IN-HDR:\035
in the ipsmtp parameter file, to map back : hash-in-fiphdr:\035
newDI: FipHdr field to replace the DI field which default: DI
is used for the 'Subject' field.
newDJ: FipHdr field to replace the DJ field which default: DJ
is used for the 'From' field.
balance-seqno: Send the Sequence number to this Balance Group
(see 'ipbalan')
balance-delete: Delete the Data on remote systems once sent
using this Balance Group (see 'ipbalan')
forcedomain: (domain name) default: do not
Strip any domain in the FipHdr field SA and add this one
in its place.
forcedomain:commies.com
So if the SA:albert@redsunderthebed.com,
then the from field becomes
From: albert@commies.com
strip-domain-on-send: Send only the logon/mail name default: domain too
and strip any domain name.
defsubject: (default Subject line if nothing specified)
default: "Message for you"
att-message: Message to replace an attachment with.
default:** Attachment has been Stripped, pls contact System Managers for
(name of attachment file)
log-line: (FipSeq) Additional information for the log file entry in ALL
extra-log-fiphdr: Fipseq FipHdr fields to add to any transmission log
This is triggered on a file by file basis by the FipHdr field
ZSMTP_LOG:(dest) in the file to send - (dest) being a valid
DU in tables/sys/USERS. Use this to pass FipHdr fields from
the incoming file to the outgoing
default: none
eg. extra-log-fiphdr:\n\SI:\SI\n
send-new-fiphdr: Fipseq FipHdr fields to add to any outgoing file to act as
defaults etc
send-extra-fiphdr: Fipseq FipHdr fields to add to any outgoing file to act as
defaults etc
The NEW variant is added BEFORE the real FipHdr, so can be used for Defaults
(and the real
FipHdr fields can override these).
The EXTRA variant will add at the end of the file's FipHdr and so will
override
any setting,
This is triggered on a file by file basis by the FipHdr field
default: none
eg. send-extra-fiphdr:\nZSMTP_TYPE:text/plain\n
stream-connections:yes default: no
Do NOT drop the connection/reconnect for every file.
Once a connection is up to a mailserver, any subsequent file
for the same server is sent in the same transmission.
(using SMTP only)
fiphdr: (Map incoming mail field to a Fip Hdr field) default: none
fiphdr:DI Subject
fiphdr:AZ Message-Id
ignore: (Mail header field to always ignore) default: none
ignore: Status
ignore: Apparently-To
ignore: X-Msmail-Priority
decode:
unencode: Decode any base64 sections. default: no
decode: This will NOT decode attachments
decode:text This will NOT decode attachments
decode:all This will decode body AND attachments
maxattempts: (no of attempts before the file is sent to woops) default: 10
This checks the FipHdr 'DO' which is managed internally. (UNIX only)
logeachfile:(dest) Send a Success/failed msg to this destination for each
file.
There is no default.
This log file is just a FipHdr with the following extra fields :
DR-File Sent OK DR:ok or DR:error
DG-Will Retry later DG:retrying or DG:stopped
DT-Some message text DT:No connection
DO-Number of last attempt DO:5
HT-Date and Time HT:25 Nov 98 15:35:25
default: no log created.
The DR and DG messages can be changed by using :
log-dr-ok:(FipSeq)
log-dr-error:(FipSeq)
log-dg-stop:(FipSeq)
log-dg-retry:(FipSeq)
eg: log-dr-ok:File \VN sent to Remote Number \DA (\$h:\$n)
msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
'logeachfile'
msg. This should be in FipHdr format and be in FipSeq. It can be
used to pass FipHdr fields in the outgoing file into the log file.
eg msgeachfile: DF:logdial\nSS:\SS\n
default: nothing added
send-smtp-directly: (yes/no)
Do not use a single or a pair of hosts and send directly to
'mailhost.(domainname of remote)'
(Same as the -H input switch)
default-smtp-host: (hostname or IPaddress)
For sending emails out - Same as the '-h' Input Switch
alternate-smtp-host: (hostname or IPaddress)
a failover hostname for if the default fails
connection-timeout: (secs)
Max time in seconds for the connection to the SMTP server
The call is aborted if it takes longer than this.
If the remote is across the computer room, this time can be
reduced to 15 or 20 secs; if over the internet, 120 secs is the norm.
default: 120 for 120 seconds, minimum is 10
send-single-smtp: send smtp in one file per session which is then cut and
reopened for the next file/session.
hostname: (hostname to use)
Cosmetic !
timing-stats: yes/no
Show timing stats if the incoming file has the HE/HR fiphdr
default: no
use-sx:
or use-external-file:
if there is an SX FipHdr field with a path to the data file, use that rather
than the data in the input file.
wrap-subject-chr: (FipSeq chr)
If the Subject field has this chr, wrap the line
default: none
This might have been added by ipxchg :
z:convert-fiphdr-to-qp:DI:DJ:utf-8:\002
wrap-subject-chr:\002
force-subject-wrap: (No of chrs)
Normally subject lines at this total
default: 76
eg force-subject-wrap:76
check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
Use this when for POP3 accesses ONLY where ipsmtp on 2 systems is accessing
the same POP3 account(s) in a redundant way.
It is used to decide whether the current host should be getting the files or
not.
ie in the parameter file is
check-primary-server:timersvr
and in the DEST_REDUN is
; psuedohost primary secondary
timersvr fip1 fip2
and in the SYSTEM file for both fip1 AND fip2 there is the same 'ipsmtp'
line with the same parameter file
If fip1 is up, 'ipsmtp' on fip1 will always get while on fip2 it will just
check/loop.
If fip1 is down 'ipsmtp' on fip2 will start processing the files.
use-tls:yes/implicit/explicit/no
The commends are for a ftp running over SSL/TLS on the remote server
default is NO
no - normal, standard FTP on (normally) port 21 for the control
yes or explicit - connect (normally) on port 110 in clear then use SSL for
USER, PASS and data
implicit - connect (normally) on port 995: use SSL for all conversations
tls-auth: (XXX)
AUTH type for TLS/SSL default: TLS
ssl-method: (1,2,3,23,999)
Version number to use for TLS/SSL default: 999 for current default (2 or 3)
ssl-password: (password)
ssl-passwd: (password) default: none
Optional password if the handshake requires a shared secret
ssl-cert: (name of a PEM certificate file) default: none
ssl-root-cert: (name of a root PEM certificate file) default: none
Optional certificates - held in tables/ssl
Then for EACH logon you wish to track INCOMING mail from the Internet specify a
line begining :
logon: (logon name) default: none
or unix: (unixlogonname) default: none
These two parameters are identical.
There is also a means, for UnixBoxes only, to poll for ALL mail
for ALL the members of a group (as in the /etc/group file) :
group: (unixgroupname) default: none
Whether 'logon', 'unix' or 'group' there are other (optional)
parameters:
sendto:(newaddress)
fiphdr:(FIP hdr sequence)
striphdr:
keephdr:
pop3pwd: (Password for POP3)
where sendto allows you to specify another name for the DA field
IPPOST will use this to route. By default
the Fip Hdr field DA will hold the logon name.
fiphdr allows you to add to the FIP hdr. This may be any
- perhaps the DU field to change the destination.
striphdr will strip the SendMail header except for the
From and Subject lines
pop3pwd specifies the password to use IF you are accessing
a POP3 server to receive incoming mail. It is NOT
required for UnixBoxes scanning local mail queues.
To add a new logon :
1. Read the 'passwd' file for the exact Unixlogon name - eg newboy
2. Enter this name in SMTP file as :
logon:newboy
3. If this is the same name as in file ATXMSG, enter line
newboy edone
where edone is the name of your editorial system
4. If 'newboy' is really called 'dirtyoldman' on the 'atex'
editorial system, the entries should be:
SMTP file - logon:newboy sendto:dirtyoldman
ATXMSG file - dirtyoldman atex
To bringup IPSMTP :
1. Create logons as above
2. Check you have destinations in the USERS file :
FIPMSG= DP:fipmain DQ:post DC:atexeol
MSGFORATX_EDSYS= DP:edsys3 DQ:atexq-grp DC:atexeol
where 'fipmain' is the name of this UnixBox.
Destination FIPMSG is generated by IPSMTP.
Destination MSGFORATX_* is generated by IPPOST IF the name is in the FIPMSG
file.
eg if the Editorial System is called ATEX (environment variable FIP_EDSYS) and
teh queue required is email-wir on system3 :
MSGFORATX_ATEX= DP:atex3 DQ:email-wir DC:atexeol
3. Input Queue spool/2smtp needs to be added
4. IPPOST needs to be running too (unless you change the FIPMSG DP/DQ to
somewhere else)
5. Outbound Format is ATXMSG for IPGTWY/IPOUT/IP2WTR etc.
Please check if this exists and has the correct setup
The current version can scan up to 1000 Unixmail boxes.
Where sections of FipHdr fields are required or changes to the output style,
use
keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysA
dmin manual for more information).
They are normally specified :
fixed:QZ 1234543
partial:QT ST,3,2,U,<,>
combie:QY ep|na,(0000000)a
option:QE ep,11,7,s
repeat:QK XK,-,3
or repeat:QP PK,,4,#X
style:QS XN,%.03d
replace:QN NN abc=DEF def=GHI
For outbound traffic, FipHdr fields used are :
Mandatory
DA holds one or more remote email addresses
multiple addresses are separated by a comma.
Can be overridden by 'newDA'
Can be changed by 'forcedomain'
Optional
SA holds the sender's email address
can be overridden by 'forcesender' (actual address)
can be overridden by 'forcesender' (from field)
can be defaulted by 'defsender'
DI holds the Subject: field
DJ holds the From: field
ZSMTP_BINARY for sending Binary files
ZSMTP_ENCODE holds the Content-Transfer-Encoding
default to Content-Transfer-Encoding:quotedprintable
ZSMTP_TYPE holds the Content-Type. no default
ZSMTP_LOG will send a log of the SMTP transmission to this
destination (for SMTP outbound traffic using the '-h' input
switch only)
ZSMTP_HOST IPaddress or hostname of the mail server if NOT the default
(for SMTP only).
Input Switches
--------------
Input parameters are : all optional :
-1 : Get once only then stop for POP3 or Receive only. default: spool
Either -p (path) or -3 (pop3 server) must be used with this option
This also incorporates -Q so no queu is scanned and no mail sent.
-3 : Hostname where the pop3 daemon is running. default: none
Note -p and -3 are mutually exclusive.
-a : queue for non-text attachments default: none
default is that attachments are passed through in text
-A : destination for non-text attachments default: none
default is that attachments are passed through in text
-C : do NOT convert unprintables to "=XX" format default: do
-d : do NOT delete each unix msg default: delete
-D : display the conversation with the remote server default: no
valid ONLY with the -1 for single shot; used for debugging troublesome
connections
-f : do NOT output the original filename as the first
line of text default: output name
-F : do NOT output the original sending system (SU) as the From
field default: From: SU
-h : Hostname where the smtp daemon is running. default: that system
-H : do NOT add an extra Mime Headers as the file
is in the correct format default: add To/From/Subject
-i : Receive spooled queue if you are using 'sffmail' default: using Path
-k : alternate host name default: none
use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster
-K : quiet mode - message if NOT the primary system default: do not message
-l : do NOT log each msg in or out default: log
-m : (ipchkmail folder name) default: process inside ipsmtp
Do not convert incoming SMTP and POP3 files immediately but pass them to
ipchkmail running on this folder which does a better job! especially for
attachments-within-attachments
-o : Next fip queue for incoming files default: spool/2go
-O : Done queue for spooled files out default: none
-p : Path on this system holding mailboxes default: depends on the OS
Unix only. The full pathname MUST be stated.
Note -p and -3 are mutually exclusive.
-q : Send Queue to spool default: spool/2smtp
-Q : Do NOT send any mail and therefore do NOT scan any input queue
default: scan
-r : eolns for the data part should be CR NL default: just NL
-R : do NOT resolve hostnames default: do
This will resolve any hostnames in the DEST_REDUN file (if there is one)
-S : do NOT split files on the "From ..." line default: do
-t : scan time of queue default: 3 secs
-T : log the trace of each transaction default: do not
Normally a one line log of each file sent is stored in
log/remote_trace with a name of 'date_Address'.
-w : for file->Sendmail, set line length. default: 76 + translate
and do NOT translate unprintables to "=XX" format
To NOT wrap text and no translations -w 0
This can be ignored for a single file if it has
the FipHdr field ZSMTP_BINARY:
-W : for files outbound, set JUST the line length def: 76 + translate
but still translate unprintables.
-x : Offline queue default: none
Normally ipsmtp will retry - and possible delay files also queued up - any
message that has not been delivered.
Use this to shuffle files that error to a different folder where another
ipsmtp will take care of the baduns
-X : Send directly to the Email server of each destination default: no, use a
single relay server
This depends on the resolv.conf or Windows DNS to be setup correctly of
course.
-z : parameter file default: mail/SMTP
-v : display version number and exit.
Now a statement of sanity (?) about incoming mail from the Internet :
If like most of the civilised world you prefer to keep well away from anything
'sendmail' can throw at you, then let ipsmtp just grab its files from the
sendmail internal queue. Or even better use POP3 and forget about it all - no
setup, no nothing extra!
If you are unlucky enough to have a Sendmail specialist on your staff, before
sending for the men in white coats, point out there is a '-p path for unix
mailboxes' switch. Your guru should dictate (they always dictate - though they
claim they are working with your best interests in mind) where he is willing to
drop files for those logons you are interested in and add the -p switch to the
tables/sys/SYSTEM file. Easy huh !
Note there was a subtle change at version 15 which changed the default from
ATXMSG to FIPMSG.
Version Control
;16v22 09nov01 cleanup after failed connections better
;a 16mar02 added remote_trace and -O done queue
;b-c 03jun03 cleanup line-enders in mimehdr and streaming for smtp
and added ZSMTP_HOST
;d-i 22sep03 added send-extra-fiphdr: and stream-connections:yes
;j-l 29jun04 added -1 and speedy
;m-p 17sep05 added alternate-smtp-host and hostname: and log-line
;q 10aug06 added remote_trace style timings to send file done msg as H4
;r-t 21sep06 allow quotes in email address and wrap long subject lines
;u 13apr07 added extra-mime-headers:
;v4 06feb08 added hash-in-address ;2 modette for size of email list
;3-4 added check-primary-host for POP3 and chkmail-folder:
;5-6 bugette with send_smtp and logging long DA
;7 3sep10 added use-sx
;8-9 10nov10 bug in pop3 when Uidl has non-alnum
;10-11 19may11 bugette when TLS does not work and b64 '=' padding
;12 22aug13 added POP3 error message on logon/pwd and added -1 for display
;13-14 30sep13 added hash-in-fiphdr ;15 30apr14 file-trace
;16 7oct14 if -H and not -C, do not encode any mime headers in the data
;17-18 26jun15 make donque variable
;19-22 25jul16 tuning and buffersizes and better error messages
(copyright) 2017 and previous years FingerPost Ltd.