=head1 NAME
icprogrammer - Interchange Programmer Reference
=head1 DESCRIPTION
=head1 Introduction
Interchange is a highly-complex but very powerful web application
server focused on ecommerce. It is built on the power of Perl, using
many of its standard modules and capabilities while defining many
more.
While Interchange focuses on e-commerce, it is really a
general-purpose database access, retrieval, and templating systems.
Besides online stores, here are some of the applications have been
built on top of it:
Auction
Calendar
Configuration management
Content management
Document archival and rental
Guestbook
Image archival and download
Intranet
MP3 jukebox
Poll
Quiz
Software repository
Web log
This reference attempts to illuminate the source code of Interchagne
and how you can write Perl enhancements, gadgets, and applications
that integrate with Interchange.
=head2 Software installation
To follow along, it is recommended you get the latest release of
Interchange (5.0 as of this writing), unpack it from the tar file, and
install it at a private directory. For the purposes of this document,
it will be assumed that Interchange is installed at
/usr/local/interchange and that the catalogs are installed at
/usr/local/catalogs.
=head2 Software prerequisites
Interchange only I a few added Perl modules, which can be
installed by getting the Perl CPAN bundle Bundle::Interchange.
Install that (usually as root) with:
perl -MCPAN -e 'install Bundle::Interchange'
To get most of the modules Interchange can use:
perl -MCPAN -e 'install Bundle::InterchangeKitchenSink'
=head2 Audience
This reference is not meant for casual users of Interchange. Though
they might learn something from reading it, it would probably not do
them much targeted good. A reasonable set of prerequisites to make
reading this document profitable include:
=over 4
=item Programming knowledge
=back
A good knowledge of Perl or B knowledge of other programming
languages is needed.
=over 4
=item Database knowledge
=back
Interchange is all about databases, and a knowledge of the concepts of
database programming and SQL is strongly recommended.
=over 4
=item Networking knowledge
=back
The more you know about networking and the web, the more comfortable
you will be with this document.
=over 4
=item UNIX knowledge
=back
Almost all production Interchange servers are UNIX-based, so knowledge
of that is helpful.
=head1 Overview of Interchange
Interchange is a daemon server, similar to a web server. Its entry
point is usually talking to it over a socket via its own protocol.
That socket can be either UNIX domain or INET domain, or an infinite
number of either.
=head2 Catalogs
Interchange as a server dispatches connections to a catalog, an
independently-configurable set of data and templates. These are for
the most part completely independent of each other, though they
inherit common global characteristics and settings. Almost all of
those can be overridden by the catalog.
=head2 Hacking
Of course Interchange's source is completely open and available. You
could, if you wished, hack on it all you wanted. However this is
strongly discouraged, for the simple reason that you can override
almost any behavior with configurations and tag definitions of your
own. In fact, if you want to override a core routine you can even do
that.
So if you are tempted to hack a routine in the core, simply override
it with:
GlobalSub <{name}) {
$out = "Your name is $Values->{name}, in case you forgot.";
}
return $out;
[/calc]
The above is completely identical to the ITL-only snippet above in
effect.
In addition, you can call defined ITL tags in your embedded Perl:
[calc]
my $out = '';
my $name = $Tag->value('name');
if($name) {
$out = "Your name is $Values->{name}, in case you forgot.";
}
return $out;
[/calc]
Again, the result is identical to the previous two examples.
User Defined Tags
ITL is comprehensibly extensible. You can produce your own ITL tags
that are fully as powerful as the ones supplied with the distribution.
In fact they are indistinguishable, as you will see when you examine
the code hierarchy.
These tags can use any Perl module, use external programs, or
basically do most anything Perl can, providing you define them in the
Global configuration. Tags defined in the Catalog configuration are
restricted by Perl's standard Opcode and Safe facilities, though they
can optionally be allowed global capability.
See L for complete information on ITL.
=head2 Talking to Interchange via socket
Interchange can run in any of several modes:
=over 4
=item Foreground
=back
The foreground, meaning the same Interchange server listens for
connections and then runs the tasks those connections cause.
=over 4
=item Forking mode
=back
One master Interchange listens for connections, then forks instances
to handle the tasks those connections cause. The forked instance
terminates at the end of the task.
=over 4
=item Prefork mode
=back
Similar to the way Apache does, Interchange can fork off a number of
instances that all listen to the sockets open for connections. The
first one to answer gets the task, runs it, then returns to listen
again. After MaxChildRequests requests, it dies and causes another new
instance to take its place.
=over 4
=item mod_perl mode
=back
Interchange can be loaded into I. See the documentation in
scripts/ic_mod_perl.PL for information.
=over 4
=item SOAP mode
=back
Interchange can listen to a socket designed to accept a SOAP
connection -- those always run in prefork mode. This mode can co-exist
with other modes, so the same Interchange server can serve both page
and SOAP requests.
=head2 Talking to Interchange over the command line
Interchange starts its servers by being invoked from the command line.
Other command line invocations can stop the server via signal, cause
addition of additional catalogs to respond to, remove catalogs from
the list to respond to, or cause execution of "cron" jobs.
=head2 Data structure overview
Interchange has three major data stuctures, which correspond to the
master server, the catalog, and the user.
You can examine two of these structures by setting in interchange.cfg:
DumpStructure Yes
This will by default dump an interchange.structure file which shows
the global configuration, and a CATALOGNAME.structure file in each
catalog directory showing that catalog's configuration.
The third structure, the user data session, can be viewed with the
following ITL placed in a page:
[dump]
The Global configuration
This is held in a set of variables inhabiting the Global package. They
define overall server behavior, and contain pointers to the catalog
structures.
The Global configuration is defined in interchange.cfg and any
files that it reads via include statements. The configuration is
produced by parsing interchange.cfg with the routine
Vend::Config::global_config.
Directives can be defined for parsing by the catalog configuration
within the global configuration -- and they can be deleted as well.
The only way to define new global directives is via hacking the
source. Luckily, this is just about never needed -- you can define
settings for use by your programs in Variable or other repositories.
The Catalog configuration
Each Interchange catalog has its own configuration completely
independent from others. It is basically produced from the file
catalog.cfg in the directory defined as the base for the catalog.
It is parsed by the subroutine Vend::Config::config.
We say basically, because there are many ways to alter catalog
configuration. (CATNAME below refers to the name of the catalog being
configured.)
=over 4
=item ConfigAllBefore
=back
Global catalog configuration preamble, affecting all catalogs, can be
defined by the Global directive ConfigAllBefore. It defaults to
catalog_before.cfg in the Interchange software directory
(/usr/local/interchange).
LI1. CATNAME.before
An individual per-catalog preamble configuration is defined in
$Global::ConfDir/CATNAME.before.
By default it would be /usr/local/interchange/etc/CATNAME.before.
=over 4
=item CATNAME.site
=back
A file in the catalog directory which is read before catalog.cfg.
Deprecated.
=over 4
=item catalog.cfg
=back
The normal configuration file.
=over 4
=item CATNAME.after
=back
An individual per-catalog postamble configuration is defined in
$Global::ConfDir/CATALOGNAME.after. This can be used to prevent
user catalogs from doing unsafe things -- for instance enforcing the
use of encryption, or preventing running in WideOpen mode.
By default it would be /usr/local/interchange/etc/CATALOGNAME.after.
=over 4
=item ConfigAllAfter
=back
Global catalog configuration postamble, affecting all catalogs, can be
defined by the Global directive ConfigAllAfter. It defaults to
catalog_after.cfg in the Interchange software directory
(/usr/local/interchange).
=over 4
=item command line
=back
Any configuration passed on the command line at Interchange startup is
applied last. For instance, to test out a catalog named foundation
with a different invocation URL without having to alter the config
files:
bin/interchange --foundation:VendURL=http://localhost/cgi-bin/found \
=over 4
=over 8
=over 12
=item *
foundation:SecureURL=http://localhost/cgi-bin/found \
=item *
foundation:RobotLimit=1000
=back
=back
=back
That will set the foundation catalog directive values VendURL,
SecureURL, and RobotLimit, overriding any settings in the
configuration files.
=over 4
=item Tied configuration
=back
Interchange has dynamic catalog configuration as well. See
I.
=head2 Session data structure
Each user session is a hash reference saved in some sort of data
repository. By default it is file-based using L, but it can
reside in any Interchange database type as well.
It is placed at the global variable location $Vend::Session, which for
programming use in UserTag and GlobalSub routines is $Session (meaning
$Vend::Interpolate::Session).
The structure is initialized when the session is created (or canceled
by the user). The initial form is described in
Vend::Session::init_session:
$Vend::Session = {
'ohost' => $CGI::remote_addr,
'arg' => $Vend::Argument,
'browser' => $CGI::useragent,
'referer' => $CGI::referer,
'scratch' => { %{$Vend::Cfg->{ScratchDefault}} },
'values' => { %{$Vend::Cfg->{ValuesDefault}} },
'carts' => {main => []},
'levies' => {main => []},
};
This structure is used as a repository for the transitory user session
values like form values, scratch variable settings, payment
transaction results, errors, and any other user-tied values. It is
also possible to add code that can be run on a user-by-user basis with
the Autoload, Filter, and Profile facilities.
=head1 Tour the source
Navigating the Interchange source requires a couple of clues. The main
program invocation point is bin/interchange in the Interchange
software directory.
=head2 From startup to serving content
Once Interchange is invoked, it does some basic program configuration
at the top of that file. The types of available database facilities
and modules are determined, and the base modules are brought in with
"use" or "require". Execution by a non-root user ID is checked.
After the initial program configuration, execution goes to the
main_loop() subroutine in bin/interchange. Some more initialization is
done, then the command line options are parsed. Options mostly will
set the program mode (i.e. start, stop, kill, test, cron, or other
command line actions), but can also set Global and Catalog
configuration values.
Once the options are parsed, Interchange will chdir() to the
Interchange software directory (/usr/local/interchange) and run its
global configuration. That means all file names passed to it during
this phase are relative to that program root.
Part of global configuration is determination of the ITL tags that
will be used by Interchange. By default, that is all files with
appropriate extensions under the code directory. Sets of tags to be
used can be set with the I and I directives.
Global configuration also includes specifying the catalogs that will
be configured and loaded in the next phase. This is done via the
I directive. An important part of that directive is supplying
the script parameter, which is used to initialize the pointer
structure which will select the catalog based on the URL coming in.
After Global configuration, catalog configuration commences, via the
::config_named_catalog() routine, which calls Vend::Config::config().
Each catalog specified in the global configuration has a base
directory. Interchange does a chdir() to that directory and parses the
various configuration files, databases and specified command-line
parameters.
After the catalog is configured, the database is opened to ensure that
database table objects are initialized properly. It is then
immediately closed.
The resulting Catalog configuration structure reference is then saved
in $Global::Selector and $Global::SelectorAlias so that the calling
URL can map to the proper catalog.
Once all configuration is done, Interchange determines the program
mode. There are only two modes -- test and serve. The test mode
simply exits the program at this point -- it is used to test validity
of the configuration.
If the mode is serve, ::main_loop() calls
Vend::Server::run_server(). Based on global configuration, one of the
server modes discussed previously is initialized and Interchange
starts listening on one or more sockets for a connection from a
client. (This is not true for mod_perl mode -- Interchange simply
exits at that point and the code is waiting for mod_perl to call it.)
While waiting for a connection, signals are disabled and handlers are
set up for TERM, HUP, INT, USR1, and USR2. TERM and INT both cause the
main server to exit; HUP signals Interchange to look for a
reconfiguration event; and USR1 and USR2 are optionally used to keep
track of how many servers are running.
B: Because signals are not especially safe in Perl prior to
5.8.0, occasionally a core dump can occur on receipt of USR1 or USR2.
This is especially true for BSD with its reentrant system calls. They
can be disabled by setting I to 0 -- I mode is
strongly suggested if that is done.
Once a connection is received, the connector parameters are checked
for security constraints and Vend::Server::connection() is called. It
reads the input from the client and constructs the environment,
%CGI::values array, and any passed entity like an HTTP POST or
multipart form (for file upload). Those are stored and and object
referring to them and containing the connection file handle is
constructed. That object is passed to main::dispatch() for processing.
The main::dispatch() routine performs more transaction setup then
determines the catalog that will process the request. It sets
$Vend::Cfg to the preset configuration for that catalog, sets file
permissions as appropriate, and the catalog's database is opened.
Once initialiation of the catalog configuraion is complete, user
initialization begins. Interchange determines the user session ID, if
any, and restores the user session from the session database or starts
a new session as appropriate. Perl objects that will be used in the
session are initialized or constructed, auto-login is run, and the
locale is determined and set. After that, the URI path is parsed,
I and I routines are run.
Finally a transaction action is determined. The action is the first
path component of the path passed to Interchange. The remainder is
passed to the subroutine implementing the action, and may be used as
default path information for content or for other purposes.
For example, if the catalog VendURL is /cgi-bin/foundation and the
URI sent to Interchange is
/cgi-bin/foundation/order/something/or/another, the action is
order, and the path sent to the action routine is is
something/or/another.
If the transaction action is not mapped via standard system actions
defined in the variable %action, or in the I or
I directives, then the action path component is restored to
the content path, and that page is served
(order/something/or/another in the example above).
If the action is mapped, it is run. If it returns a true value, the
page to be served is determined by the setting of
$CGI::values::mv_nextpage. The action can produce send its own output
and return a non-true value, in which case Interchange will terminate
the transaction at that time.
After the action is run and/or content is served, Interchange runs
AutoEnd, saves the user session, closes the catalog database, and
finally main::dispatch() returns. The calling
Vend::Server::connection() does some cleanup and returns to the server
loop. If the server was forked for that transaction only, it sends a
signal indicating it is done, cleans up PID files, and exits. If it is
in the foreground or in PreFork mode, it scrubs the Vend:: and CGI::
namespaces and returns to waiting for the next connection.
=head2 Notes about databases
Interchange maintains objects for all of its database tables defined
in I. These can be of diverse SQL, DBM, and LDAP types.
When the database is initialized at catalog configuration time, the
individual database tables may be opened depending on type. In
general, SQL and LDAP types are always opened, and DBM types are not.
Opening a database table can be expensive in terms of CPU and IO time.
So when the database is opened for a page transaction, Interchange
creates a "dummy" table object that waits for a real access. Those
objects are trivial to create, and a fast processor can create
hundreds of thousands per second.
When access is made, the database table is really opened and the
expensive initialization is done. This allows many tables to be ready
for access while only the ones used take up CPU and IO time.
Copyright 2002-2004 Interchange Development Group. Freely
redistributable under terms of the GNU General Public License.