How to use extensions in Tcl

Jean-Claude WipplerMay 1998

Extensions are modules written
in a language different from Tcl (C or C++, usually) which extend the functionality
of Tcl in some way. This is an extremely powerful mechanism - the
sophisticated Tk graphical GUI toolkit is basically an extension of Tcl.
This is an attempt to summarize
how to install and use "Extensions" in the Tcl scripting language.

This text was written for
the script programmer who writes utilities and applications in Tcl - and
to some degree also for anybody who just wants to use a ready-made software
solution based on extensions and wants to get it all up and running with
a minimum of hassle. There is no information in here about creating extensions,
although extension writers might find it useful to skim this text
to better serve the people using their extensions.

To turn Tcl onto a "better Tcl" - or to support a different programming style

To interface the Tcl scripting environment to existing software libraries

To hide implementation details by providing only object code releases

To rewrite Tcl bottlenecks as C/C++ code for improved performance

As a way to incorporate Tcl in an existing body of C/C++ code

The last reason is actually
the opposite of extending the Tcl environment - in this case, the Tcl interpreter
becomes part of an existing application and activated when appropriate.
Whether one considers the mix of Tcl with non-Tcl code as "Tcl calling
extensions" or as "an application calling the Tcl interpreter", both are in fact
surprisingly similar in terms of getting the whole kaboodle working. In
both situations, lots of little issues need to be addressed to be able
to solve the entire puzzle.

Pure Tcl is both simple and
powerful. Very elaborate programs can be built using just Tcl (using
tclsh
- the Tcl command shell) or Tcl with Tk (using wish
- the windowing shell). The apparent simplicity lies in the fact
that Tcl performs a script as if the computer understands the commands
which form the script. So all you need to do is install the Tcl
system on your machine, and bingo... your computer has gained the ability
to perform a script written in Tcl. Then, just create or download a script
which does what you want and you will be all set.

Unfortunately, the moment
a programming language like C or C++ enters the picture - this simplicity
is brutally sacrificed. All of a sudden, you may need to deal with
"source code", "object code", "compiling and linking", "make", "static
vs. dynamic linkage", "shared libraries", and more - basically irrelevant
- technical terms.

What's the cause of all this?
Well... the Tcl software is playing a clever trick on you. Your computer
doesn't have a clue how to perform a script of Tcl commands. It needs
a separate tclsh
or wish
program to decipher the script and figure out what it's supposed to be
doing. Both tclsh
and wish
are interpreters - a special class of programs (written in C as
a matter of fact), which go through a script to decipher and perform the
steps laid down in that script. The only thing computers know, is
how to perform tasks specified in machine code. And both tclsh
and wish
are stored as machine code.

Too illustrate the number of steps involved in using extensions in Tcl,
here's roughly what happens on Linux:

Y O U

T C L

U N I X

$ tclsh

- Not running -

Shell examines PATH to find tclshFound in /usr/local/bin/Turns out to be a symbolic link to tclsh80Prepares to load /usr/local/bin/tclsh80Needs to resolve all shared libraries first
Loader looks for shared lib libtcl8.0.soFound in /usr/local/lib/Look for others, e.g. libc.so and libdl.soUse LD_LIBRARY_PATH to find and load all

Prepares to load <path>/XX.soNeeds to resolve all shared libraries first
Loader looks for shared lib libtcl8.0.soFound in /usr/local/lib/Look for others, e.g. libc.soUse LD_LIBRARY_PATH to find and load lib

Commands written in the C and
C++ language are not directly understood by computers. First, you need a compiler
to translate the high-level commands of C and C++ into an elaborate sequence
of machine language commands. The original commands are stored as
text files and called source code, as opposed to the gibberish produced
by compilers, which is called object code. In principle, object
code is the end result, but it almost always needs to be combined with
other pieces of object code - standard modules and/or software supplied
by others. This process is called linking - and the result
is an executable file. This is the file you can "run",
"launch", or "execute" - these verbs all mean the same thing: "start doing
what it's supposed to do".

This is why tclsh
is an executable file (on Windows, it is actually called "tclsh.exe"),
and so is wish.

In practice, there are usually
two more steps involved: configuration (before starting compilation), and
installation (after completing the link). The tool which automates most
of this is called make - it is driven by a set of instructions stored
in a makefile.

First some advantages: machine code is blindingly
fast (usually several orders of magnitude faster than Tcl, in fact). Furthermore,
it is possible to do things in machine languge which cannot be done in
pure Tcl. There are many situations where you cannot avoid using machine code.

Yet machine code is also
notoriously tricky, and it can crash - the computer can lock up, your
screen goes blank, or other confusing things start to happen. Modern compilers
and languages help a lot to avoid mistakes, but... crashes are still a
fact of life. A benefit of scripting languages like Tcl is that they drastically
reduce this problem, because no new, untested (and potentially faulty),
sequence of machine code is needed to perform a new script. The robustness
of Tcl is essentially determined by the robustness of the Tcl executable
file - not by the complexity of the scripts it performs.

A major source of problems
is caused by the fact that each computer type (CPU type, actually) works
only with its own specific machine code. Also, because each operating
system requires some highly specific software (the OS interface),
a different executable file is required to perform the same task on each
separate combination of:

Distributing a complex piece
of software (such as Tcl) as ready-made executable file for each possible
combination is not feasible. Instead, Tcl and extensions are usually supplied
in source code form, along with the mechanism that compiles and links the
software in a wide range of environments. Note that with N computer
types and M operating system revisions, there are N x M different executable
files, but the compilation and linking process need only deal with N +
M variations (usually even less, compilation on Unix is often the same - with
just one or two minor variations).

As you can imagine, machine
code issues add a substantial amount of complexity to the process of software installation.

Extensions - machine code modules
which extend the capabilities of Tcl - are enormously powerful, but introduce
the issues related to machine language described earlier. In a way,
extensions are the best of both worlds, to the point even that Tcl was
designed from the start with this functionality in mind. Extensions
are at the heart of much of the power and success of Tcl.

But alas, that means you'll
have to go through a sometimes frustratingly messy process of creating
machine code for them.

Everything beyond this point
deals with the task of getting the source code of C and/or C++ software
converted into a correctly operating set of machine code. Fortunately,
there are a number of important tools which do almost everything for you,
and if the extension writer has done his homework, they should make this
process trivially simple. By outlining the process and the goal, I hope
you will be able to form a mental picture and figure out where to look
if anything fails to work as expected.

There are essentially two
ways to incorporate an extension:

Create a modified version of
tclsh
or wish
which includes the extension (static linkage)

Create a dynamically loadable
module, which many Tcl systems know how to activate (dynamic linkage)

Actually, a third option is
slightly different in concept, but technically similar to the static linkage
approach just mentioned:

Define the application in C
or C++, then link it to the Tcl core libraries (embedding)

Which one is used depends on
many factors, which are beyond the scope of this document. Suffice
to say that this is usually a decision made by the author of the extension,
so you'll have little choice and may even find out that each extension
you want to use is different in this respect. Some platforms add
specific constraints on what options you have. All this, evidently, doesn't
really help to make life easier...

To use an extension - or a set
of extensions - you need to address the following issues:

Decide which extension version
to use

Obtain the extension software

Apply last-minute changes (bug-fixes,
patches) if needed

Configure the extension build
mechanism for your system

Find missing pieces, if any,
and install them

Compile and link the source
code

Run a test to verify that the
extension works

Install the extension in a place
where Tcl will be able to find it

Once you've completed these
steps, you're ready to use the extension. Often the whole process
works like a charm, especially if you have a widely used machine and operating
system, use a popular release version of Tcl, and if the author of the
extension is keen on seeing the extension being successfully used by others.
Otherwise, please close all your windows (those with real glass in them)
- because you might be tempted to jump out at some point - things really
can become that frustrating... Please also keep in mind
that there is a slight but real chance that you won't succeed
in getting the extension to work flawlessly. At the end of this document
are some tips on what to do when these steps don't work as expected.

If you're still reading...
good - things will work out fine, they usually do. So let's go through
each of these steps.

Like most software components, extensions carry version number to track
software changes and bug fixes. If the extension has a homepage on the web,
and the software you have is old (more than a few months perhaps), you may wish
to visit the website and check its current status. It can be frustrating to struggle
through an installation or end up with an unstable extension, only to discover
that problems with it have been solved in a newer release and that you
could have avoided them.

If you want solid software, stay away from alpha (and perhaps also beta)
releases. These indicate confidence levels with respect to testing. If you want
the latest gizmo's and like to live on the cutting edge of software development,
get the most recent release there is. In that case, a tip: if you ever contact
the author, don't "complain about your problem", but "report a confusing issue"...
and let the power of these four words surprise you. Yes, they will...
because - in general - alpha and beta mean "be thankful I'm making this
version public, even though the work is not done yet".

If you don't already have a copy of the software, or if it's not recent enough, or
if you're not 100% certain that it is the correct release for your system, then these are
also good reasons to locate the extension's homepage on the net and
see if you need to download more files.

If you start off with an incompatible release, you're going to waste a lot of time.
Therefore, it's worth finding out as much as you can about its requirements as
early on as possible.

Incompatibilities can exist in a number of areas:

Wrong machine type (if this is a binary release)

An extension which requires a specific version of Tcl/Tk

Dependency on software which you do not have

Dependency on other extensions (each having their own requirements)

On Unix: requiring "superuser" privileges to install - which you may not have

Requiring specific software (such as Tcl) to be installed in specific places

Known incompatibilities with other software you are using

In general, extension writers are painfully well aware of these issues, and will
point out in great detail what is required, and what the known problem spots are - if any.

On Unix installations, a tool called autoconf is becoming the de-facto standard
for installation, because it is able to verify and resolve an incredible number of
installation issues. If it is mentioned in the supporting documentation, you can
assume that on Unix, compatibility will probably be ok.
Autoconf will be discussed again in step 4 below.

On Windows and Macintosh, there are several standard installers, which take care
of many issues. In this case, it is not uncommon for extensions to be supplied as
one-step self-installing packages, which indeed work "right out of the box".

On Linux, a tool which is rapidly gaining acceptance is RPM, the Redhat Package
Manager. It offers the simplicity of Windows and Macintosh installers, on Linux.
If you work with Linux, chances are that you already learned to love "rpm -i".
RPM can be used for binary-only as well as source-code based installs, but it requires
a specially prepared RPM package (usually, the extension author will have to do this).
If you can, use RPM. It is smart enough to figure out dependencies, can handle future updates,
and even knows how to uninstall a package. It is also able to install a
package over a network.

This is an issue which applies more to large software applications than to extensions,
which are usually very small. Patching is a way to avoid having to download an
entire package again, in which merely a few tiny changes were made - it works by describing
the differences from one minor release to the next.

You may run into extensions which have been released as a package, but
for which tiny (but usually important) bug-fixes have been made available, in the form
of patches. These patches need to be applied to source files, and result in
one or more minor modifications. The way to apply patches is usually described
in the accompanying documentation on the website or CDROM.

To illustrate the way dependencies can play nasty tricks with you: patches are often based on a popular
tool called "patch" (of course). But suppose you don't have patch? Well, you'll have
to get that package first, then. When you do, you find out it needs to be compiled.
But what if you don't have a compiler? Well, then you'll have to get that as well.
When you do, you find out that the compiler comes with several patches... oops, a dead end!
No, don't worry. These problems (and worse: needing a compiler to compile itself before
you can use it) are well understood and have been dealt with. In the real world, such
things are very unlikely. But it shows the intricate nature of some of these issues.

It is very important to apply all patches before you
proceed with the next step.

At about this point, the way in which extensions are installed starts to vary substantially,
depending on whether you're installing on Unix, Windows, or Macintosh.

Unix configuration

This is the most complex environment to configure, because of the extreme range of machines
on which Unix is available. At the same time, in the true spirit of Unix, this is also the
platform on which installation is usually the most automated.
To illustrate this, in a perfect world, the following commands will do everything that
is needed:

% ./configure
% make
% make test
% make install

After you have installed several extensions, you will see that many installations are
indeed minor variations of this. And if you're very lucky, you might even wonder why this
whole document was even written, and what the mumbo-jumbo and at times panicky
undertone in this document is all about.

But this world isn't perfect yet.

Windows configuration

Configuration on Windows comes in two flavors: either you have a binary installation,
and need to place each file in the proper place (if only it were as simple as it
sounds!), or you have source code and use one of the supported C/C++ compiler environments
to compile the extension yourself. For the latter, you'll usually need either Microsoft
Visual C++ (MSVC), or Borlands C++ (BORC) - these are both commercial products.
Some extensions support both, others have been
tested only with one of them (they are not 100% compatible, alas).

Binary installations require fewer steps, and are simpler. If an Installer is
used, you can simply run it and respond to a few questions.

Otherwise, most packages for Windows come in the form of "zip" archives. Just
unpack them, using WinZip or some other similar utility. Then follow the instructions
in the accompanying "README.TXT" file.

In the case of source code distributions, there will be either "IDE files" or makefiles
(often called "makefile.vc" if intended for use with MSVC, or "makefile.bc" for BORC).
I usually copy the appropriate one to "makefile", and look at it
with a textfile editor to check and adjust all the path settings. If you use MSVC or
BORC: the rest is mostly standard C/C++ stuff, and should be clear to you.

Macintosh configuration

Similar to the Windows installations, the Mac installations are also either binary or
source code. The binary installations usually include an Installer, which
takes care of everything.

Source code installations often use the Metrowerks CodeWarrior development
environment, and include a project file for it. There have been many releases of this
popular development environment, so you should check which one is required.
CodeWarrior has no trouble with projects created with older releases (but may
tell you to use their conversion tool, see their installation notes for details).
The version numbering scheme of CodeWarrior can be confusing - just keep in mind that
all "Pro" releases are more recent than releases 7 .. 11.

Older versions of CodeWarrior
cannot read project files which were created by newer versions.

Chances are, if you're not into C or C++ programming in a substantial way, that
not everything you need is set up on your system to install extensions.
On the other hand, few extensions depend
on other extensions, because an extension tends to be designed to add
a specific set of features to Tcl, and often is quite independent of other extensions.

Note: this step may be needed in different places, since the fact that something is
missing might not show up until step 6 (compiling), step 7 (testing), or even occasionally
step 8 (installation). It is described here, before you are likely to need it,
so that you will recognize the symptoms if they do appear.

You may find out that your system does have a C compiler, but a weak one, or
that your extension is designed in C++ and you don't have a compiler for it.
Things can break at any level really. You might not have the "make" tool, or
you might find out that the extension you use is really an interface for another
product, which you do not have. The extension might do things which are known to
expose bugs in a required library, so you have to install a newer release of that first.
The list is endless.

Note that the above describes scenario's on Unix, but similar things can happen on
Windows and Macintosh.

Once you run into such a problem, you will have to resolve it by obtaining and installing
the missing piece (and on a bad day, that in turn may need yet something else).
Then you need to find a way to restart the installation -
which is not always as simple as it sounds.

Beware of some very subtle side-effects. An example of this is if your Unix system
has its own C compiler, and you just found out it is not suitable for the new extension.
So you might decide to install gcc ("GNU cc", a rather complex installation which
takes a considerable amount of time to complete, btw). Now, you might be tempted to resume
by typing "make", the second step in the "perfect world" steps listed before.

Unfortunately, this won't work.
The "./configure" step usually creates the makefile driving all compilations, and it assumes
(rightly so, at the time) that it had to use the original C compiler. So the next step
should be to restart from step 1? Unfortunately, this won't work either. First of all,
you now have two C compilers on your system, and not all configure scripts are
set up to prefer gcc... But an even nastier issue, is that you may have compiled result
files left over from the initial attempt. Mixing object files from different compilers,
and sometimes even from the same compiler using different settings, is a sure road to
disaster. As always, it may be a while before you run into problems - but you will...

The solution, is to fix the problem caused by the missing (or obsolete, or incompatible)
piece, and then try to start with a clean slate. This may take a lot more time, since
lengthy compilations will have to be repeated as well, even if there was nothing wrong with
them. For that reason, you may prefer to try continuing, and decide to restart at
the first sign that not everything is peachy. With a bit of luck, you will save time.
Just make sure to restart with a clean slate and try once more if you run into any type
of unexpected problem. Solving these types of problems requires a lot of logical
reasoning - the last thing you want is oversight, by failing to follow the instructions
exactly to make sure problems are not caused merely because you had to restart a
failed installation. This cannot be stressed enough:
when in doubt, start once more - from scratch!

So how does one restart from scratch? Well, on Unix, try "make distclean". In Windows,
try to do an uninstall using the Add/Remove Programs control panel). On Macintosh,
start the Installer you used to attempt an installation, and look for a "Remove" button.

Then go back to step 4: "Configure the extension build". It's not hard. It just takes time.

This is almost exclusively a Unix issue, since Windows and Mac software is usually available
in machine code format. Fortunately, on Unix, the autoconf and make tools take care of everything
in most situations. In Unix, there are a number of things that may break:

Since the make command is central to the build process, you may have
to get a better make if there are problems with the make tool currently
on your system. The accepted standard seems to be gmake ("GNU make").
I have never run into such a problem.

If you have gcc, then you can often force the configuration setup
to use it instead of any other compiler on your system. The convention is
to pass a special argument to the configure script:

% ./configure --enable-gcc

This is an example of a range of configuration options you may need to experiment with.
To find out what options are available, and what their defaults are, use:

% ./configure --help

Another one, is to force using dynamic linking (shared libraries) with:

% ./configure --enable-shared

If the configuration step complains about not being able to find the directory
where Tcl is installed, try:

% ./configure --with-tcl=/usr/local

or perhaps:

% ./configure --with-tcl=/usr

Note that using any of these options almost always implies that
you must restart the installation from scratch as described in section 5.

Don't be alarmed by a huge list of commands racing over your screen as you perform
this step. Autoconf, in a reasonably restrained manner, describes what it is doing
as it goes (it ought to complete within a minute or so). Make, on the other hand,
will throw an endless list of nearly incomprehensible "commands" at you - trying to
explain what it is doing no doubt, though most people really couldn't care less...

This is a good time to let the system run on its own and think of all the great
things you will be doing with the new extension. Some "makes" can take hours,
although in the case of simple extensions you can expect it to complete within minutes.

When done, if the last lines generated by make don't contain any alarming messages, you can
assume that everything is ok. Chances are good that the rough part of the ride is over.
If there are "errors", "aborts", or "failures", then this is the time and place where it
gets, eh... interesting. If the error messages don't help in figuring out what went
wrong, see the section at the end for suggestions on how to resolve the problem.

This step is frequently absent from extension packages. On Unix, if you type "make test",
you might see something like

make: *** No rule to make target `test'. Stop.

The extent of testing depends entirely on the extension writer, but even a very
simple test which verifies just a simple operation is very useful to convince people
that the whole process has now produced a working extension.

Note that tests are usually performed before an extension is installed in the
place where all extensions are supposed to be. This is far more flexible, because it
allows you to verify new extensions before they overwrite any older versions you may have.
If you are re-installing, make backup copies if you have any reason to doubt that the
new version will be successfully installed. The nightmare scenario is a failed installation
which replaces an older but working version. You end up with nothing working at all, and
may be forced to go back - find the older version, and repeat a complete installation of it.
There have been situations where this fallback option was no longer possible...

This is not always easy to set up, so some extension tests can only be done after
the new extensions is installed (step 8). The included documentation should make it
clear whether there any tests and how they can be performed.
Once the testing phase is over, you have essentially completed the process and successfully
created the machine code version of the new extension.

To be able to use the extension, the extension itself and all the files needed by it have
to be placed in very specific locations of your disk. There are up to four different types
of files:

The extension itself is a shared library which Tcl must be able to load

The extension may need supporting scripts to operate successfully

The extension may depend on other shared libraries and applications

The documentation files have to be in a standard place to be used as on-line help

Each platform uses its own approach to deal with these issues:

Unix installation

On Unix, the final installation process is usually automated and can then be performed
with the command:

% make install

Installation in system directories requires super-user privileges, so you will
often have to do this step as "root".

Windows installation

In Windows, DLL's will be found as long as they are either in the current directory
(at the time the library is loaded), or on the command search path.

Usually, each extension lives in its own directory and contains all the support files
and documentation. The standard installation path of Tcl itself is
C:\Program Files\Tcl\, and since it by default expects extensions in a "lib"
directory, version 8.9 of extension "Aaa" usually has to be installed as
C:\Program Files\Tcl\lib\Aaa8.9\.

Note the convention of appending the version number to the directory name.

Macintosh installation

On the Mac, extensions are either dynamic and implemented as code fragments, or
static and linked to create a new tclsh or wish binary.
Code fragments are normally given a file name ending in ".DLL" when built with
Metrowerks CodeWarrior, and
need to be stored in the Extensions folder inside the
System Folder (note that the name "Extensions" is coincidental: the Mac
supports all sorts of system "extensions", such as INITs).

When one of the above eight steps fails, you'll often get some sort of error
messages, such as "undefined external" or "can't find ...". The trouble is that
such messages hardly ever tell you what you need to do to fix them. Furthermore, it's usually
not obvious at all whether it was some oversight on your part, someone else's mistake or
carelessness,
or whether it's going to be easy to solve, or even whether it can be solved
(you might be trying to use an extension on a machine which no one has ever tried before).

What you need to do is rule out possible causes, one step at a time, in the assumption
that you or someone else will eventually arrive at the only sound explanation.
It's not easy... but for those who know enough about the underlying technology,
it's a matter of logical deduction.

Sometimes, there are several different problems, which can complicate things considerably.
Then again, most installation problems are likely to fail because of one silly little reason.
Well... they will look silly once solved, and once you realize how many steps must
have been going right to end up with a succesful installation!

Can't find Tcl [Unix, configure]
Many "configure" scripts expect to see Tcl installed in /usr/local (with tclsh in
/usr/local/bin, and the libs in /usr/local/lib). Else try specifying its location,
perhaps something like "./configure --with-tcl=/usr". If you don't know where
Tcl was installed, try a full disk search with "find / -name tclConfig.sh -print".
This file is in the lib directory, so if find reports "/usr/lib/tclConfig.sh", you
should pass just the "/usr" to configure.

Can't find <extension>
The extension is not in the expected location. Use "[puts $tcl_pkgPath]"
to see what the standard places are. Use "[puts $auto_path]" for
a list of where Tcl is currently looking (you can add more directories and retry).
Try an explicit "[load <path>/<extension>]" to verify the exact location.

Can't resolve symbol '...' [Unix, tclsh or wish]
This indicates that tclsh or wish is trying to load
an extension which contains unresolved symbols. On Linux, you can use
the "ldd" command to find out which shared libraries the extension references,
and which one could not be found.

Can't resolve symbol 'Tcl_...' (or 'Tk_...') [Unix, tclsh or wish]
Your Tcl installation was not compiled to use shared Tcl/Tk libraries. Look at
the tclsh or wish executable itself - if they are a few hundred kB or more, then they need to be
rebuilt. Reconfigure and rebuild Tcl itself using:
make distclean;
./configure --enable-shared;
make;
make install

Crashes... [Unix]
Shared libraries must often be compiled as "position independent code" (-fPIC).
For this reason, you usually cannot link any static libraries to an extension.

The moral of this story is: don't be paranoid, except during software installations.

Despite the fact than many installations often proceed flawlessly when you follow
instructions like these - or better still, the instructions included with the extension - there is
a wide range of problems that can arise. Most of these are minor, but at times it may be
quite puzzling what needs to be fixed, and how.

This document has described the entire process from a high-level perspective to give you an
idea of what is happening and why, along with a step-by-step approach you can use to get
many different types of extensions properly working on your system.

With thanks to Glenn Elmore and Andreas Kupries for several corrections and improvements.

NOTE: Although I cannot help with any specific extensions, if you have questions, comments, or
suggestions on how to further simplify the process or how to better describe it, please
let me know so I can make adjustments.