Tcl is a programmable programming language. Almost every functionality
of the language is exported to the user through commands: conditionals,
creation of procedures, control structures, mathematical expressions,
all are implemented as commands, that take string as arguments.
Because the user can create new commands, or even replace core commands
with user defined ones, the language can be extended and modified
in a radical way. You can write commands implementing your own control
structures, a new way to deal with mathematical expressions, functionalities
that are closely realted to the program you are writing, or an object
oriented programming extension. So the expert Tcl programmer will
write a program in a new way: in the first stage he will specialize Tcl
to make the language better for the task, then he will write the program
using this new specialized language.

We already saw how to create Tcl procedures, but in order to create more
powerful procedures, more powerful commands are required, this commands
are eval, uplevel, upvar, and may others: some of this commands will
be explained in this chapter, with real examples on how to program
Tcl in Tcl in order to increase the power of the language.

9.1 Programs executing programs: the eval command

In the first chapter of this book, we already shown that Tcl has
a very dynamic nature. For instance you can put a command name in
a variable and call it like this:

% set a llength
llength
% $a {1 2 3}
3

You may even use interpolation to create the command name (or any
of its arguments) at runtime:

% [string range "Xllen" 1 end]gth {1 2 3}
3

Still there is something of very desiderable that we can't do without
the use of a command called eval, that's to evaluate a string as
a Tcl program. That's exactly what the eval command does: it
runs the Tcl script you pass as argument, and uses the return value of
the last command executed in that script as return value.
That's an example just to figure how it works:

In the first line we called eval with the string "set foo 10" as argument.
Eval will just evaluate it, and return the value returned by the script.
Then we set the variable myscript to a string representing a valid Tcl
command and execute it using eval. This is a very important operation
to do, immagine for example that you want to write a new Tcl control
structure called repeat. The goal is to avoid a for or while loop
when you want to just repeat a given Tcl script a number of times,
using instead something like "repeat 10 {lappend mylist foo}", to create
a list with 10 elements of value "foo". The second argument of the repeat
command is a script, and it is not possible at all to execute this
script in a loop without to use eval. This is a possible implementation
of the repeat command:

proc repeat {n script} {
while {[incr n -1] >= 0} {
eval $script
}
}

Cut & paste this in the tclsh to check if it's working well:

% repeat 4 {puts Hello}
Hello
Hello
Hello
Hello

Great, we just extended Tcl in Tcl. Actually we'll see in a moment that
this implementation of repeat is not very correct, but for know our goal is
to understand very well how eval works.
An important information about eval is that it can take multiple arguments,
they will be concatenated using the concat command, and the result of
the concatenation will be evaluated. So actually to write:

eval $foo $bar

is the same to call

eval [concat $foo $bar]

The concat command does a simple operation: it removes every space at
the right and at the left of every argument passed, and then concatenate
all the resulting strings using a single space as separator between
every string, returning the result obtained to the caller.
It's possibly simpler to show the behaviour using some example:

While concat performs just an operation on strings, you should notice that
if all its arguments are valid lists, the result is a single list that is the
concatenation of all the original lists. So concat can be used to concatenate
lists. Because eval concatenates the arguments like concat, we have to
study the concat behaviour very carefully when the arguments are pieces
of scripts so that can we can apply what we understand about concat in
order to use eval very well.

Let's start with an example, but first we need to recall the last version
of the + procedure we wrote talking about procedures with variable number
of arguments.

proc + {x args} {
foreach e $args {
set x [expr $x+$e]
}
return $x
}

Now suppose you have a list of integers, and you want to sum all this
integers using the + procedure. + is designed to take every integer
to sum as a separated argument, so we can't write something like "+ $mylist",
but it's possible to get the work done using eval:

% set mylist [list 1 2 3 4 5]
1 2 3 4 5
% eval + $mylist
15

It works, and the behaviour is explained recalling that eval concatenates
it's arguments like concat. We can call concat instead of eval, using
exactly the same arguments to check what's the resulting script that eval
will finally evaluate after the concatenation step:

The resulting script is "+ 1 2 3 4 5", and it is perfect because the +
procedure will be called with every list element as a different argument.
This behaviour is useful, but we want also a way to avoid it, that's, what
about if I want to use eval with an argument that should be treated as
an single` argument even if it contains spaces? Just passing the arguments
to eval will not work:

An error is returned, because [concat $a $b] will produce "puts Hello World",
so puts will be called with three arguments, instead of two like in
"puts {Hello World}". In order to fix the problem, we have to quote the
arguments using the list command. First of all we can try to
see the difference in the produced string calling concat directly:

As you can see, in the last line we used list in order to quote the command:
list will take care to use braces around arguments containing spaces or
other characters interpreted by Tcl in a special way.
Using or not using the list command to quote arguments makes we able to
select what argument of eval should be expanded like in the
case of the + procedure used to sum a list of numbers, and what should
be not. For example in the code:

eval $a [list $b $c $d] $e

if $a and $e contain spaces, the content will be interpreted as
more then one argument, while $b $c and $d will always be interpreted
as a single argument because of the quoting done by the list command.

Don't worry if all this is not very clear for now, it's a complex topic that you will
master after some pratice. For now let's introduce a new command that
will allow us to write a new, better implementation of repeat.

9.2 Breaking the rules with uplevel

In the previous section we wrote this implementation of the repeat command.

proc repeat {n script} {
while {[incr n -1] >= 0} {
eval $script
}
}

It works in some case, like if we use it to print four times "Hello World"
on the screen like this:

repeat 4 {puts "Hello World"}

But if you look at the implementation of repeat, there is something of
strange. The script argument, containing the body of the loop, is executed in the
context of the repeat procedure, not in the context of the caller.
In the case of a script like {puts "Hello World"} there are no problems
because this script can be executed in any context without troubles, it
does not refer to local variables, it's just a command with a constant argument.
Instead this will create troubles:

What happens is that the script "puts $a" is executed in the context
of the repeat procedure, where the $a variable does not exists.
What we need to fix this procedure, is a command very similar to eval
but able to run a script in the context of the caller. This command
is called uplevel. The uplevel command is exactly like eval, but it
takes as first argument the level where the script will be executed.
This is the command signature:

uplevel ?level? arg ?arg ...?

For instance, if level is 1, the code is executed one level back, in the
context of the caller procedure, if level is 2 the code is executed in
the context of the caller of the caller, and so on. Look at the following
example carefully:

The output of this program is "Tcl is a programmable programming language",
but it's important to realize why. The program creates three procedures,
a, b and c, and starts calling the procedure a.
The a procedure will create a local variable myvar with the
"Tcl is a ..." string as content, and will call b, that will call
c, that will execute the script "puts $myvar" in the context of the
caller of the caller (that's, the a procedure). Because in the context of a
the myvar variable exists and is set to a given value, the program will
print it, and exit.

Now what you need to change in the repeat procedure to make it working
is to replace eval with uplevel 1, and it will be sane.

This version will work as expected: you can't distinguish it from a core
command, you just have a new control structure to use when needed.
This new procedure of course will work when the last implementation used
to fail:

% set a 10
10
% repeat 2 {puts $a}
10
10

The only difference between repeat and other Tcl commands performing loops
is that it does not work with break and continue. Of course it's possible
to fix this problem too, and we will see how to do it in the chapter about
the error handling features of Tcl.

9.3 Passing variable names to procedures

Now that we have a tool like uplevel we can figure how it's possible to
implement procedures able to get variable names as arguments, and to
modify or use the content of this variables from the caller's context.
This is the case of commands like incr, append, lappend, foreach.
In some way this is the Tcl way to do pass by reference,
but instead of a reference what is passed is the name of a variable of
the caller.

Let's try to implement a procedure that given a variable name converts the
string stored in that variable in uppercase, setting the uppercase version of
the string as the new content of the variable.

Note that the implementation of toupper returns a null string, in order
to stress on the work the function does as side effect, and not on its
return value. You should be able to understand how toupper works,
but we will analyze it step by step.

Starting from the first line of the procedure:

set oldval [uplevel 1 [list set $varname]]

This command set the oldval variable to the value of the script
"set $varname" executed in the context of the caller. Do you remember
that the set command with only one argument returns the value of
the specified variable? But this variable lives in the context of the
caller so we need to use uplevel, and quote the script with list
because the value of $varname may contain spaces.

The next line of the procedure is just basic Tcl:

set newval [string toupper $oldval]

We use the string command to set an uppercase version of $oldval
as value of the newval variable. At this point, we are ready to
modify the variable with name $varname in the caller's context to
store the new uppercased value of the string:

uplevel 1 [list set $varname $newval]

This is very similar to the first line of the procedure, but here set
is called (again, on the caller's context thanks to uplevel) with
three arguments, the last being the new value to assign. Again list
is required in order to make sure that even if $varname or $newval
contain spaces, they will be handled as a single argument.

Finally the last line of the procedure returns the empty string.

This all works well... but there is a problem: it is a bit too much complex
to be comfortable, so Tcl has a command called upvar to make this simpler.
The following is the equivalent program using upvar instead of uplevel:

This looks simpler, but how it works? upvar is able to bind a variable
name living in the current procedure, with one living in a different
level. This is the procedure signature:

upvar ?level? otherVarmyVar ?otherVar myVar ...?

The level argument works exactly like the level argument of uplevel,
so a level value of 1 means to bind with a variable of living in the
context of the caller. In the above example, the code:

upvar 1 $varname var

means: bind the local variable name var, with the caller's variable
name $varname. Every time the toupper procedure will use the
variable var the effect will be to access the caller's variable
named $varname (the actual value of varname of course depends on
the value of the argument passed to the toupper procedure).

It's possible to specify more than one pair of variable names to upvar
like in:

upvar 1 $listname list $somename foo

in order to bind more variables with a single upvar command.

9.4 Mapping scripts to lists

Now that we are able to extend the language, we can write a procedure that
will prove to be very useful in a number of circumstances. This procedure
called map is used to map a script to a Tcl list. Every element of a list
is used as argument of a Tcl script, and the result is used to generate
a new list (composed of the resulting elements). For example mapping a
script that returns the square of its input to the list "1 2 3 4"
we will obtain the list "1 4 9 25", and so on. The following is a first
implementation of map, but we will write a better one in the chapter
about Functional programming with Tcl.

This map procedure is very similar to the foreach command,
but at every iteration, the return value of the script is appended to a
list that is returned by map at the end of the loop. For example in order
to compute the squares of the first 5 natural numbers you can write:

map x {1 2 3 4 5} {expr $x*$x}

The return value of this command is the list {1 4 9 16 25}. Another example
is to use map to turn a list of strings in the list of their lengths. This
time tclsh is used for an interactive session:

% set l {I will be translated into a list of length}
I will be translated into a list of length
% map x $l {string length $x}
1 4 2 10 4 1 4 2 6

It makes a lot of task simpler! Actually what map does is to encapsulate
in a procedure a piece of code that you otherwise will type many times,
that's something like this:

set result {}
foreach e $list {
lappend result [string length $e]
}

This explains why experienced programmers think that programming languages with
great extension capabilites are better: the programmer can look for patterns
in the code (symptoms of repetitive actions that the programmer is doing,
and that the language may do for him), and write a procedure to make a specific
task occurring frequently much simpler.

9.5 The rename command

In the first chapters we mentioned that using proc it is possible to create
procedures with names of already existing commands (both core commands or
used defined ones). Not only it's possible to overwrite an existing command
name, it is also possible to rename existing commands using the rename
command. The usage is very simple:

rename oldNamenewName

rename will change the name of the command oldName, into the new name
newName, with one exception: if the newName argument is an empty string,
the oldName command will be removed from Tcl at all. This feature may be
used in order to make the language more secure in critical contexts (removing
all the commands that are a potential security problem), but we will see
this better talking about Safe Interpreters in the next chapters.

The rename command enters in the picture traced by this chapter because
the ability to rename a command leads to the ability to wrapper existing
commands. For example you may like to wrapper the puts command in order to
act as the eval command when called with multiple arguments, concatenating
all the arguments using a space as separator and printing the result on the screen,
but at the same time, we need to take the original puts implementation under
some other name, because our new puts will need to call it to perform
the output. We can do all this using rename:

rename puts _puts
proc puts args {
_puts [join $args]
}

After the above code is executed, puts will work with multiple arguments:

Wrappering procedures may be interesting, for example you may like to extend
the proc functionalities in order to define the description of the
procedure as additional argument, and then write a command to get the
documentation from the command name, or you may wrapper set in order to
print debugging information about modified variables in a log file in
order to trace some bug, and so on.

9.6 Expanding lists into arguments in Tcl 8.5

Talking about eval we shown how it is possible to use this command
in order to perform expansion of the elements of a list as single
command arguments. In the example we used this feature and the +
procedure to compute the sum of a list of integers:

% set mylist [list 1 2 3 4 5]
1 2 3 4 5
% eval + $mylist
15

Starting from Tcl 8.5 there is a better way (both performance-wise and from
the point of view of the user) in order to perform argument expansion.
It's not a command, but new syntax. Basically prepending an argument
with the string {expand} has the result that the Tcl interpreter will expand
every element of the original argument interpreted as a Tcl list,
into a single argument. For instance the scripts:

{expand}{puts Hello}

and

set mylist [list puts Hello]
{expand}$mylist

are both equivalent to just

puts Hello

We can rewrite the sum of integers element of a list using {expand}
without to use eval:

+ {expand}$mylist

Every element of $mylist will be expanded as an argument of the + command.

The introduction of {expand} in the Tcl syntax was controversial, many
people didn't liked the syntax, or the idea to make the Tcl syntax more
complex (fortunately the idea of simplicity is very important among people of the
Tcl comunity). Strictly speaking it's possible to write every kind of
program without to use {expand} at all, but actually there are many situations
where argument expansion is needed, and eval is not the best tool for
argument expansion (the main goal of eval is different, it is to evaluate
Tcl scripts). I'm confident that after all the decision of the Tcl Core Team
to introduce the {expand} syntax into the language was a good one: now that Tcl 8.5
is near to be released the Tcl comunity is starting to appreciate how comfortable
and fast argument expansion can be.