Tmcdapi

TMCD Node Configuration API

TMCD Node Configuration API

"TMCD" is the most common term used to describe the Emulab node configuration
protocol. TMCD stands for the Testbed Master Control Daemon and is the name
of the server side daemon. TMCC (expansion left to the interested reader)
is the client-side application. So yes, technically the protocol should be
called TMCP, but it isn't; deal with it.

For the big picture of how TMCD fits into the whole configuration process,
refer to the client self-configuration document.
This document just describes the API for TMCD.

Description of TMCD Commands

bossinfo

A "meta" call that return the DNS name and IPv4 address of the "boss"
server to which to make all other TMCC calls. Obviously this info cannot be
obtained from boss! The boss info comes from (in order of preference):

value compiled into the tmcc binary

or:

value of BOSSNAME environment variable (only used in elabinelab)

contents of bossnode file in one of the directories: /etc/testbed, /etc/emulab, /etc/rc.d/testbed, /usr/local/etc/testbed, or /usr/local/etc/emulab.

set to the name of the host acting as the primary DNS for the node.

reboot

Deprecated. Returns nothing.
Used to provide notification to the server of a node reboot.
This transition is now detected and handled elsewhere (stated).

This call is not cached.

nodeid

Returns the Emulab canonical node ID (e.g., "pc40") for a node.
This is not the experiment's name for the node (e.g., "node1").

status

Return the project name (pid) and experiment name (eid) of containing
experiment (or "free") and the user-specified name of the node:

ALLOCATED=<pid>/<eid> NICKNAME=<vname>

This call is not cached.

ifconfig

Return info about IPv4 interfaces to configure, one line per interface.
Used for both standard interfaces and "virtual" interfaces. Interfaces
are identified on nodes by their MAC addresses as they have different
"names" under Linux, BSD or Windows and may even move around on the
same OS. Some interface types (currently just wireless interfaces)
may have additional info specified via INTERFACE_SETTING lines, one
line per key/value pair on each such interface. Virtual interfaces
have different info returned. Such interfaces can be type: veth
(Utah-local virtual ethernet interface), vlan (802.1q tagged vlan),
or alias (an IP alias on an existing interface).

accounts

Returns info about users and groups to setup on the node, one line per
user or group. For nodes which do not mount user home directories via
NFS (e.g., Windows, shared, ProtoGENI and remote nodes) returns one or
more SSH public keys to install in the local home directory. For nodes
running SFS, returns one or more SFS keys.

<mac0> and <mac1> are used to identify the physical interfaces which are
the endpoints of the link. The client runs a program called findif to map
a MAC address into an interface to configure. Identification is done in
this manner since different OSes have different names for interfaces
(e.g., "em0", "eth0") and even different versions of an OS might label
interfaces in different orders.

<pipe0> and <pipe1> identify the two directions of a link, with <delayN>,
<bwN> and <plrN> being the associated characteristics.

<link> is the name of the link as given in the NS file and is used to
identify the link in the delay-agent.

<queueN params> are the parameters associated with queuing which
pretty much directly translate into Dummynet configuration parameters.

<vnode0> and <vnode1> are the names of the nodes at the end points of the
link as given in the NS file.

The NOSHAPING parameter is not used by the delay agent. It is used for
link monitoring to indicate that a bridge with no pipes should be setup.

<mac> is used to identify the physical interface which corresponds to
the endpoint of the link on this node.

<type> is the type of the link or LAN, either "simplex" or "duplex".
This is used as an indication as to whether the reverse pipe needs to
be setup (duplex) or not (simplex).

<link> is the name of the link as given in the NS file and is used to
identify the link in the delay-agent.

<node> is the node receiving the info (us).

<IP> and <netmask> are no longer important. They were used to enable
endnode shaping on physical links that were multiplexed using IP aliasing.
These were used along with a local modification to IPFW to apply multiple
rules to an interface based on the network of the "next hop". We no longer
allow this (though the rules are still setup, see below) as it did not
completely work.

<pipe> and <rpipe> identify the two directions of a link, with <(r)delay>,
<(r)bw>, and <(r)plr> being the associated characteristics.

<queue params> are the parameters associated with queuing which
pretty much directly translate into Dummynet configuration parameters.

hostnames

Return host name info for all other hosts in the experiment, one line
per host. Info includes a canonical name, aliases, and the IP. The
canonical name is formed by concatenating the vname of the target host
and the vname of the link it is reached over.

NAME=<canon name> IP=<IP> ALIASES=<list of alias names>

Used to generate the local /etc/hosts file. Note that these name to IP
mappings are for experimental interfaces and not the control network.

NOTE: this call has been deprecated in favor of individual hosts using the
topomap information to generate their own hosts file.
This call should only be used if that map does not exist.

This call is not cached.

rpms

Returns a list of RPM packages to download and install on the node, one
line per RPM package:

RPM=<RPM-path>

The returned RPM path is an absolute path on the server.
The RPM itself can be fetched either across NFS (if on a node-shared
filesystem) of via HTTPS from the Emulab boss (see the os/install-rpm script).

deltas

tarballs

Return a list of tarballs to download and install on the node,
one line per tarball.

DIR=<local-dir> TARBALL=<tarball-path>

The returned TARBALL path is an absolute path on the server.
DIR is the directory on the node in which tar should be run.
The tarball itself can be fetched either across NFS (if on a node-shared
filesystem) of via HTTPS from the Emulab boss (see the os/install-tarfile
script).

startupcmd

Returns a command the node should execute once it has finished booting,
as well as the UNIX uid that should be used to the command.

CMD='<command line to execute>' UID=<unix-uid>

TMCD imposes no restrictions on what CMD can look like other than it
cannot contain embedded single-quotes. The current convention (implemented
in the startcmd script) is that it is a single command line invoked via
Perl exec().

startstat

Used to return an exit status code (from the start command) back to TMCD
on boss. The returned value (interpreted as a decimal number) is placed
in the database.

This call is not cached.

startstatus

An alias for startstat. The one and only alias in TMCD.
The reason behind this is lost in the sands of time...

This call is not cached.

ready

Used to return an indication that a node is "ready." TMCD just stores the
value (0 or 1) in the database. This is a user-only mechanism, a node's
"ready" value has no meaning to the server-side of Emulab. Used in
conjunction with the readycount command.

This call is not cached.

readycount

Returns the number of nodes in the calling node's experiment that have
reported "readyness" via the ready command, and the total number of
nodes in the experiment.

READY=<nodes-ready> TOTAL=<nodes-total>

NOTE: Use of the centralized, heavyweight ready/readycount
polling mechanism is discouraged via threats of bodily harm. It was an
exquisitely bad idea since it just screams "Put me in a loop and call me
as fast as you can from all 1000 of your nodes!"
Use the
emulab-sync mechanism instead.

This call is not cached.

mounts

Returns remote filesystems that should be mounted. Currently only NFS mounts
are supported on local nodes.

REMOTE=<remote-host:remote-path> LOCAL=<local-path>

Where REMOTE is in the standard NFS syntax; e.g., fs.emulab.net:/share.
The information returned varies depending on the node type:

information; i.e., the root directory for all projects or for all users.

Virtual nodes do not get any mount information. Their mounts are

controlled and setup by their host.

Remote nodes do not get any mount information.

The typical set of exported filesystems included: /share, /proj/<project>,
/group/<group> (if experiment is in a sub-group of a project),
/scratch/<project> (if the facility supports a /scratch filesystem),
and a set of users directories. Exported user directories are those for
all users in the experiment's project (or sub-group). The set of user
directory exports will not mix admin and non-admin users. For example,
if the user set for a project includes both admin and non-admin users but
an experiment is swapped in by a non-admin user, only the non-admin
directories will be exported.

At one time, SFS mounts were supported for both local and remote nodes.
This information may still be returned, but the client-side support has
atrophied. SFS mount information would be returned if the TMCD server and
the client both had an SFS ID. These were a subset of the mounts exported
via NFS.

SFS REMOTE=<remote-SFS-id:remote-path> LOCAL=<local-path>

Where REMOTE was in the standard SFS syntax.

sfshostid

Used by a node to report its SFS ID. Does nothing if SFS is not in use.
Otherwise, the server uses the information to SFS symlinks of the form:

reset

Forces stated to reset any next\_pxe\_boot\_* and next\_boot\_* fields.
No output is returned to the client.

This call is not cached.

routing

Return IPv4 routing information for this node in the experimental toplogy.
Returns nothing if Emulab-supplied routing is not enabled. Otherwise
returns a line defining the form of routing desired:

ROUTERTYPE=(static|static-ddijk|static-old|manual|ospf)

where "ospf" means to run a dynamic routing daemon,
"manual" means construct routes from the provided ROUTE lines
(described below),
and ""static*"" mean using pre-computed static routes.
Both "static" and "static-ddijk" require that the node compute
routes from the information returned by the TMCD
topomap call.
"static-old" is deprecated and means that the complete set of routes
was computed on the Emulab server and is returned in ROUTE lines.

Any of the ROUTERTYPEs may return one or more ROUTE lines, describing
individual routes. These should be included in addition to whatever
the node itself might compute.

DEST identifies the destination IP address that the route is for.
If DESTTYPE is "net", then DESTMASK is the network mask for the route
(otherwise it is implied to be 255.255.255.255).
NEXTHOP is the IP address of the next hop router.
COST is an integer values passed straight through (uninterpreted) from
the user (i.e., the NS file). The client script can use this or ignore
it as it sees fit.
SRC identifies (via an IP address) the interface to which the route applies.

trafgens

Returns traffic generator agent information for a node.
Returns one of more lines of the form:

TRAFGEN is the user-given name of the traffic agent as specified in the
NS file.
MYNAME and MYPORT are the IP address and port number on which the agent
will be sending or receiving traffic.
Traffic generators are paired, there is always a source and a sink.
ROLE determines which end of a pair is on this node, and PEERNAME and
PEERPORT identify the other half.
PROTO is the IP protocol of the traffic.
GENERATOR says which program is generating the traffic.
The default traffic generator is tg, but at one
point we also supported traffic generation out of the
NS simulator,
which supported more sophisticated traffic models.

nseconfigs

Returns nseconfigs info. Raw data is pulled from db and returned to
client as-is. This request is not allowed via UDP.

state

Client passes in new node state, TMCD passes this on to stated to update the node's state machine.

This call is not cached.

creator

Returns the username of the experiment creator.

tunnels

Returns information about tunnel endpoints to create on a node.
Returns one or more lines of the form:

which describe attributes of the tunnel endpoints identified by TUNNEL and
MEMBER. The attributes sent depend on the "style" (KEY='tunnel_style')
of the tunnel. For "vtun" style tunnels (VALUE='vtun'), attributes include:

sdparams

where
"reg" is the "regular" reporting interval (used when a node is active),
"agg" is the "aggressive" reporting interval (used when a node is idle),
"load" is the load average threshold below which a node is considered idle,
"expt" is the packets-per-interval rate (either input or output) below

which an experimental interface is considered idle, and

"ctl" is the packets-per-interval rate (output only) below which the

the control interface is considered idle.

NOTE: the returned values are currently hardcoded into tmcd!

programs

Returns a list of agents to run, one line per agent, preceeded by a line
listing the user to run the agent as:

where UID is the local node user which the program agent should be run as,
AGENT is the name through which the event system identifies the program,
DIR is where to run the program from,
TIMEOUT is how long to wait in seconds for the agent to exit,
EXPECTED_EXIT_CODE is the "normal" exit status to expect from the program,
and COMMAND is the command line to run.

syncserver

where SERVER is a DNS name of the form <vname>.<eid>.<pid>.emulab.net
which will resolve to the control net interface of the experiment node
acting as the sync-server for a node's current experiment.
ISSERVER indicates whether the caller is the sync-server node.

keyhash

Returns a per-experiment secret key used by nodes to authenticate with the
Emulab webserver when requesting tarballs or RPMs.

KEYHASH HASH='<hex-string>'

Remote nodes issuing this request will get an error if not using an
SSL-encrypted connection.

eventkey

Returns a per-experiment secret key used by nodes to authenticate with the
Emulab event system.

EVENTKEY KEY='<hex-string>'

Remote nodes issuing this request will get an error if not using an
SSL-encrypted connection.

fullconfig

Return all configuration data for requesting node. The data returned
depends on node type (physical or virtual) and the purpose of the node
in the experiment (delay node, experimental node, etc).

All nodes will receive the configuration data for each of the following
TMCD commands.

nodeid

ifconfig

accounts

delay

linkdelay

rpms

tarballs

startupcmd

mounts

routing

trafgens

nseconfigs

creator

tunnels

programs

syncserver

keyhash

eventkey

watchdoginfo

firewallinfo

userenv

tiptunnels

traceinfo

motelog

Physical nodes will also receive this data:

vnodelist

subnodelist

ipodinfo

ntpinfo

sdparams

routelist

role

Virtual nodes will also get this data:

jailconfig

plabconfig

This call is not cached (DUH!)

routelist

Return info for each route for all virtual nodes mapped to the requesting physical node

virtual node

router type

source IP

destination IP

route type (host or network)

mask

gateway

cost

role

return the role of the node

rusage

Client sends the load average for the last 1, 5, and 15 minutes plus the
amount of disk space currently used. The server returns a value
indicating whether or not account data needs to be updated.

If the requesting node is a planetlab node, the up/down data is stored
in a file on the server.

The first form (TYPE=none) is returned to all nodes in an experiment for
which there is no control network firewall.

The second form (TYPE=remote) is returned to nodes behind the firewall
in a firewalled experiment. FWIP is the IP address of the firewall node
for this experiment.

The final form is returned to the firewall node in a firewalled experiment.
TYPE indicates the technology used to implement the firewall (currently,
there is only "ipfw2-vlan"). STYLE is one of "open", "basic", "windows",
"closed", or "elabinelab" and indicates the Emulab rule set being used.
IN_IF/IN_VLAN are the "inside" interface MAC address and switch VLAN number,
used to identify and configure inward-facing network interface.
OUT_IF/OUT_VLAN are the "outside" interface MAC address and switch VLAN number,
used to identify and configure the outward-facing network interface.
There will be zero or more VAR/VALUE lines indicating variables that
need to be substituted in firewall rules.
There will be zero or more RULENO/RULE lines indicating the actual
technology-specific firewall rules.
Finally, there will be zero or more HOST/CNET* lines giving host name, IP
address, and interface MAC address information for each host inside the
firewall.

hostinfo

Client sends info about itself to store in the database. Currently,
this only supports the version of the boot CD for CD-booted systems.
Expects CD version as a string no longer than 31 characters from the set
[a-zA-Z0-9-].

This call is not cached.

emulabconfig

return config for an inner emulab
FIXME

This call is not cached.

eplabconfig

return config for an inner planetlab
FIXME

This call is not cached.

localize

Returns "site localization" information for the node.
This is intended to make it easier for different Emulab sitesa to share
OS images without manual customization. However, we are currently a
looooonnnng way from that!
Currently, site localization consists of just the SSH public key for the
root user on boss.

ROOTPUBKEY='<root-pubkey>'

rootpswd

Returns a random password for root user in crypted format. Generates the password the first time it's called. Remote nodes must be using an SSL-encrypted connection.

booterrno

bootlog

battery

Client sends battery capacity and voltage, both floats. Nothing returned to client.

This call is not cached.

topomap

Returns a binary blob of compressed (gzipped) experiment topology information.
When uncompressed, the data include basic connectivity information for nodes
and links in an experiment.

This call is an abomination on top of an already hacky protocol.
Along with ltmap and
ltpmap, form
the unholy trio of unbounded binary blob returning TMCD calls.
This call should only be made using TCP, though this is not checked.

This call is not cached.

userenv

Returns list of user environment variables in name=value format.

tiptunnels

where VNODE is the name used to connect to the console line,
SERVER is the name of the console proxy machine,
PORT is the port to use to contact the proxy,
KEYLEN is the length of KEY, the secret presented to the proxy to connect.

relayconfig

Returns servername-port number mappings for tip servers: FIXME

Type

Server

Port

traceinfo

Returns information about links that should be "traced" (packets captured).
There will be one or more lines of tcpdump-like parameters of the form:

where LINKNAME is the user-given (NS file) name of the link or lan,
used together with VNODE, the virtual name of the node, to uniquely identify
a network endpoint to the event system.
IDX is a globally unique index used to create a unique TCP port number.
MAC0, MAC1 and VNODE_MAC are local node interface MAC addresses
which identify interfaces to trace.
TRACE_TYPE tells whether to collect entire packets, headers or just a
summary.
TRACE_EXPR is a libpcap expression for filtering packets.
If set, TRACE_SNAPLEN specifies the amount of each packet to capture.

ltmap

Returns a binary blob of compressed (gzipped) experiment topology information.
When uncompressed, the data include information about the logical
characteristics of nodes (e.g., user-specified DNS name)
and links (e.g., bandwidth and latency) in an experiment.
Used by linktest.

This call is an abomination on top of an already hacky protocol.
Along with topomap and
ltpmap, form
the unholy trio of unbounded binary blob returning TMCD calls.
This call should only be made using TCP, though this is not checked.

This call is not cached.

ltpmap

Returns a binary blob of compressed (gzipped) experiment topology information.
When uncompressed, the data include information about the physical
characteristics of nodes (e.g., OS image on disk)
and links (e.g., MAC addresses of end points)
in an experiment. Used by linktest.

This call is an abomination on top of an already hacky protocol.
Along with topomap and
ltmap, form
the unholy trio of unbounded binary blob returning TMCD calls.
This call should only be made using TCP, though this is not checked.

This call is not cached.

elvindport

Client sends elvind port. Nothing returned to client.

This call is not cached.

plabeventkeys

Returns all event keys on plab node to service slice. FIXME

Project ID

Experiment ID

Key

This call is not cached.

intfcmap

Returns a map of all PC nodes to their MAC addresses.

This call is not cached.

motelog

Return motelog info for this node:

MOTELOGID=<id> CLASSFILE=<file> SPECFILE=<file>

Awaiting arrival of the clue stick.

portregister

client specifies service and port to register, stored in db

if client specifies only service, look up port and return port and
nodeid to client Remote nodes must be using an SSL-encrypted connection.

This call is not cached.

bootwhat

Returns bootinfo information

Boot method

PART (local partition or MBR)

SYSID (local partition by partition type)

MFS (load MFS from network)

Optional commandline to pass to OS bootloader or kernel

This call is not cached.

tpmblob

Returns the TPM-encrypted private key associated with the public key
in that certificate.

BLOBHEX=<encrypted-key>

BLOBHEX is the ascii-hex encoded key.

tpmpubkey

Returns an X509 certificate uniquely identifying the node.

TPMPUB=<X509-cert>

tpmdummy

Returns nothing. A test call for TPM-attested identification.
Fails if the node is not TPM-attested.