Masterfrisbeeserver

Configuration info for the frisbee master server.

Configuration info for the frisbee master server.

The frisbee master server
runs on a known port and accepts requests to upload and download "images."
Images may be specially prepared, highly-compressed imagezip .ndz images or
just regular files. The master server authenticates the request and forks
a frisbeed (to download) or image collector (to upload) process to handle
the rest. By being aware of all activity, it can enforce aggregate resource
usage limits if necessary.

Scenarios.

Emulab

A master frisbee server runs on boss. It is configured so that it
uses the Emulab "config file" backend, which means it extracts configuration
info on demand from the Emulab DB rather than using a literal config file
that we constantly modify (and HUP the server). This may be easier said
than done as we have to map DB table info into something coherent for the
model below (this is why I call it an Emulab backend and not a general
database backend; the latter would just use a DB to store config info in
master frisbee format).

For image distribution, frisbeelauncher would go away. os_load, the
elabinelab and subboss interfaces would no longer fire it off. tmcd would
just return an image-id to clients via the "loadinfo" call. The frisbee
client, along with inner bosses and subbosses, will request images by name
using the GET interface. At elabinelab or subboss setup time, we will have
to create the appropriate DB state to control which images those entities
have access to.

For image collection, boss should just have to boot nodes in the admin MFS
along with the current start command. That start command currently gets
all the info it needs on the command line and that can remain the same.
The info now includes a server and image-id rather than an NFS-accessible
pathname. The create image path will need to be modified so that it ensures
that the frisbee master server allows the image to be uploaded from the node
in question.

Standalone

Here we allow anyone to download the frisbee software along with a CD- or
dongle-boot image that contains a bootable Linux/BSD with frisbee installed.
They build the server side on Linux/BSD and configure the text mfrisbee.conf
file for the master server. By default the config file would have to be
useful "out of the box", we'll have to figure out what that means. At the
very least it would limit up/downloads to/from a certain local subtree.
It might not require any authentication or be set to IP authentication of
nodes for a particular subnet. We will need to have some method for creating
per-user or per-node certificates, keys, or passwords for stronger
authentication.

For image distribution, they would boot from the dongle/CD and initially
type a command line to up/download images. One could imagine a GUI you
run on your machine as well that provides a list of images to load or gives
a form to fill out for upload or whatever. Something else to think about
is what we can do to ease automation of large-scale disk loading. One thing
would be to provide a network boot process rather than CD/dongle. Maybe the
GET interface is then modified to allow a null request where the server will
return a "default" image to load or some other indicator if it should not
load its disk at all. The latter would tell the process to boot from disk
instead of load the disk.

For image collection, it is pretty much the same: boot a CD/dongle and
manually run an imagezip command, someday a GUI.

Comments

So have we really made frisbee more standalone? Not really for the initial
disk-loading case, but we have made it slightly easier (or maybe faster)
for saving images. More importantly though, we have made it easier to use
frisbee to scalably download arbitrary files in the regular OS context.
You can just fire off a frisbee command on any number of nodes and it will
work; no wrapping a file as an image, picking a MC addr/port, starting a
server and then running a bunch of frisbees on nodes.

Authentication

What authentication makes sense for frisbee (downloading images)?
Note first of all that authentication does not imply any on-the-wire
encryption or integrity protection. (You can, however, create encrypted
and/or integrity protected images with imagezip which can then be transferred
with frisbee.) Since frisbee uses a UDP multicast protocol, it does not
prevent unauthorized parties from downloading an image; i.e., joining the
multicast group. So the best we do here, without modifying the frisbee
protocol, is to have the client/server check the IP address of the sender
and filter based on that. So the only meaningful per-user authentication
or key- or certificate-based authentication we can do for download is in
the master server. This would allow us to do something like a secure
transfer of the image encryption or signing key. It also allows us to
limit the number of frisbeed processes that get spawned for feeding images
(since only authenticated users/nodes would be able to do a GET which in
turn fires off the frisbeed process) and thus it can help control resource
consumption on the server.

For uploading images, we could do a lot better since we are just going to
use a unicast TCP stream. So we can establish an SSH/SSL connection if
desired.

This asymmetry bothers me, but I don't know what to say.

Anyway, part of the master server configuration will be the type(s) of
authentication it will support. As we have seen, these might be different
for upload vs. download. The simplest type is "none" which will allow
any request to be serviced. The server will run with limited privilege
of course to ensure that frisbeed doesn't offer up or clobber any protected
system files. "IP" will validate based on the sending IP address of the
request. This configuration is suitable for environments such as Emulab
where host-based authentication is what we need and IP-spoofing is prevented.
"SSL" will allow host or user based authentication. I suppose this could
allow per-image authentication as well. We could have a "password" type
requiring requests to provide a password to allow per user/host/image access.

For Emulab, we just need "IP". For standalone, we would also want a "none"
and maybe some form of key/cert/password auth along with "IP".

Image IDs

What exactly is an image ID? The only criteria here is that it be globally
unique. We could do a standard UUID, but I would prefer to keep it to
something fairly human readable. So for now just an ASCII string. Note
that these are just used by the client to name the image at the server.
The server will map that ID into a UUID or path or Venti hash or whatever.

For Emulab an ID will just be a "pid/eid" style string. For standalone,
it is also just a string, with it being up to the user to figure out a
unique naming convention.

Request types

GET <image-id> <options>

Determine if the caller can access the image identified by <image-id>
with the desired <options>. If so a frisbeed is spawned and the necessary
information returned for the caller to use it. <options> specifies desired
characteristics, which the server can grant or not:

To use this, the frisbee client will have a command line option where an
image-id is specified rather than the current address/port info. It will
use the GET call to find out the addr/port information from its master server.
This eliminates the need for an out-of-band channel for communicating this
info.

For return values we must have method, address and maybe frisbee-options
so I can mess with client-side tuning. authentication/encryption will be
important when I get the "secure frisbee" changes merged in.

Note that there are keys in both directions. One could imagine that the
creator of an image might want to choose the keys used, but I am not sure
where these would be put. I view the config file as RO to the master
server (even if it is a DB) so I don't see it changing that. It could
store them in files alongside the image I suppose. Alternatively, the
user could let the master server choose them and they would be returned
as shown. Still have to store them somewhere though...needs more thought.

To use this command, imagezip will have an option, taking a server name and
image-id, allowing it to upload its image rather than storing it locally.
After getting back the reply, it will either open a new connection or start
sending the image data on the current connection.

We might also want some sort of "imageupload" that will upload an existing
image rather than the imagezip style create-and-stream-it.

LIST

Return a list of images that the caller (user or node) has access to.
Includes information like: sizes of images, current IP info for running
servers, and max download bandwidth.

"get-options" is a hack to address misc. things like server timeouts
and multicast keepalives. They could each have their own option name,
but really, who cares.

Authentication section

authtype <node|user> <upload|download|updown> <none|SSL|IP|any>

Global option to specify how authentication is done. Could be multiple
lines to control authentication by node or user. There can be different
types for upload vs. download of images. Type "none" will allow matching
only against the "any" <host-id> or the "anybody" <user-id> in the
authentication rules below. "SSL" will accept only SSL connections
with certs for users or nodes. "IP" will tell the master to only match
host rules which match the sender of the request (useful for Emulab where
IP-spoofing is prevented).

host <host-id> <permissions> <image-id> ...

Node <host-id> has the specified permissions on the indicated images.
<host-id> can be a name or an addr/mask.
Permissions indicate which of the above operations is permitted (or ALL).

user <user-id> <permissions> <image-id> ...

User <user-id> has the specified permissions on the indicated images.
Permissions indicate which of the above operations is permitted (or ALL).

Resource use section

host <host-id>
image-size=<bytes> # max bytes for an upload of any one image from host
total-size=<bytes> # max bytes for all uploads from host
image-rate=<bytes/s> # max xfer rate for any one image from host
total-rate=<bytes/s> # max aggregate xfer rate for host

user <user-id>
image-size=<bytes> # max bytes for an upload of any one image from user
total-size=<bytes> # max bytes for all uploads from user
image-rate=<bytes/s> # max xfer rate for any one image from user
total-rate=<bytes/s> # max aggregate xfer rate for user

Example configurations

(I'm using a Bacula-like syntax in the following, but we might want to go
with a more XML-ish syntax like Apache's.)

Emulab

For Emulab, the real configuration would be in the Emulab DB, but here is what
it would map to as a standalone configuration. Emulab projects (groups)
control what images can be placed on what node. Users create experiments
which belong to a particular project, and those experiments contain nodes.
Thus at any time, a particular node has access to a particular set of images.
Since nodes have fixed control network IP addresses, at the end of the day,
image access can be controlled by IP address.

So in theory, the Emulab master frisbee config file is a set of "host"
authentication lines which which contain the current set of standard and
project-specific images that can be loaded on that host.

Clearly, if we were going to write this in a text format, we would want some
form of aliasing and/or wildcarding. For example an IP alias that would allow
you do define a name to associate with a group of IP addresses:

But what about uploading of new images? As shown, the imageid for a new image
would have to already exist in order to allow upload. In reality, the Emulab
config which is resident in the DB would do just that. We would dynamically
create the imageid before firing off the process that would do the upload.

But this is not particularly satisfying. Even in Emulab we might want users
to be able to upload arbitrary experiment-related files via this mechanism.
So maybe augmenting with:

Note that it is superficially weird to authenticate based on node IP address
when allowing upload of experiment (user) files. This is okay in Emulab
because at any time a node is allocated to a particular experiment (user)
and we dynamically change the config when the mapping changes.

I'm going to ignore the "Resource Usage" section for now since we can get away
without it, I haven't thought much about it, and I don't know how to implement
some of the features (e.g., aggregate BW limiting). We can use the images
section "get-options" option to specify everything we do today.

An imageid can be up to 256 bytes (large enough to hold an ASCII-encoded
SHA-1024 hash). Right now the only supported method is multicast.

The client uses the -S option to specify the IP address of the master server
and the -F option to specify an imageid. If no error is returned, the image
is downloaded using the returned addr/port. If -Q is used in place of -F,
then the client makes a "status only" call getting back info about whether
the named image is accessible to the client and whether a server is currently
running.

On the server side, the new master server (mserver.c) has an Emulab
configuration "backend" that supports host-based authentication.
The IP address of the caller is mapped to a node_id/pid/gid/eid combo
that is used to determine access. On a request, the specified imageid is
treated either as a pathname (if it starts with '/') or an image identifier
of the form "<pid>/<imagename>". If it is a pathname, we check to make
sure that pathname (after running through "realpath") is contained in one
of the directories accessible to that node in its current experiment context;
i.e., /share, /proj/<pid>, /groups/<pid>/<gid>, or /users/<swapper-uid>.
If it is an image identifier, the DB is queried to ensure that access is
allowed to that image; i.e., it must be "global" or in the appropriate
project/group.

The master server forks a frisbeed for each valid request, if one is not
already running. The multicast address selection is still based on the
emulab_indicies.frisbee_index field, but the address/port/server info is no
longer stored in the frisbee_blobs table (frisbee_pid, load_address,
load_busy are not set).

Note that this is not yet integrated. Further work is required to eliminate
frisbeelauncher.

Hierarchy (inner bosses, subbosses).

I have not thought this through yet.

Interaction with Venti and a new imaging subsystem.

Think about it. For Venti, this is a backend storage issue; i.e., having
a modular interface for fetching and storing image data and having some
URI-like naming convention for files so that we can choose the correct
storage interface. How this all integrates with the bigger picture of
a new imaging subsystem is still TBD.

Interaction with the trusted disk-loading system (TDLS).

Think about it. This might affect authentication if we wanted to use
the TPM-protected identity key for a node. Also, key exchange (for the
image signing and encryption keys) will need to be done using the node key.