Improving security and performance for capability systems

This technical report is based on a dissertation submitted March 1988 by
the author for the degree of Doctor of Philosophy to the University of
Cambridge, Wolfson College.

Abstract

This dissertation examines two major limitations of capability systems:
an inability to support security policies that enforce confinement and a
reputation for relatively poor performance when compared with
non-capability systems.

The dissertation examines why conventional capability systems cannot
enforce confinement and proposes a new secure capability architecture,
called SCAP, in which confinement can be enforced. SCAP is based on the
earlier Cambridge Capability System, CAP. The dissertation shows how a
non-discretionary security policy can be implemented on the new
architecture, and how the new architecture can also be used to improve
traceability of access and revocation of access.

The dissertation also examines how capability systems are vulnerable to
discretionary Trojan horse attacks and proposes a defence based on rules
built into the command-language interpreter. System-wide garbage
collection, commonly used in most capability systems, is examined in the
light of the non-discretionary security policies and found to be
fundamentally insecure. The dissertation proposes alternative approaches
to storage management to provide at least some of the benefits of
system-wide garbage collection, but without the accompanying security
problems.

Performance of capability systems is improved by two major techniques.
First, the doctrine of programming generality is addressed as one major
cause of poor performance. Protection domains should be allocated only
for genuine security reasons, rather than at every subroutine boundary.
Compilers can better enforce modularity and good programming style
without adding the expense of security enforcement to every subroutine
call. Second, the ideas of reduced instruction set computers (RISC) can
be applied to capability systems to simplify the operations required.
The dissertation identifies a minimum set of hardware functions needed
to obtain good performance for a capability system. This set is much
smaller than previous research had indicated necessary.

A prototype implementation of some of the capability features is
described. The prototype was implemented on a re-microprogrammed
VAX-11/730 computer. The dissertation examines the performance and
software compatibility implications of the new capability architecture,
both in the context of conventional computers, such as the VAX, and in
the context of RISC processors.