Now www.xbox.com
is exhibiting broken AAAA behavior, and it looks
like an F5 "GTM" appliance may be to blame. www.xbox.com
is a CNAME for www.gtm.xbox.com, which is itself a CNAME for
wildcard.xbox.com-c.edgekey.net (which is in turn a CNAME for another
CNAME, but that's not important here). The nameservers for gtm.xbox.com
({ns1-bn, ns1-qy, ns2-bn, ns2-qy}.gtm.xbox.com), contrary to the DNS
spec, do not return the CNAME record when queried for the AAAA record
for www.gtm.xbox.com:

Consequentially, any IPv6-only host attempting to resolve www.xbox.com will fail,
even though www.xbox.com has IPv6 connectivity and ultimately (once you follow 4 CNAMEs) has an AAAA record.
You can witness this by
running ping6 www.xbox.com from a Linux box (but flush your DNS caches first).
Fortunately, IPv6-only hosts are rare, and dual-stack or IPv4-only hosts won't
have a problem resolving www.xbox.com because they'll try resolving the
A record. Nevertheless, this buggy behavior is extremely
troubling, especially when the bug is with such simple logic (it doesn't
matter what kind of record is being requested: if there's a CNAME, you return it),
and is bound to cause headaches during the IPv6 transition.

I can't tell for sure what DNS implementation is powering
the gtm.xbox.com nameservers, but considering that "GTM" stands for "Global
Traffic Manager," F5's DNS appliance product, which has a history of AAAA
record bugginess, I have a hunch...

In my previous blog post,
I discussed the unique way in which titus,
my high-security TLS proxy server, isolates the TLS private key in a
separate process to protect it against Heartbleed-like vulnerabilities in
OpenSSL. In this blog post, I will discuss the other isolation techniques
used by titus to guard against OpenSSL vulnerabilities.

A separate process for every connection

The most basic isolation performed by titus is using a new and dedicated process for every TLS
connection. This ensures that a vulnerability in OpenSSL can't be used to compromise
the memory of another connection. Although most of the attention around Heartbleed focused on
extracting the private key, Heartbleed also exposed other sensitive information, such as user
passwords contained in buffers from other connections.
By giving each TLS connection a new and dedicated process, titus confines an attacker to accessing
memory related to only his own connection. Such memory would contain at most the session key material
for the connection and buffers from previous packets, which is information the attacker already
knows.

Currently titus forks but does not call execve(). This simplifies the code
greatly, but it means that the child process has access to all the
memory of the parent process at the time of the fork. Therefore, titus
is careful to avoid loading anything sensitive into the parent's memory.
In particular, the private key is never loaded by the parent process.

However, some low-grade sensitive information may be loaded into the parent's
memory before forking. For instance, the parent process calls getpwnam() during initialization,
so the contents of /etc/passwd may persist in memory and be accessible by child processes.
A future version of titus should probably call execve() to launch the child process so it
starts off with a clean slate.

Another important detail is that titus must reinitialize OpenSSL's random
number generator by calling RAND_poll() in the child process after forking. Failure
to do so could result in multiple children generating the same random numbers, which would
have catastrophic consequences.

Privilege separation

To protect against arbitrary code execution vulnerabilities, titus runs
as dedicated non-root users. Currently two users are used: one for
running the processes that talk to the network, and another for the
processes that hold the private key.

Using the same user for all connections has security implications.
The most serious problem is that, by default, users can use the ptrace()
system call to access the memory of other processes running as the same
user. To prevent this, titus disables ptracing using the PR_SET_DUMPABLE
option to the prctl() syscall. This is an imperfect security measure:
it's Linux-specific, and doesn't prevent attackers from disrupting other
processes by sending signals.

Ultimately, titus should use a separate user for every concurrent connection.
Modern Unix systems use 32 bit UIDs, making it completely feasible to
allocate a range of UIDs to be used by titus, provided that titus reuses
UIDs for future connections. To reuse a UID securely, titus would need to
first kill off any latent process owned by that user. Otherwise, an attacker
could fork a process that lies in wait until the UID is reused.
Unfortunately, Unix provides no airtight way to kill all processes owned by a user.
It may be necessary to leverage cgroups or PID namespaces, which are
unfortunately Linux-specific.

Filesystem isolation

Finally, in order to reduce the attack surface even further, titus chroots into
an empty, unwritable directory. Thus, an attacker who can execute arbitrary
code is unable to read sensitive files, attack special files or setuid binaries,
or download rootkits to the filesystem. Note that titus chroots after reseeding
OpenSSL's RNG, so it's not necessary include /dev/urandom in the chroot directory.

Future enhancements

In addition to the enhancements mentioned above, I'd like to investigate using seccomp
filtering to limit the system calls which titus is allowed to execute. Limiting titus to
a minimal set of syscalls would reduce the attack surface on the kernel, preventing an
attacker from breaking out of the sandbox if there's a kernel vulnerability in a particular
syscall.

I'd also like to investigate network and process namespaces. Network namespaces would
isolate titus from the network, preventing attackers from launching attacks on systems
on your internal network or on the Internet. Process namespaces would provide an added
layer of isolation and make it easy to kill off latent processes when a connection ends.

Why titus?

The TLS protocol is incredibly complicated, which makes TLS implementations
necessarily complex, which makes them inevitably prone to security vulnerabilities. If you're
building a simple server application that needs to talk TLS, the complexity of the TLS implementation is
going to dwarf the complexity of your own application. Even if your own code is securely
written and short and simple enough to be easily audited, your application may nevertheless be vulnerable if you
link with a TLS implementation. Titus provides a way to isolate the TLS implementation, so its complexity
doesn't affect the security of your own application.