Jonathan Bennett's new Android
'Fwknop2' client is ready for prime time. It is available now in the
Google Play store as well as on
F-Droid,
and Jonathan put together a nice video demonstration of Fwknop2 being used to access
SSHD on a router running OpenWRT. Also demonstrated is the usage of NAT to transparently
access SSHD running on a system behind the router. This illustrates the ability
fwknopd offers for creating inbound NAT rules for external SPA clients - something that is,
to my knowledge, unique to the fwknop in the world of SPA software. Finally, in an
innovative twist, the Fwknop2 client can read encryption and authentication keys from
a QR code via the phone's camera. This simplifies the task of getting longer keys
- particularly those that are base64-encoded - to the phone for SPA operations.

After a long while without updates to the fwknop clients on Android or iOS, I'm
excited to announce that Jonathan Bennett
has developed an entirely new client for Android called Fwknop2. This
client adds significant features such as the ability to save configurations (including
both encryption and authentication keys), proper handling of base64 encoded keys, and
support for manually specified IP addresses to be allowed through the remote firewall.
Further, Fwknop2 integrates with
JuiceSSH
so that an SSH connection can seamlessly be launched right after the SPA packet is
sent. Finally, in an interesting twist, Fwknop2 will be able to read encryption
and HMAC keys via a QR code via the camera. The QR code itself is generated from
key material in --key-gen mode (which is currently available in the standard fwknop
client and will be available in the fwknopd server in the next release).

Fwknop2 will be available on both the Google Play store and via F-Droid within
the next few hours, and in the meantime the APK is available on github
here.

Below are a couple of screenshots of the new Android client in action - these are
from an Android VM running under Parallels on Mac OS X (Yosemite):

People who use Single Packet Authorization (SPA) or its security-challenged cousin
Port Knocking (PK) usually access SSHD running on the same system where the SPA/PK
software is deployed. That is, a firewall running on a host has a default-drop
policy against all incoming SSH connections so that SSHD cannot be scanned, but a
SPA daemon reconfigures the firewall to temporarily grant access to a passively
authenticated SPA client:
This works
well enough,
but both port knocking and SPA work in conjunction with a firewall, and "important" firewalls
are usually gateways between networks as opposed to just being deployed on standalone hosts.
Further, Network Address Translation (NAT) is commonly used on such firewalls (at least
for IPv4 communications) to provide Internet access to internal networks that are on
RFC 1918 address space, and also to allow external hosts access to services hosted on
internal systems.

The prevalence of NAT suggests that SPA should integrate strongly with it. Properly done,
this would extend the notion of SPA beyond just supporting the basic feature of granting
access to a local service. To drive this point home, and to see what a NAT-enabled vision
for SPA would look like, consider the following two scenarios:

A user out on the Internet wants to leverage SPA to access an SSH daemon that
is running on a system behind a firewall.
One option is to just use SPA to access SSHD on the firewall first, and then
initiate a new SSH connection to the desired internal host from the firewall itself.
However, this is clunky and unnecessary. The SPA system should just integrate with
the NAT features of the firewall to translate a SPA-authenticated incoming SSH
connection through to the internal host and bypass the firewall SSH daemon
altogether:

A local user population is behind a firewall that is configured to block all access
by default from the internal network out to the Internet. Any user can acquire an IP on
the local network via DHCP, but gaining access to the Internet is only possible after
authenticating to the SPA daemon running on the firewall. So, instead of gaining
access to a specific service on a single IP via SPA, the local users leverage SPA
to gain access to every service on every external IP. In effect,
the firewall in this configuration does not trust the local user population, and Internet
access is only granted for users that can authenticate via SPA. I.e., only those users
who have valid SPA encryption and HMAC keys can access the Internet:

(A quick note on the network diagrams above - each red line represents a connection that
is only possible to establish after a proper SPA packet is sent.)

Both of the above scenarios are supported with fwknop-2.6.6, which has been
released today. So far, to my knowledge, fwknop is the
only SPA/PK implementation with any built-in NAT support. The first scenario of gaining
access to an internal service through the firewall has been supported by fwknop for a long time, but
the second is new for 2.6.6. The idea for using SPA to gain access to the Internet
instead of just for a specific service was
proposed by "spartan1833"
to the fwknop mailing list, and is a powerful extension of SPA into the world of NAT - in
this case, SNAT in iptables parlance.

Before diving into the technical specifics, below is a video demonstration of the
NAT capabilities of fwknop being used within Amazon's AWS cloud. This shows fwknop using
NAT to turn a VPC host into a new gateway within Amazon's own border controls for the
purposes of accessing SSH and RDP running on other internal VPC hosts. So, this illustrates
the first scenario above. In addition, the video shows the usage of SPA
ghost services
to NAT both SSH and RDP connections through port 80 instead of their respective default
ports. This shows yet another twist that strong NAT integration makes possible in the SPA
world.

Now, let's see a practical example. This is for the second scenario above where a system
with the fwknop client installed is on a network behind a default-drop firewall running
the fwknop daemon, and the new SNAT capabilities are used to grant access to the Internet.

First, we fire up fwknopd on the firewall after showing the access.conf and fwknopd.conf
files (note that some lines have been abbreviated for space):

Back on the server, below are the rules that are added to grant Internet access to
the spaclient system. Note that all ports/protocols are accepted for forwarding
through the firewall, and an SNAT rule has been applied to the spaclient source IP
of 192.168.10.55:

Conclusion
Most users think of port knocking and Single Packet Authorization as a means to passively
gain access to a service like SSHD running on the same system as the PK/SPA software itself.
This notion can be greatly extended through strong integration with NAT features in a firewall.
If the SPA daemon integrates with SNAT and DNAT (in the iptables world), then both internal
services can be accessed from outside the firewall, and Internet access can be gated by SPA too.
The latest release of fwknop supports both of these scenarios, and please email me
or send me a message on Twitter @michaelrash if
you have any questions or comments.

Last week there was an interesting question
posed by Peter Smith
to the fwknop mailing list that focused on running fwknop in
UDP listener mode
vs. using libpcap. I thought it would be useful to turn this into a blog post, so here is Peter's
original question along with my response:

"...I want to deploy fwknop on my server, but I'm not sure If I should use
the UDP listener mode or libpcap. At first UDP listener mode seems to be
the choice, because I don't have to compile libpcap. However, I then
have to open a port in the firewall. Thinking about this, I get the
feeling that I'm defeating the purpose of using SPA, by allowing
Internet access to a privileged processe.

If an exploitable security issue is found, even though fwknop remains
passive and undiscoverable, an attacker could blindly send his exploit
to random ports on servers he suspects running fwknopd, and after
maximum 65535 tries he would have root access. I'm not a programmer, so
I can't review the code of fwknop or SSH daemon, but if both is equally
likely of having security issues, I might as well just allow direct
access to the SSH daemon and skip using SPA.

Is my point correct?..."

First, let me acknowledge your point as an important issue to bring up. If
someone finds a vulnerability in fwknopd, it doesn't matter whether fwknopd
is collecting packets via UDP sockets or from libpcap (assuming we're
talking about a vulnerability in fwknopd itself). The mere processing of
network traffic is the problem.

So, why run fwknopd at all?

This is something of a long-winded answer, but I want to be thorough. I'll
likely not settle the issue with this response, but it's good to start the
discussion.

Starting with your example above with SSHD open to the world, suppose there
is a vulnerability in SSHD. What does the exploit model look like? Well, an
attacker armed with an exploit can trivially find the SSH daemon with any
scanner, and from there a single TCP connection is sufficient for
exploitation. I would argue that a primary enabling factor benefiting the
attacker in this scenario is that vulnerable targets that listen on TCP
sockets are so easy to find. The combination of scannable services +
zero-day exploits is just too nice for an attacker. Several years ago I
personally had a system running SSHD compromised because of this. (At the
time, it was my fault for not patching SSHD before I put it out on the
Internet, but still - the system was compromised within about 12 hours of
being online.)

Now, in the fwknopd world, assuming a vulnerability, what would
exploitation look like? Well, targets aren't scannable as you say, so an
attacker would have to guess that a target is running fwknopd and the
corresponding port on which it watches/listens for SPA packets [1].
Forcing an attacker to send thousands of packets to different ports (assuming a
custom non-default port instead of UDP port 62201 that fwknopd normally
watches) is likely a security benefit in contrast to an obviously
targetable service. That is, sending tons of SPA exploit packets to
different ports is a common pattern that is quite noisy and is frequently
flagged by IDS's, firewalls, SIEM's, and flow analysis engines. How many
systems could an attacker target with such heavyweight scans before the
attacker's own ISP would notice? Or before the attacker's IP is included
within one of the Emerging Threats compromised hosts lists? Or within
DShield as a known bad actor? 10 million? How many of these are actually
running fwknopd? I know there are some spectacular scanning results out
there, so it's really hard to quantify this, but either way there is a big
difference between sending thousands of > 100-byte UDP packets to each
target vs. a port sweep for TCP/22.

Further, when a system is literally blocking every incoming packet [2],
an attacker can't even necessarily be sure there is any target connected to
the network at all. Many attackers would likely move on fairly quickly from
a system that is simply returning zero data to some other system.

In contrast, for a service that advertises itself via TCP, an attacker
immediately knows - with 100% certainty - there is a userspace daemon with
a set of functions that they can interact with. This presents a risk. In my
view, this risk is greater than the risk of running fwknopd where an
attacker gets no data.

Another fair question to ask is about the architecture of fwknopd. When an
HMAC is used (this will be the default in a future release), fwknopd reads
SPA packet data, computes an HMAC, and does nothing else if the HMAC does
not match. This is an effort to try to stay faithful to a simplistic
design, and to minimize the functions an attacker can interact with - even
for packets that are blindly sent to the correct port where fwknopd is
watching. Thus, most functions in fwknopd, including those that parse
user-supplied data and hence where bugs are more likely to crop up, are not
even accessible without first passing the HMAC which is a cryptographically
strong test. When libpcap is also eliminated through the use of UDP, not
even libpcap functions are in the mix [3]. In other words,
the security of fwknop does not rely on not being discoverable or scannable - it is merely
a nice side effect of not using TCP to gather data from the network.

To me, another way to think of fwknopd in UDP mode is that it provides a
lightweight generic UDP authenticator for TCP-based services. Suppose that
SSHD had a design built-in where it would bind to a UDP socket, wait for a
similar authentication packet as SPA provides, and then switch over to TCP.
In this case, there would not be a need for SPA because you would get the
best of both worlds - non-scannability [4] plus everything that TCP provides
for reliable data delivery at the same time. SPA in UDP listening mode is
an effort to bridge these worlds. Further, there is nothing that ties
fwknopd to SSH. I know people who expose RDP to the Internet for example.
Personally, I'm quite confident there are fewer vulnerabilities in fwknopd
than something like RDP. Even if there isn't a benefit in terms of the
arguments above in terms of service concealment and forcing an attacker to
be noisy, my bet is that fwknopd - even if it advertised itself via TCP -
would provide better security than RDP or other services derived from
massive code bases.

Now, beyond all of the above, there are some additional blog posts and other
material that may interest some readers:

[1] If an attacker is in the position to watch legitimate SPA packets from
an existing client then this guesswork can be largely eliminated. However,
even in this case, if fwknopd is sniffing via libpcap (so not using the UDP
only mode), there is no requirement for fwknopd to be running on the system
where access is actually granted. It only needs to be able to sniff packets
somewhere on the routing path of the destination IP chosen by the client. I
mention this to illustrate that it may not be obvious to an attacker where
a targetable fwknopd instance runs even when SPA packets can be monitored.
Also, there aren't too many attackers who are in this position. At least,
the number of attackers in this position is _far_ lower than those people
(everyone) who are in a position to discover a vulnerable service from any
system worldwide with a single TCP connection.

[2] As you point out, in UDP-only mode, the firewall must allow incoming
packets to the UDP port where fwknopd is listening. But, given that fwknopd
never sends anything back over this port, it remains indistinguishable to
every other filtered port.

[3] There are more things built into the development process that may be
worth noting too that heighten security such as the use of Coverity, AFL
fuzzing, valgrind, etc., but these probably takes us too far afield from
the topic at hand. Also, there are some roadmap items that have not been
implemented yet (such as privilege separation) that will make the
architecture even stronger.

[4] Assuming a strong firewall stance against incoming UDP packets to
other ports so one cannot infer service existence because UDP/22 doesn't
respond to a scan but other ports respond with an ICMP port unreachable,
etc.

The American Fuzzy Lop fuzzer has become a critical
tool for finding security vulnerabilities in all sorts of software. It has the ability to send fuzzing
data through programs on the order of hundreds of millions of executions per day on a capable system,
and can certainly put strain on your hardware and OS. If you are fuzzing a target program with the AFL
mode where a file is written and the target binary reads from this file, then AFL is going to conduct
a huge number of writes to the local disk. For a solid-state drive this can reduce its life expectancy
because of write amplification.

My main workstation is a Mac running OS X Yosemite, and I run a lot of Linux, FreeBSD, and OpenBSD
virtual machines under Parallels for development purposes. The drive on this system is an SSD which
keeps everything quite fast, but I don't want to prematurely shorten its life through huge AFL fuzzing
runs. Normally, I'm using AFL to fuzz fwknop from an Ubuntu-14.10 VM, and what is
needed is a way to keep disk writes down. The solution is to use a RAM disk from within the VM.

First, from the Ubuntu VM, let's get set up for AFL fuzzing and show what the disk writes look like
without using a RAM disk from the perspective of the host OS X system. This assumes AFL 0.89 has been
installed already and is in the current path:

We're not running AFL yet. Now, from the Mac, launch the Activity Monitor (under Applications
> Utilities) and look at current disk utilization:
So, not terrible - currently 31 writes per second at the time the snapshot was taken, and that includes
OS X itself and the VM at the same time. But, now let's fire up AFL using the digest cache wrapper on
the Ubuntu VM (the main AFL text UI is not shown for brevity):

And now let's take a look at disk writes again from OS X:
Whoa, that's a massive difference - nearly two orders of magnitude. AFL has caused disk writes to spike
to over 2,700 per second with total data written averaging at 19.5MB/sec. Long term fuzzing at this
level of disk writes would clearly present a problem for the SSD - AFL frequently needs to be left running
for days on end in order to be thorough. So, let's switch everything over to use a RAM disk on the Ubuntu
VM instead and see if that reduces disk writes:

Here is disk utilization once again from the Mac:
We're back to less than 10 writes per second to the SSD even though AFL is going strong on the Ubuntu
VM (not shown). The writes for the previous fuzzing run are still shown to the left of the graph (since
they haven't quite aged out yet when the screenshot was taken), and new writes are so low they don't
even make it above the X-axis at this scale. Although the total execs per second - about 2,000 - achieved
by AFL is not appreciably faster under the RAM disk, the main benefit is that my SSD will last a lot
longer. For those that don't run AFL underneath a VM, a similar strategy should still apply on the main
OS. Assuming enough RAM is available for whatever software you want to fuzz, just create a RAM disk and
run everything from it and extend the life of your hard drive in the process.

Over the past few months, the American Fuzzy Lop
(AFL) fuzzer written by Michal Zalewski has become a tour de force in the security field. Many interesting bugs
have been found, including a late breaking bug in the venerable cpio utility
that Michal announced to the full-disclosure list.
It is clear that AFL is succeeding perhaps where other fuzzers have either failed or not been applied, and this
is an important development for anyone trying to maintain a secure code base. That is, the ease of use coupled with
powerful fuzzing strategies offered by AFL mean that open source projects should integrate it directly
into their testing systems. This has been done for the fwknop project with some basic
scripting and one patch
to the fwknop code base. The patch was necessary because according to AFL documentation,
projects that leverage things like encryption and cryptographic signatures are not well
suited to brute force fuzzing coverage, and fwknop definitely fits into this category.
Crypto being a fuzzing barrier is not unique to AFL - other fuzzers have the same problem.
So, similarly to the libpng-nocrc.patch included in the AFL sources, encryption,
digests, and base64 encoding are bypassed when fwknop is compiled with
--enable-afl-fuzzing on the configure command line. One doesn't need to apply
the patch manually - it is built directly into fwknop as of the 2.6.4 release. This is
in support of a major goal for the fwknop project which is comprehensive
testing.

If you maintain an open source project that involves crypto and does any sort of
encoding/decoding/parsing gated by crypto operations, then you should patch your project so that
it natively supports fuzzing with AFL through an optional compile time step. As an example, here is a
portion of the patch to fwknop that disables base64 encoding by just copying data manually:

Fortunately, so far all fuzzing runs with AFL have turned up zero issues (crashes or
hangs) with fwknop, but the testing continues.

Within fwknop, a series of wrapper scripts are used to fuzz the following four areas
with AFL. These areas represent the major places within fwknop where data is consumed
either via a configuration file or from over the network in the form of an SPA packet:

Each wrapper script makes sure fwknop is compiled with AFL support, executes afl-fuzz
with the corresponding test cases necessary for good coverage, archives previous
fuzzing output data, and supports resuming of previously stopped fuzzing jobs. Here is
an example where the SPA packet encoding/decoding routines are fuzzed with AFL after
fwknop is compiled with AFL support:

And now AFL is up and running (note the git --abbrev-commit tag integrated
into the text banner to make clear which code is being fuzzed):
If the fuzzing run is stopped by hitting Ctrl-C, it can always be resumed as follows:

$ ./fuzzing-wrappers/spa-pkts.sh resume

Although major areas in fwknop where data is consumed are effectively fuzzed with
AFL, there is always room for improvement. With the wrapper scripts in place, it is
easy to add new support for fuzzing other functionality in fwknop. For example, the
digest cache file (used by fwknopd to track SHA-256 digests of previously seen SPA
packets) is a good candidate.

UPDATE (11/30/2014): The suggestion above about fuzzing the digest cache file proved
to be fruitful, and AFL discovered a bounds checking bug that has been fixed in
this commit.
The next release of fwknop (2.6.5) will contain this fix and will be made soon.

Taking things to the next level, another powerful technique that would make an interesting
side project would be to build a syntax-aware version of AFL that handles SPA packets
and/or configuration files natively. The AFL documentation hints at this type of
modification, and states that this could be done by altering the fuzz_one() routine in
afl-fuzz.c. There is already a fuzzer written in python for the fwknop project that is
syntax-aware of the SPA data format
(see: spa_fuzzing.py),
but the mutators within AFL are undoubtedly much better than in spa_fuzzing.py. Hence, modifying
AFL directly would be an effective strategy.

Please feel free to open an issue against fwknop in
github if you have a suggestion
for enhanced integration with AFL. Most importantly, if you find a bug please let me
know. Happy fuzzing!

The 2.6.4 release of fwknop is available for
download. New functionality has been developed
for 2.6.4, including a new UDP listener mode to remove libpcap as a dependency for
fwknopd, support for firewalld on recent versions of Fedora, RHEL, and Centos
(contributed by Gerry Reno), and support for Michal Zalewski's 'American Fuzzy Lop'
fuzzer. Further, on systems where execvpe() is available, all system() and popen()
calls have been replaced so that the shell is not invoked and no environment is used.
As usual, fwknop has a Coverity Scan score of zero,
and the code coverage report achieved by the 2.6.4 test suite is available
here.

Here is the complete ChangeLog for fwknop-2.6.4:

[server] Added a UDP server mode so that SPA packets can be acquired via
UDP directly without having to use libpcap. This is an optional feature
since it opens a UDP port (and therefore requires the local firewall be
opened for communications to this port), but fwknopd is careful to never
send anything back to a client that sends data to this port. So, from the
perspective of an attacker or scanner, fwknopd remains invisible. This
feature is enabled in fwknopd either with a new command line argument
--udp-server or in the fwknopd.conf file with the ENABLE_UDP_SERVER
variable. When deployed in this mode, it is advisable to recompile fwknop
beforehand with './configure --enable-udp-server' so that fwknopd does
not link against libpcap.

[server] Replaced all popen() and system() calls with execvpe() with no
usage of the environment. This is a defensive measure to not make use of
the shell for firewall command execution, and is supported on systems
where execvpe() is available.

(Gerry Reno) Added support for firewalld to the fwknopd daemon on RHEL 7
and CentOS 7. This is implemented using the current firewalld '--direct
--passthrough' capability which accepts raw iptables commands. More
information on firewalld can be found here:
https://fedoraproject.org/wiki/FirewallD

[server] Added support for the 'American Fuzzy Lop' (AFL) fuzzer from
Michal Zalewski. This requires that fwknop is compiled with the
'--enable-afl-fuzzing' argument to the configure script as this allows
encryption/digest short circuiting in a manner necessary for AFL to
function properly. The benefit of this strategy is that AFL can fuzz the
SPA packet decoding routines implemented by libfko. See the test/afl/
directory for some automation around AFL fuzzing.

(Bill Stubbs) submitted a patch to fix a bug where fwknopd could not
handle Ethernet frames that include the Frame Check Sequence (FCS)
header. This header is four bytes long, and is placed at the end of each
Ethernet frame. Normally the FCS header is not visible to libpcap, but
some card/driver combinations result in it being included. Bill noticed
this on the following platform:
BeagleBone Black rev C running 3.8.13-bone50 #1 SMP Tue May 13
13:24:52 UTC 2014 armv7l GNU/Linux

[client] Bug fix to ensure that a User-Agent string can be specified
when the fwknop client uses wget via SSL to resolve the external IP
address. This closes issue #134 on github reported by Barry Allard. The
fwknop client now uses the wget '-U' option to specify the User-Agent
string with a default of "Fwknop/<version>". In addition, a new command
line argument "--use-wget-user-agent" to allow the default wget
User-Agent string to apply instead.

[python module] When an HMAC key is passed to spa_data_final() then
default to HMAC SHA256 if no HMAC mode was specified.