The boards used here weren't really designed for hash breaking. However, this
application is so simple in terms of chip interconnect that it can be fitted into
virtually any FPGA-based hardware. Thanks to a JTAG
reverse-engineering tool written for this project, enough connectivity could
be resolved to do that and 2 boards were good enough (after small repairs) to
run NSA@home.

The boards contain 15 Virtex-II Pro (XC2VP20) FPGAs in 3 identical sets of 5
(here called "channels"). Each channel also owns a Spartan-II (XC2S50) FPGA that
was originally used as a control chip, and a DSP (ADSP21160M) which probably
calculated transform parameters. There is also a shared XC2S50 chip, which is
not used in this application, just like the DSPs. The clock distribution tree
unfortunately contains 2 domains, which means the 39MHz channel clock had to be
distributed from chip to chip, using internal Virtex-II DCMs to clean it up.

To control the boards, an USB boardlet has been built and
the Spartan-II chips programmed to share the USB interface using a ring protocol.
Spartan-II chips translate USB-carried command packets
into bus transactions (16-bit address, 18-bit data) and the bus read data into USB
packets again. They also service SHA1 breaker IRQs to minimize the number of results
that had to be merged into one - there is a buffer for 1024 18-bit reads in each
Spartan-II.

The Spartan-IIs also drive a serially-controlled 16-character LED matrix display
(2x Siemens/Infineon SCD5583A) that is used to show the current board state.

The boards were notably inconvenient to use without an enclosure. Since the
original case for them was a custom-built card cage - dimensions are nonstandard -
the boards were housed into hacked-up computer 1u cases. As each board dissipates
about 120W while operating, six ducted fans are required to provide cooling.
The internal arrangement of the 1u case can be seen below.

After some debugging, which involved issues like signal integrity inside and
outside the FPGAs, overheating, power supply issues but no digital design bugs,
the boards successfully booted. A closed case with a board displaying "Hello
World" looks like this:

(c) 2007 Stanislaw Skowronek

Contact me: nsa unaligned org (figure out where to put the @ and .)irc.unaligned.org #nsa