$\begingroup$I thought it might be versioning, maybe changes made during AES.. but these are the vectors posted on the serpent homepage. I'll investigate tomorrow, compare to final submission.$\endgroup$
– JGUNov 18 '14 at 1:33

$\begingroup$@owlstead Do you know where the Bouncy Castle vectors came from? Looks like the discrepancy may be due to a difference in s-box implementations.$\endgroup$
– JGUNov 18 '14 at 1:55

1

$\begingroup$it could be that bouncy castle is using the AES submission bit ordering. the only reason the s-boxes would be different is if it is serpent-0, which uses DES s-boxes$\endgroup$
– Richie FrameNov 18 '14 at 5:08

1

$\begingroup$@John I couldn't find any other test vectors either, but I guess it is solved now...$\endgroup$
– Maarten Bodewes♦Nov 18 '14 at 8:41

2 Answers
2

Bouncy Castle seems to be using a reversed byte order for inputs and outputs when compared to NESSIE test vectors.

In order to replicate the NESSIE vector in Bouncy Castle, the order of all inputs and outputs needs to be flipped at the byte level, therefore the following results are given from a NESSIE compliant implementation (Set 1, Vector 120):

These results match vector 14 in reverse byte order, using a 100 block loop test, the BC test does not seem to perform an outer key loop like other Monte Carlo test implementations, such as NIST CAVP. The CAVP MCT performs 100 outer key loops with 1000 inner block loops for testing AES implementations. If continued, next iteration in the key loop would be as follows:

$\begingroup$Send a mail to one of the BC maintainers...$\endgroup$
– Maarten Bodewes♦Nov 18 '14 at 8:40

1

$\begingroup$as per user13484's answer, it want matter. however, if you look at their code, their counters go in reverse, and they use word to byte conversions which reverse the byte order. NESSIE implementations count foreward, and can treat byte arrays as word arrays, leading to better performance and less potential leakage.$\endgroup$
– Richie FrameNov 18 '14 at 10:42

$\begingroup$I added the Array.Reverse step to key, plain and ciphertext, and now results are aligning.. thanks for the insight$\endgroup$
– JGUNov 18 '14 at 17:55

$\begingroup$how can I reverse using CBC encoding?$\endgroup$
– drizztAug 14 '15 at 23:32

Okay, originally I answered saying we were doing the right thing, however, when the BC project approached the Serpent authors in 2009 it appears there was a breakdown in communication. We have just been told that the NESSIE vectors are in fact the correct ones. The previous incarnation, based on a direct interpretation of floppy 4 in the Serpent AES submission, actually had reversed vectors and keys in it.

So from BC Java 1.54, and already in BC C# 1.8, Serpent now conforms to the NESSIE vectors and in line with common practice (yes, this wasn't a rare mistake...) the previous version of Serpent which incorporates the reversal is now called Tnepres.

Note also: as we're trying to spread the word on this, there are also some versions of Serpent where, while the implementers were evidently aware of the reversal issue in the inputs/outputs, they clearly didn't realise that the keys were also reversed. If you run into one of these, please alert the implementer if possible. Quite a few of the NESSIE vectors will pass if this mistake is made, it requires testing with a broad range of key values to show it up.

$\begingroup$I prefer the NESSIE order as it allows higher performance software implementations (maybe), and I assume that is why it was used. The standard Serpent order was optimized for hardware implementation (supposedly)$\endgroup$
– Richie FrameNov 18 '14 at 10:31