Linear-Feedback Shift Register is not Random

Continuing the unnamed CTF write-up series, today I will be presenting my tought process and solution to the last level of a cryptography-oriented challenge. Cracking this challenge took a few days, mostly due to my lack of free time and also because I was unable to find any hints online or people on IRC whose brains to pick in regards to this challenge.

The previous levels of this CTF used a few primitive ciphers, ranging from ROT-13 to Vigenère, however the final level is supposed to get more serious, moving from text-level ciphers to byte-level. The cipher itself is not mentioned, only that hex editors are required beyond this level. A directory is available with a sample key file, an example plain text and its resulting cipher text. Another executable binary is also available, which allows us to encrypt any arbitrary text with the same key which was used to encrypt the password of the next level, thus introducing the element of chosen-plaintext attack.

Fiddling around with the provided files in CrypTool was not going anywhere, therefore I decided to look at the provided HINT2 file, which hints at the encryption method being an 8-bit LFSR.

LFSR just generates a byte stream, which is then XOR'd to the source file, making the encryption same as the decryption process. Given a key, (z1,z2,...,zm)(z_{1}, z_{2}, ..., z_{m})(z​1​​,z​2​​,...,z​m​​), the byte stream is generated according to the following linear relationship:

My first thought was to try and use the provided encryption binary on the encrypted password file to perform a decryption. That, however, would've been too simple, and obviously didn't work. There is a known-plaintext attack on LFSR (when applied with XOR), where you can recover the byte stream LFSR generated by simply XOR'ing the known and cipher files, and then using the result to decrypt other encrypted files. As such, the following Python script was born:

Since I ran out of ideas to try, I grabbed the encrypt6 binary from the server (using scp, since network connections are otherwise blocked) and threw it into IDA. As the HINT2 file said, there is an lfsr() function in the binary:

However, if this is all it did, then the previous XOR trick should've worked, which suggests that LFSR is only one half of the process. A quick look at where this function is called leads to this function block:

As we can see, the encryption process consists of subtracting "A" from the character, adding a character to it from the keyfile (v17[v11]) plus the output of the LFSR (v13). The challenge description states that a static key and a random number is used during the encryption, however, as it has no calls to any rand functions, it seems that the LFSR is supposed to be the source of the randomness.

Studying the for loop in the disassembled code leads me to modifying the above Python script with the following changes: