Exceptions relieve the programmer of tedious writing boilerplate code -- without removing the semantics of said code -- and they allow the programmer to arrange the code so that error handling code is more separate from the main program logic.

I was implementing password authentication for VNC in node.js the other day and faced a problem where it would just never successfully authenticate. I checked my implementation several times and it seemed fine. Then I tried to implement it in Python just to see if I was missing something obvious. It also didn't work. Then I tried doing the authentication through openssl and it was also giving a result I didn't expect. Finally I found some old C code and wrote node-des, which worked. I still haven't figured out why it didn't work in node, python and openssl. Perhaps my readers can help me understand what was going on!

Here is a quick overview of how VNC password authentication works. On the VNC server you set the authentication password. When a VNC client connects, the VNC server generates random 16 byte garbage called challenge and sends it to the client. The client DES encrypts the challenge with the password as the key. The result is called the response and is also 16 bytes in size. The client then sends it to the VNC server. The VNC server then also DES encrypts challenge with the password and compares it with the response. If challenge matches response, the session gets authenticated.

Let's say the password set on the VNC server is browsers, and the challenge that the VNC server sends to the client is the following 16 bytes:

0c fd 6b 8c 5c 04 b3 e5 01 40 b9 de 33 9e 0d db

Then DES encrypting these 16 bytes with the password browsers should produce the following response:

44 ee fc 48 83 bb 95 f1 c0 82 c3 e2 98 c3 6a 4a

I sniffed this data from a real VNC client - VNC server communication and I use this as a test case for node-des as it's the correct result. However now let's look at what happens if I encrypt the challenge with node's crypto module, with Python and with openssl.

First node's crypto module. I wrote the following program to DES encrypt the 16 challenge challenge bytes with the password browsers.

So node.js doesn't produce the expected 16 byte result. In fact I tried all the possible variations of the des algorithm that openssl (which node.js bases its crypto module on) supports. Perhaps I chose the wrong des implementation as there are many?

I had no idea what was going on at this point, so I decided to try Python and see if I can get the result I expect through Python's DES algorithm. Worst case I could spawn Python each time I need to authenticate at VNC. I used the pyDes module for my experiment and installed it to virtualenv as following:

Being totally clueless, I decided to use the plain command line openssl tool just to see if node.js or Python were messing up the binary data or something like that. As I had sniffed the challenge from the VNC session, I put it in a file challenge and ran the openssl tool as following:

The output was again different, 24 bytes in size and it didn't match the response.

At this point I had had enough with all these different results and I started searching the web for the simplest possible DES implementation as I wasn't up to implementing DES myself. I found some C code that was public domain and wrote node-des that produces the results that I expect and that VNC server accepts.

Victory! It worked and my VNC code was functional! The question is, why did every single attempt to DES encrypt with node, python and openssl fail? Any ideas? Let's get to the bottom of this in the comments!

Update

One of my readers noticed that many of the words in node's output were of the form cx xx, so he asked me if I was sure that node wasn't converting the output to utf8? I was pretty sure it wasn't, buffers are buffers of raw bytes after all, but this looked too suspicious. I looked up in the documentation and found out that the default encoding for the buffers was utf8...

So I modified the original program and replaced console.log(Buffer(response)) with console.log(Buffer(response, 'binary')).

Comments

Since DES is a reversible crypto algorithm and not a one-way hash function, you could attempt to DES decrypt all the various outputs you've obtained here, and see what they reverse back into. Perhaps that may yield some hints?

Paul, that's unlikely to work. Flipping any single bit should result in an output that looks random and completely unrelated. That isn't always perfectly true in practice, which is how differential cryptanalysis is possible, but, for all practical purposes, DES is resistant to differential cryptanalysis

Hrm? This is nothing to do with cryptanalysis. We already have the ciphertext and the key. We simply want to look at what the cleartext comes out as. E.g. it could be that the input that was processed had a linefeed on the end, or some whitespace, or any of a number of simple implementation bugs at the encoding stage.

from the source code you're using in it, and comments from other people on the web it looks like VNC uses "almost DES" they're making DES drop the MSB instead of the LSB like normal since the password is normally ASCII (when it was created anyway) then it'll drop the 7th bit that's always a 0 instead of one that's part of the password. Not doing this would seem to allow you to shift one character to the right or left in the alphabet without consequence (depending on the LSB of course) which is not a good thing for passwords.

http://www.vidarholen.net/contents/junk/vnc.html has a little more about it

Some crypto/digest APIs provide an additional function to complete the digest cycle. For example, Java's API uses both an update() method to add bytes to the digester and a digest() method to wrap up the computation, flush things, pad, etc.

All those who use DES should be aware of what you just found out. From the spec:
http://www.itl.nist.gov/fipspubs/fip46-2.htm
"Blocks are composed of bits numbered from left to right, i.e., the left most bit of a block is bit one."

So...implementers must process from LSB to MSB of each byte...otherwise you won't match other DES implementations (as you found out).

If they don't mention what order they expect the data bytes in then that's poor documentation on their part (as you found out again).

So I just came across this issue as well, as I was trying to fix a bug with non-ASCII passwords not working in our client's VNC implementation. It actually is EVEN WORSE.

The code already handled the bit reversing thing, so ASCII passwords had worked for years. Although RFB specified no string encoding, I found (by reading the source) that the Xvnc server uses whatever the system defaults to, which is UTF-8 on most Linux setups. Still didn't work after picking that encoding in the client. Lots of web searching (including finding this page) didn't solve it either.

Finally, after staring at Xvnc's DES source for some more, I worked out that not only does it reverse the bits in each byte, but because DES is 56-bit, it only uses the low 7 bits from each byte of the password. The crypto library used by the VNC implementation I'm working on evidently does not ignore the (now lowest) bit though.

So the correct way to generate the DES key for VNC ends up being to mask each byte with 0x7f and THEN reverse it. So the UTF-8 for "Ä", 0xc3 0xa4, must end up as 0xc2 0x24 in the DES key, not 0xc3 0x25. Argh!

And of course, because the encoding isn't specified in the protocol, different servers use different ones. TightVNC on Windows appears to use Latin-1 or a variant, at least when Windows is in EN-GB locale. Sigh.