TLS/SSL negotiation uses two different ciphers over time. Your RSA keys are only used for the first few packets of the connection.

On initial connection, the client and server exchange their RSA (asymmetric) public keys (which are 512,1024,2048, or 4096 bit).

The public key allows each party to encrypt data in the knowledge that only the other party (who has a corresponding private key which they keep secret) is able to decrypt it and an eavesdropper can’t intercept it.

This initial stage is used to exchange keys (56,64,128 or 256 bit) for the second symmetric cipher only.

Once keys have been exchanged, both ends switch to a less computationally intensive symmetric cipher (such as RC4 or AES) to exchange actual data, and what length your RSA key was (as long as it was sufficient) is completely irrelevant. Using a massively long RSA key will slow down initial connection negotiation, and break compatibility with some things (as you’ve found). 2048 is more than sufficient IMHO, and should work everywhere.

A few points:

Key length is not necessarily proportional to security. A massive key on a crap cipher is still weak encryption. Of course though an unreasonably short key on a brilliant cipher can still be brute forced.

Increasing the key length on certificates will slow down the initial key exchange process (sometimes significantly once you get beyond 2048 bit) but have no effect on the strength of the encryption used during the actual session.

Increasing the key length of course increases the time taken to brute-force it. Note though (as per the recent MD5 forgery attack) that brute forcing something once the cipher and key are sufficiently good is the least of your worries. Nevertheless 512-bit RSA keys are frowned upon (they’re not computationally infeasible to brute force), 1024 is most common, and 2048 is a good tradeoff between length and performance.

Whether the actual mainstream encryption in an HTTPS session is 128 or 256 bit depends on a combination of browser and server capabilities – IIS <=6, for example, won’t do 256 bit encryption (that might have changed with IIS 7 but I’m about 99% sure it hasn’t).

If you’re an inquisitive type, Bruce Schneier’s awesome book Secrets and Lies (while a little dated now in some respects) is a good introduction to all this guff.

One thing that the attack does show is that security is only as strong as the weakest link, so concentrating on keysize might be wasting valuable time that could be better spent looking at other factors of their system.

The views expressed on this weblog are mine alone. In no way do they represent the views or opinions of my employer, my family and most anyone else, nor is it endorsed or approved by them in any way, shape or form. My opinions are my own.