Good afternoon. I am having an issue with the RSA library on iOS. I am generating a 4096bit RSA key pair and storing it on the iOS device. Then later decrypting data using that private key. The RSA instance is configured to use OAEP padding with SHA512 hashing. RSA decryption using the private key works for maybe around 1 week (it seems to be related to the length of time the key is stored on the device. I have no idea why). Then all of a sudden every single decryption attempt throws this error:

I made sure the entire xml rsa key pair that is being used is the exact same as it was when it was generated indicating the key didn't get corrupted while stored on the device. Additionally, as soon as I generate a new key pair, everything starts working again. The back end which performs rsa encryption using the device's public key serves many platforms, none of which exhibit this issue. The only difference is that the iOS client is using the Chilkat lib.

Unfortunately no. It does not appear to be related. That appears to be a confusion between padding types. This issue is quite strange because it happens after a period of time where all other things remain the same. As in, at no point is any other padding used except OAEP.

I just ported my application to Cocoa (OSX) and the interoperability between OSX and Windows (.NET) is consistently broken - as in every single time the encryption / decryption between the two systems fails.

I am using RSA with SHA512 as the hash and OAEP padding. The .NET library is BouncyCastle and the OSX library is Chilkat Cocoa.

Could there be an issue with the OAEP implementation using SHA512 within Chilkat? The .NET and PHP libraries I am using have never exhibited issues talking to eachother, however, Chilkat has presented an intermittent issue with PHP and a consistent issue with .NET.

Check to see if you're setting the LittleEndian property. Make sure it's at the default value of False/NO to produce Big-Endian output that conforms to OpenSSL/BounceyCastle. Little-Endian output conforms to what the Windows CryptoAPI produces.

Adam, I think the only way to get to the bottom of this is for me to explain a little more about RSA encryption and OAEP padding. Then with that understanding, we'll be able to determine where the error first appears.

With RSA encryption, the amount of data to be encrypted will always be small. There's a limit as to how many bytes can be encrypted, and this depends on the RSA key size and the padding algorithm. The output any RSA encryption is always exactly equal to the bit size of the key. Given that you have a 4096 bit key, the output is always 512 bytes (512 bytes * 8 bits/byte = 4096 bits). The max number of application bytes that can be encrypted is equal to this length minus the overhead for padding. (Let's not worry about exact numbers -- we just want to conceptually understand it.)

RSA encryption is a 2-step process. First the application data is padded to 512 bytes, then the result goes through a mathematical computation called exponential modulation. The 512 bytes is really just a gigantic integer, and the result is another gigantic integer (also 512 bytes).

RSA decryption is the reverse: First we do exponential modulation (using the private key) and the result should be the padded data. Then we unpad to recover the original data.

If the format of the padding is understood, and if Chilkat can log the padded data inside the LastErrorText (which I'll provide), then we can identify whether valid padded data is passed to the exponential modulation (exptmod) when encrypting, and whether valid data is returned from the exptmod when decrypting.

When something is OAEP padded, it will have this format:

EM = 0x00 || maskedSeed || maskedDB.

One test to see if we definitely do not have OAEP padded data is to see if the 1st byte is NOT 0. It must be 0 for it to be OAEP padded. Remember though.. the input and output of the exptmod function is a BIG positive integer. The OAEP padded data is the large integer fed to the exptmod when encrypting, and returned from the exptmod when decrypting. If we see less than 512 bytes, and the leading byte is not 0, then it implies the leading bytes are 0. (After all 0x00000022 is the same number as 0x22) If we have exactly 512 bytes and the 1st byte is NOT 0, then it is definitely not OAEP padded.

The maskedDB, when unmasked, has this format:

DB = lHash || PS || 0x01 || M.

So..if Chilkat can log the unmasked DB in the LastErrorText, we should see the original data preceded by a 0x01 byte. This is another way to verify that the OAEP padded data is correct. (Also, preceding the 0x01 byte, we should typically see a quantity of 0 bytes.)

For example, I ran a test program to generate a 4096-bit RSA key, and then encrypt/decrypt the string "hello". Here are the LastErrorText's, where we can see the OAEP padding. (More comments to follow..)

I can provide you with a build that has this extra logging -- just let me know exactly which build (iOS? .NET for some framework?, etc.) If you pass something to Chilkat for decryption, and the exptmod produces garbage output (i.e. not conforming to OAEP padding), then either the input data was corrupted, or the wrong key was used.

Good evening and thank you for your reply. Very informative. I do understand the basics of RSA and this helped elucidate it even further.

To your point about the wrong public / private key being used for the process - I do not believe this is the case as I have debugged and compared the keys being used already. If one takes the same RSA key pair using Bouncycastle and Chilkat (configured per my description above), the resulting cipher is decrypt-able only on the platform it was encrypted with - if the cipher is crossed to the other platform (all else equal) I get the errors described.

Of course, there is a good enough chance I am doing something incredible stupid, but perhaps there is a problem with either library. I am unsure which. I do, however, have the benefit of being able to inspect the OAEP implementation within Bouncycastle. I have not yet taken the time to dissect it per RFC2437.

Please provide your modified library for Cocoa (ObjC, OSX). Also, I want to call out the fact that your support has been phenomenal and I really do appreciate your time.