I'm using openssl library and I've a problem with RSA_private_encrypt() and RSA_public_decrypt(). I'm using a key size of 256 bit with no padding of the input message. I've noted that when my input message ( random generated ) has 248 bit or less the functions fail with the following error :

1 Answer
1

First, RSA "encrypt with private key" and "decrypt with public key" are semantically wrong; these don't provide confidentiality which is the purpose of encryption, but they can provide integrity which is the purpose of signature.
The fact that RSA sign/verify operations are mathematically similar to encrypt/decrypt led to this misuse of terminology back in the 1980s, which unfortunately was (and remains) enshrined in the bottom-level function names in OpenSSL. (Higher level functions especially EVP fix this.)

Second, RSA without padding, so called "textbook" RSA, is not considered secure; it was also discovered back in the 1980s that it could be broken in cases that could not be reliably avoided. Actual use of RSA today uses padding, which differs between signing and encryption even though the math operation at the core (modular exponentiation) is almost the same. This means signing a value that is entirely random (unpadded) is basically useless.

Third, RSA 256 bits is much too small to be secure. I haven't worked out the numbers, but it could be factored very quickly by someone who wanted to break your key -- if you hadn't already published it. The recent FREAK attack (on SSL/TLS) demo factored 512 bits using AIR something like 1000 Amazon cloud machines in about 8 hours.

But those said, what you describe does work for me. I'm not sure what your "Output public key" is, but I can sign and verify/recover your second value (beginning 01) with the same signature you show for "Output private key", and I can sign and recover the value beginning with 00 the same way.