Using OpenSSL RSA_sign() in IKE context - Openssl

This is a discussion on Using OpenSSL RSA_sign() in IKE context - Openssl ; Hi,
while modifying some IKE code to use OpenSSL in order to have RSA
signature done by engines (support via opensc on Linux, more
specifically eGate and eToken PRO devices), I encountered the following
issue.
RSA_sign() has a 'type' argument ...

Using OpenSSL RSA_sign() in IKE context

Hi,

while modifying some IKE code to use OpenSSL in order to have RSA
signature done by engines (support via opensc on Linux, more
specifically eGate and eToken PRO devices), I encountered the following
issue.

RSA_sign() has a 'type' argument that describes the hash that was used
for generating the data. It is used to add digest info before padding
and private key encryption occur to complete the signature. There is a
specific case in the code, namely for SSL which does not use a digest
info but a fixed length 36 bytes hash value (concatenation of md5 and
sha1 hashes).

IKEv1 has the same kind of requirement. RFC 2409 has the following in
Section 5.1:

Since the hash algorithm used is already known there is no need to
encode its OID into the signature. In addition, there is no binding
between the OIDs used for RSA signatures in PKCS #1 and those used in
this document. Therefore, RSA signatures MUST be encoded as a private
key encryption in PKCS #1 format and not as a signature in PKCS #1
format (which includes the OID of the hash algorithm). DSS signatures
MUST be encoded as r followed by s.

Note that I cannot call RSA_private_encrypt() as a workaround because
the devices do not support it (even if on tokens, I have added specific
keys that allow decryption) and it does not seem the expected way to do
it.

At the moment, I am using NID_md5_sha1 type for RSA_sign() and have
temporarily deactivated the length check (36 bytes) in libp11 code but
this is also a temporary workaround. What would be the correct way to
handle that issue? Initially, I wanted to add a new NID_IKE_digest type
that would make RSA_sign() behave like with NID_md5_sha1 but with a
relaxed check on the length. Because the removal of the length check
basically provides access to private key encryption via signature
primitive, I thought I should ask before.