Monday, January 2, 2012

Manual verify PKCS#7 signed data with OpenSSL

Recently I was having some trouble with the verification of a signed message in PKCS#7 format. To troubleshoot why the library I was using kept rejecting the message I wanted to verify the signed message step by step, using OpenSSL. Below is a description of the steps to take to verify a PKCS#7 signed data message that is signed with a valid signature. Though I imagine these steps will apply to CMS messages for a big part too, I haven't looked into this.

Update 2013-04-12: this post was written to explain all the steps involved in the verification of a PKCS#7 message. Which might come in handy when troubleshooting compatibility issues. If however you're just interested in performing PKCS#7 encryption, decryption, signing and/or verification please have a look at my new post: PKCS#7 and OpenSSL. Which is also a good start when you are troubleshooting PKCS#7 communication.

Generate certificate
Generate a RSA test key and certificate, if you don't have one available.openssl req -x509 -nodes -newkey rsa:1024 -keyout keyfile.key -out certificate.cer
Generating a 1024 bit RSA private key
.........++++++
.........................................++++++
writing new private key to 'keyfile.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:

OpenSSL req is used to generate a self signed test certificate with an available private key. Here's an explanation of the used parameters.

-x509

output a certificate instead of a request

-nodes

don't encrypt the private key

-newkey rsa:1024

create a new RSA private key of 1024 bits

-keyout keyfile.key

store the private key in keyfile.key

-out certificate.cer

store the certificate in certificate.cer

Create a file to be signedecho "Some text" > data.txt

Sign the data with keyfile and certificate
The signed data in this example is created with the command below.
(-md is available since OpenSSL 1.0.0)openssl smime -sign -md sha1 \
-binary -nocerts -noattr \
-in data.txt -out data.txt.signed -outform der \
-inkey keyfile.key \
-signer certificate.cer

OpenSSL smime is used to sign the data. Here's an explanation of the used parameters.

-sign

instruct OpenSSL to sign the data specified

-md sha1

the message digest algorithm to use is SHA1

-binary

treat the data as binary, otherwise the data is interpreted as SMIME data
(in SMIME all newlines are replaced by 0x0D 0x0A)

-nocerts

don't include the certificate used for signing in the PKCS#7 message

-noattr

don't include any signed attributes

-in data.txt

the file to be signed is data.txt

-out data.txt.signed

save the signature in data.txt.signed

-outform der

save the signature in DER format

-inkey keyfile.key

the keyfile for the certificate that's to be used for signing is keyfile.key

-signer certificate.cer

the certificate to sign the data with is certificate.cer

Find offset of hex data
OpenSSL asn1parse is used to allocate the signature in the PKCS#7 message. The PKCS#7 message in data.txt.signed has the following (simplified) structure.

Extract the public key from the certificate
Since the signature is encrypted with RSA, and OpenSSL requires a separate key file to perform RSA encryption, the following command is used to extract the public key from the certificate for use with rsautl.openssl x509 -inform pem -in certificate.cer -noout -pubkey > pubkey.pem

OpenSSL x509 is used to extract the public key. Here's an explanation of the used parameters.

-inform pem

depending on your certificate use pem or der to instruct OpenSSL to read the specified file as PEM or DER encoded data

35 comments:

Hi , I get the proper hash values when i use -noattr option but i am not getting the same hash values when -noattr is not used while signing the data. Signature looks to be different in the two cases ? what could be the reason ?

You could take a look at the signed structure generated using 'openssl asn1parse', e.g. openssl asn1parse -inform der -in data.signed (or use an graphical ASN.1 viewer/editor).

-noattr tells openssl to not sign any attributes, otherwise openssl includes some standard signed attributes. These default attributes include extra security information like contentType and signingTime.

How this extra data is included in the calculation of the hash is beyond the scope of this article. If you are interested, RFC2315 (http://www.ietf.org/rfc/rfc2315.txt) might serve you some useful information.

Hi , Basically I am interested mainly in how to decrypt this hash value using the public key. The above mentioned command "openssl rsautl -verify -pubin -inkey pubkey.pem < signed-sha1.bin > verified.bin" decrypts it and gives some hash value as output but then the sha256sum/sha1sum of data do not match with it . So as per my understanding i need to get the exact command which would give correct results even when i do not use option -noattr while signing the data.

Hi I tried the above command , it will work fine ,But My requirement is to verify only the signature using the public key .I am not going to use data-attribs.txt.signed(signed data in the command ) as this command then use high ram for large size data .These are the steps that i am following and the final hash is mismatching : 1. openssl req -x509 -newkey rsa:1024 -keyout keyfile.key -out certificate.cer // generating certificate and private key

Hash1 != Hash2 ,that is the problem i am facing I need to get the hash values same for this example atleast.I have used an utility to extract the Signature which i am getting it exactly same as hexdump of output of command "openssl asn1parse -inform der -in data.txt.signed" So i only doubt command 5.

Using a data file, with random data, of 100 MB my openssl uses only 4 MB of memory to verify using the command in my last reply.

According to RFC2315, when your PKCS#7 file contains authenticated attributes, the hash you are calling HASH1 is "the message digest of the complete DER encoding of the Attributes value containted in the authenticatedAttributes field."I have been trying to find out how to calculate the message digest that compares to the HASH1, but haven't been lucky yet.

Could you maybe describe a little more detail to what you're trying to achieve?

Ok, I've found it. I hope it's understandable from the comments, since I can't properly indent the text.Be aware that the script provided below is purely to demonstrate how verifying could be done manually.If you are writing code/a script that will be used in a production environment where you need security, this probably is not what you should use!openssl (or libopenssl) provides lots of opertuneties to properly verify signed messages and will perform more (all?) specified checks.I cannot and will not guarantee any security or even proper functioning of the instructions provided!

Extract the signed attributes, it's the cont[0] in the SignerInfo, just after the certificate:8. dd if=data.txt.signed of=signed-attribs.bin bs=1 skip=616 count=$[ 3 + 216 ]

Change the ASN.1 SEQUENCE to a SET by changing the first byte to 0x319. echo -ne "\x31" | dd conv=notrunc bs=1 count=1 of=signed-attribs.bin

Hash the signed attributes, the result can be compared to the hash from 6.Thereby verifying the signed attributes are signed by the public key provided in 5.However, this doesn't verify the actual data!!10. sha1sum signed-attribs.bin

Hello Chris, your post was a blessing to find. Did learn a lot of stuff thanks to you.

I'm trying to extract authenticated attributes from a PKCS#7 signed message generated by iOS (it's for over the air enrollment) as part of the SCEP protocol (http://tools.ietf.org/pdf/draft-nourse-scep-23.pdf)

Here is the signed file received by the server (there is an encrypted CSR inside) : https://dl.dropbox.com/u/2310128/pkcs7/signed.p7s

According to the specs (p16/39), there should be several authenticated attributes in the SignerInfo :

So the dd command should look like this:dd if=signed.p7s of=signed-sha1.bin bs=1 skip=$[ 1689 + 3 ] count=1281689 is the offset where the ASN.1 octet string starts (including the ASN.1 tag's header). The length of the header (hl=) is 3. So the actual signature starts at byte 1689 + 3 = 1692. To make it more obvious where the numbers come from, I prefer to use bash to calculate it, that's why I've used $[ 1689 + 3]. The length (l=) of the signature is 128.

The rest of your steps seem in order. Though, you might want to save the result of the last command to a file to compare the hash of the data to.(openssl rsautl -verify -pubin -inkey signed-pub.pem verified-sha1.bin)

Regarding your other question. I think the values are present (check a few).You'll just have to lookup the OIDs yourself, since openssl doesn't know of them.

Googling one of the OIDs, I've found this site that mentions the same fields you did and has links them to OIDs that are present in the signed.p7s.http://jscep.googlecode.com/svn-history/r876/trunk/docs/org/jscep/asn1/SCEPObjectIdentifiers.html

There I've noticed the name 'SCEP'. Which led me (with Google) to this draft: http://tools.ietf.org/internet-drafts/draft-nourse-scep-20.txtHave a look at Appendix A for the OIDs.

To make interpreting the OIDs easier you could create file with OIDs and provide it in the asn1parse command with -oid. (http://www.openssl.org/docs/apps/asn1parse.html)

Please let me know if anything in my explanation is still unclear to you, I'm happy to help out (when I've got the time).

Also, the priv-key you're using, is that a key used only for testing? If so, maybe you could send it to me, so I could try using it to decrypt the data. Though, I can't make any judgements on the sensitivity of the key or encrypted data.

The certificate attached in the PKCS#7 most likely contains (should contain according to PKCS#7-standard) the public key used to encrypt the data. To verify if you're using the correct private key, you could compare the modulus of the public key in the certificate and the private key you're using. The modulus should match.

So i continued to try and try, and realised that the '-raw' option probably just fools me into thinking that i had decrypted the data.

So i extracted the pubkey from the self-signed certificate (CN=D63DA9A9-B713-4B6A-8BAE-DBFFBDF3C69D) and did 'rsautl -verify' on data.encrypted, and then did a 'asn1parse', and... that seems to be a md5 checksum. So not a CSR at all that i had hoped for.

So instead i 'dd' out the pkcs7-data that exists at position 57 in pkiop.der.(bigdata.unknown)

I do a 'asn1parse' on it, and now i see some more stuff.http://pastebin.com/JigTePAw

The 'test' certificate seems to be involved:)

So i try to decrypt/verify the HEX DUMPS in there with various private keys. No luck.But the CSR has to be in there somewhere:)

Gonna get at it next week again, and post back here if i finally find an answer.

Now i can see the CSR.The errors i did were...* not using the -noverify option to smime* trying to use smime/-decrypt on _only_ the HEXDUMP part that i saw when using asn1parse, instead of the whole... "envelope" or whatever its called.

Now i am trying to figure out how the format of the certificate reply to the iPhone should be.When i find that out i'll reply back here for completeness:)

Could you help me discover how to manually deal with certificate I have?

First of all here's my PKCS#7 package: http://pastebin.com/tVL1rGK4The data extracted from it: http://pastebin.com/w1mifmuiFinally the certificate file I got from the publisher http://pastebin.com/CzzC473k

When authenticated attributes are contained in the message, it's the hash of these that is actually signed. To protect the (i.e. sign) the contained data, the message digest is included in the authenticated attributes.

(for info on oap and oad see my most recent post: http://qistoph.blogspot.nl/2013/04/pkcs7-and-openssl.html)

These are the authenticated attributes, which could be read ascontentType: pkcs7-datasigningTime: 130416111717ZmessageDigest: 0CAF4BF8778254973AC4690762D93F128CF3BABA

As you can see the messageDigest matches your data's digest.

To verify the signature of the authenticated object, you'll have to calculate the hash of those.For some reason, yet unknown to me, it's required to calculate the hash over a SET of attributes.

I'm planning on writing a more detailed blog post about these authenticated objects, but for now here's the solution for this specific message:$ openssl pkcs7 -inform pem -in package -outform der -out package.der$ oad package.derNote offset, header length and data length of the authenticated attributes (offset=2263, hl=2, l=93)The following dd-command copies the authenticated attributes, but skips the first byte.The echo command produces a single byte with the value 0x31, which is the tag, in ASN.1, for a SET.

Also, for most languages libraries exist to help in properly verifying PKCS#7 messages. If you could be a little more specific on the language you're developing in, maybe I can help you in advising such a library.

Thanks for your Valuable and helpful Tutorials.i am writing 1 module which will verify the signed data using RSA public key.

Input what i will get is

1> Original data2> Signed data3> Public key / Certificate

Both data and key will be in Hex format.

You can refer to code herehttp://pastebin.com/CdPkW8wpAnd Data What we have is herehttp://pastebin.com/ExYDcJ2t

as of now for testing we are extracting key from certificate.Problem what i am facing is Verify function saying invalid signature. i guess the problem is in data format what we are reading in buffer.Please help.i need it urgently.

Following this awesome post has got me as far as gleaning cert details from the SCEP request, but how do I get this to a point that I have a CSR for the CA to sign, and then how is that sent back? I've read up a bit on SCEP, but my poor brain couldn't handle the PKCS7 and 10 stuff ;-)

So far I've got an existing SCEP client to create a cert private key/cert pair, and upload the cert via a faked "pkiclient.exe" CGI, which I saved as scep-request.pem

At that point I don't know what to do next. How do I change that into a CSR that I can then sign, and then how should that be fed back to the client?

I must say SCEP seems harder than it needs to be. Why couldn't it just involve making a HTTPS connection to the SCEP server, and just send a standard CSR over and get the signed version back? I can't see the security advantage of using signed blobs over HTTP instead of unsigned blobs over HTTPS?

hey Chris van Marle sorry for late reply.......i didnt get any notification for your reply. Ya that problem is solved.Problem was in SHA1 and sha256. Signature uses SHA1 but certificate what we got has SHA256 written as Hash algorithm.later we got some idea and tried with SHA1.and signature verified successfully.Thank you very much.