Private keys extracted from .pfx and from separate encoded key file look different but both do work

Question

I have a CertAndKey.pfx file and corresponding EncryptedKey.pem - both provided from CA. The following commands result with 2 different decrypted key files key1.pem and key2.pem:

openssl rsa -in EncryptedKey.pem -out key1.pem

openssl pkcs12 -in CertAndKey.pfx -out key2.pem -nodes

(and certainly I've edited key2.pem, public part was purged and private key part was left.)

The difference in decrypted keys is not only "begin/end rsa private key" vs "begin/end private key" lines (key1 does have "RSA" word in this line while key2 don't). Key bodies are also differ and their lengths aren't equal. However, both key1 and key2 apeared to work fine in TLS handshake.

I tried also to decode the keys and compare them in binary form:

Certutil -decode key1.pem key1.bin

Certutil -decode key2.pem key2.bin -

and these .bin files are different as well and aren't equal in length.

I am quite new to this, can anybody explain this oddity for me?

Answer 1

The formatting is the actual difference.

The first command outputs keys in "PEM" or "PKCS#1" format. The text header identifies the key type as RSA, and if it's encrypted then there's a sub-header for the cipher parameters (algorithm and IV). The actual data inside is a flat array of numbers (n, e, d, p, q) and if it weren't for the surrounding text, you'd have to guess it was an RSA key.

The second command outputs keys in "PKCS#8" format. Here the same meta-information has been moved inside the binary data: those RSA parameters are now nested inside another array together with an OID indicating this is indeed an RSA key. If encrypted, then the cipher parameters are stored inside as well.

But both formats are similar in that they use ASN.1 DER for structuring the binary data – this means you can see the difference if you pipe both keys through openssl asn1parse -i or upload them to this nice decoder.

You can pipe a key through openssl pkey to convert it between formats.

Note that results of some commands may depend on OpenSSL version. Over time, various subcommands which previously output keys in PKCS#1 format have switched over to producing PKCS#8 by default.