File encrypted with OpenSSL cannot be decrypted with LibreSSL

Question

I've used OpenSSL on an Amazon Linux 2 VM to encrypt a file like so:

openssl aes-256-cbc -salt -out ~/my_file.txt.enc -pass pass:[redacted]

However when decrypting the file on OSX using the command below, I keep getting "Bad Decrypt".

openssl aes-256-cbc -d -salt -in ~/my_file.txt.enc -pass pass:[redacted] > my_file.txt

Are there differences between the decryption process using LibreSSL on OSX and OpenSSL on Linux, or have I got something else wrong in the decryption process?

Answer 1

Based on the versions you provide in your comment, it sounds like what @MattCaswell indicated should be the indeed the culprit: OpenSSL changed the default digest from md5 to sha-256 in version 1.1.0, see in the History text of the documentation. Based on this, your 1.0.2k version must be using the md5 digest.

Try passing -md md5 as an argument to your decrypting command to decrypt old files, like this:

openssl aes-256-cbc -d -salt -md md5 -in ~/my_file.txt.enc -pass pass:[redacted] > my_file.txt

If you are encrypting new files, it is better to add -md sha-256 in the encryption command instead, to keep it more portable, since new versions use this by default (and sha-256 is presumably more secure than the older md5 anyway).

Update

Ignore the previous update (striked out below), it turns out both openssl 1.0.2 and LibreSSL 3.3.6 use the old key derivation function by default anyway, nevertheless it is a good idea to add -pbkdf2 in both your encryption and decryption commands anyway, in order to use a better key derivation function.

Regarding the actual problem, I tried the exact versions you mentioned, and both these combinations work (encryption in 1.0.2k-fips, decryption in LibreSSL 3.3.6):

Using md5 (no extra argument in 1.0.2k-fips, -md md5 in LibreSSL):

cat test.txt | openssl aes-256-cbc -salt -out test.enc -pass pass:passwd
openssl enc -d -aes-256-cbc -pass pass:passwd -md md5 -in ~/test.enc

Using sha256 (-md sha256 in openssl 1.0.2k-fips, no argument in LibreSSL):

cat test.txt | openssl aes-256-cbc -salt -md sha256 -out test2.enc -pass pass:passwd
openssl enc -d -aes-256-cbc -pass pass:passwd -in ~/test2.enc

So it looks like the md md5 option should actually help. Could you double check your commands, maybe you mistyped the password or checking the wrong files? Of course it also wouldn't harm to use the -md xxx option in both the encryption and decryption anyway, just to be on the safe side.

Based on your comment that -md md5 did not help, it looks that this is not the only problem. Additionally, it looks like the newer versions are using the PBKDF key derivation function, which older versions (like 1.0.2) were not using.

The best approach will be of course, if you have the possibility to update your amazon linux to a newer version, and use the safer PBKDF algorithm.

If this is not possible, you could use an actual key and iv for encryption and decryption instead of a password (which is transformed to a key and iv by the key derivation function, which is different in new and old openssl). Assuming you have a good random source, instead of passing -pass pass:[redacted], use -K <64 random hex characters> and -iv <32 random hex characters>, and pass the same key and iv in both encryption and decryption.

That is of course assuming you want a command to encrypt in amazon linux and decrypt in osx. If you already have the encrypted files and look for a way to decrypt them in OSX, you need to find a way to extract the actual key and iv from the old password (not sure what algorithm was used in old version of openssh, but I think it was some kind of simple hash, will require some research).