LibreSSL aes encryption incompatible with OpenSSL?

Question

Machine A has OpenSSL 1.0.2k and I encrypt a dummy file like this:

openssl enc -aes-256-cbc -salt -in plain.txt -e -pass pass:abcd1234 -out cipher.enc

Machine B has LibreSSL 3.3.3 and when I decrypt the file from machine A like this:

openssl enc -aes-256-cbc -in cipher.enc -d -pass pass:abcd1234 -out plain.txt

It fails to decrypt, giving this error:

bad decrypt
4444794304:error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt:evp/evp_enc.c:549:

Note that if I try to decrypt it using the same command line on machine A, it works fine.

I don't know how outdated OpenSSL 1.0.2k is but unfortunately I have no control over upgrading it (it's a remote 3rd party server).

Is there an incompatibility between OpenSSL and LibreSSL when using AES encryption?

Answer 1

Found the solution, posting here in case it helps anyone else facing the same problem.

Apparently, in a later version of OpenSSL (later LibreSSL) the default hashing algorithm for key derivation was changed from md5 to sha256.

More details here: https://github.com/libressl-portable/portable/issues/378

So now I explicitly specify the hashing algo in both command lines. To encrypt on Machine A:

openssl enc -aes-256-cbc -salt -in plain.txt -e -pass pass:abcd1234 -md sha256 -out cipher.enc

And to decrypt on Machine B:

libressl enc -aes-256-cbc -in cipher.enc -d -pass pass:abcd1234 -md sha256 -out plain.txt

And now it works fine.

[Edit] -- In case the default key derivation parameters may ever change in the future, or the old implicit defaults are considered insufficient at some point, perhaps it's best to explicitly define those as well. For example by including -pbkdf2 -iter 100000.
So then it becomes:

openssl enc -aes-256-cbc -salt -in plain.txt -e -pass pass:abcd1234 -md sha256 -pbkdf2 -iter 100000 -out cipher.enc

To encrypt, and to decrypt:

libressl enc -aes-256-cbc -in cipher.enc -d -pass pass:abcd1234 -md sha256 -pbkdf2 -iter 100000 -out plain.txt

(openssl and libressl can be used interchangeably in this way)