Openssl signed with openssl pkeyutl verified with openssl dgst

Question

I want to sign hash with openssl pkeyutl and verify with openssl dgst -verify.

Down below are my testing private and public keys (EC keys):

private.pem:

-----BEGIN EC PARAMETERS-----
BggqhkjOPQMBBw==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIFYL7prqjwKcVpKp4VF0kSshVoNCu3QzFeJHjvq78n4FoAoGCCqGSM49
AwEHoUQDQgAEkcL/M+0hEuW/VUNCZT5Jc1pyw9gm4vphWldTdAqMJhC8eTiP/gao
rTkz6+iFfOPbJTEzRD8y36WqYAlS+65W8A==
-----END EC PRIVATE KEY-----

public.pem:

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEkcL/M+0hEuW/VUNCZT5Jc1pyw9gm
4vphWldTdAqMJhC8eTiP/gaorTkz6+iFfOPbJTEzRD8y36WqYAlS+65W8A==
-----END PUBLIC KEY-----

$ echo -n "123456" | openssl dgst -sha256 
8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92

S̶H̶A̶-̶2̶4̶5̶ SHA-246 of 123456 is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92.

Let's sign SHA256(123456) with pkeyutl

$ echo -n "8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92" | openssl pkeyutl -inkey private.pem -sign | openssl base64 -e -A
MEYCIQDIJHf2SQJliMNvPwgCqanzqWxleK/YGSCd15RK8IYPEQIhAOlcvXH9ASQRMRNgKgMr4ZZLL3nyaCsTHBeU0iReZMmp

So we have signature encoded in base64 MEYCIQDIJHf2SQJliMNvPwgCqanzqWxleK/YGSCd15RK8IYPEQIhAOlcvXH9ASQRMRNgKgMr4ZZLL3nyaCsTHBeU0iReZMmp

Now let's verify with openssl pkeyutl -verify

$ openssl pkeyutl -verify -pubin -inkey public.pem  -sigfile <(echo -n "MEYCIQDIJHf2SQJliMNvPwgCqanzqWxleK/YGSCd15RK8IYPEQIhAOlcvXH9ASQRMRNgKgMr4ZZLL3nyaCsTHBeU0iReZMmp" | openssl enc -A -base64 -d ) -in <(echo -n "8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92")
Signature Verified Successfully

So it has been verified successfully.

AND NOW the problematic part: So original data we wanted to sign is "123456". Let's use openssl dgst for this:

$ echo -n "123456" | openssl dgst -sha256 -verify public.pem -signature <(echo -n "MEYCIQDIJHf2SQJliMNvPwgCqanzqWxleK/YGSCd15RK8IYPEQIhAOlcvXH9ASQRMRNgKgMr4ZZLL3nyaCsTHBeU0iReZMmp" | openssl enc -A -base64 -d)
Verification Failure

Which returns Verification Failure.

Does openssl dgst do something else while creating sha256 digest from original "123456" text?

From my understanding it should work because pkeyutl signs the same sha256 hash as it is generated via openssl dgst -sha256. Here we are using the same private and public keys so I dont get why the openssl dgst -verify is not able to verify the signature genenrated via pkeyutl,

Answer 1

TLDR: IT'S THE HEX

SHA-245 [you meant SHA-256] of 123456 is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92

Not really. We often say this sort of thing as a shortcut or convention, but you appear to have misunderstood it. 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92 is the SHA-256 of 123456 displayed in hexadecimal meaning base-16 which we commonly abbreviate as hex, or we also call this 'encoded in' or 'represented as' hex. The actual hash value is 32 bytes of arbitrary bits which do not form characters and thus cannot be correctly displayed, or posted on Stack, or otherwise used by humans; that's why we almost always use hex or base64 to handle such data outside of programs and files. (To be even more exact, the hash is 256 bits which is only 32 bytes on a system or device that uses 8-bit bytes -- which not all do, but all the ones you encounter as an ordinary user do. In particular, all devices where OpenSSL works use 8-bit bytes, so for OpenSSL SHA-256 is 32 bytes.)

You need to either convert the hex representation back to the actual data -- for which there are several options on Unix covered by numerous Qs here or serverfault or unix.SX, but less on Windows (outside of WSL which effectively is Unix) -- or use dgst -binary which doesn't create the hex representation in the first place, and is thus simpler:

$ echo -n 123456 |openssl sha256 -binary |tee hash |od -tx1
0000000 8d 96 9e ef 6e ca d3 c2 9a 3a 62 92 80 e6 86 cf
0000020 0c 3f 5d 5a 86 af f3 ca 12 02 0c 92 3a dc 6c 92
0000040
$ openssl pkeyutl -sign -inkey private -in hash |openssl base64 -e -A |tee sig
MEUCIQCxK8aJ195ihqV/XJtwESXnpD7lQX47uppM8SD/JMehiAIgKDqNXvPfvC7SqbSJ3yBaqt3mJ+GIHD9sDP1YGBh9shA=
$ openssl pkeyutl -verify -pubin -inkey public -in hash -sigfile <( openssl base64 -d -A <sig )
Signature Verified Successfully
$ echo -n 123456 |openssl sha256 -verify public -signature <( openssl base64 -d -A <sig )
Verified OK
$

PS: note not all versions of echo support -n; see https://unix.stackexchange.com/questions/65803/why-is-printf-better-than-echo

Update re proposed edit by Vlado (which community rejected): when using openssl pkeyutl -sign for RSASSA-PKCS1-v1_5 signature (i.e. RSA but not RSA-PSS) you need -pkeyopt digest:sha256 to create, and with -verify to check, a standard-conforming signature which will interoperate with openssl sha256 -verify (and also -sign). That option is not needed for ECDSA signature, which was the question asked here, although it is good to be aware of the difference. (Also not needed for DSA, which few people today use. And not applicable to EdDSA, because neither pkeyutl -sign nor dgst -sign currently support that.)