Anlyzing PSK-TLS handshake (Handshake Finished record) in Wireshark

Question

I am doing testing with some ethernet device, for which I use an own TLS implementation (using OpenSSL for the actual cryptographic functions). There are pre shared keys used. When I am connecting to the device using the cipher suite TLS_PSK_WITH_AES_256_GCM_SHA384, it aborts the handshake with a bad record MAC alert.

When analyzing the handshake in Wireshark, I set the applied PSK, Wireshark correctly decrypts my Client Handshake Finished record, see below screenshot. Now I am wondering: Would this work if there was something wrong (possibly with the hash calculation) on my (client) side? Or is there rather something wrong on the server side?

I am meanwhile pretty confident in my implementation. Usually TLS_PSK_WITH_AES_128_GCM_SHA256 is used and there are no problems with that.

Answer 1

In the case of AES-GCM, a bad_record_mac alert means that the server cannot verify the integrity of the Finished message based on the GCM authentication tag.

It's still possible for Wireshark to decrypt the message. As far as I can tell from the source code, Wireshark uses the libgrypt library which has separate functions for decrypting the ciphertext (gcry_cipher_decrypt) and verifying the authentication tag (gcry_cipher_checktag). The latter function doesn't seem to be called in Wireshark when decrypting TLS records, so a tag-related issue won't be recognized until the record reaches the server.

To find the exact problem, I would go through the following steps:

• Is the authentication tag that the TLS client has calculated for the Finished message correct? Log the AES key, the message and the tag in your client, then calculate the tag again with something like the OpenSSL CLI and compare the results.
• If the tag itself is correct, does the client assemble the Finished message and the TLS record correctly?
• Has the record been been corrupted in transit?
• If none of this is the case, check whether the server correctly calculates and verifies the authentication tag.