Unable to decrypt aes-256-gcm encrypted data in java

Question

I have encrypted a file by using OpenSSL aes-256-gcm. As aes-256-gcm not directed supported by command line I have installed LibreSSL and I am able to use the below command to encrypt the data of a file.

openssl enc -aes-256-gcm -K 61616161616161616161616161616161 -iv 768A5C31A97D5FE9 -e -in file.in -out file.out

I need to decrypt the data of file.out in Java and I am unable to do that.

Sample code :

    // Get Cipher Instance
    Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");

    String key = "61616161616161616161616161616161";
    byte[] IV = "768A5C31A97D5FE9".getBytes();

    // Create SecretKeySpec
    SecretKeySpec keySpec = new SecretKeySpec(key.getBytes("UTF-8"), "AES");

    // Create GCMParameterSpec
    GCMParameterSpec gcmParameterSpec = new GCMParameterSpec(128, IV);

    // Initialize Cipher for DECRYPT_MODE
    cipher.init(Cipher.DECRYPT_MODE, keySpec, gcmParameterSpec);

    // Perform Decryption
    byte[] decryptedText = cipher.doFinal(cipherText); // for the data by reading file.out

However, I am getting an exception saying javax.crypto.AEADBadTagException: Tag mismatch!

Answer 1

That should't work. Commandline openssl enc doesn't support AEAD ciphers/modes, although early versions of 1.0.1 (below patch h, in 2012-2014) failed to catch if you incorrectly specified such a cipher and silently produced wrong output. If you are actually using LibreSSL and not OpenSSL, it appears to have inherited this problem and not fixed it, even though the whole point of the LibreSSL project was that they were going to fix all the bugs caused by the incompetent OpenSSL people.

If this were a cipher that worked correctly in OpenSSL (and also Java), like aes-256-ctr, then your only problem would be that openssl enc -K -iv take their arguments in hex (suitable for a shell context), whereas Java crypto is called from code that can handle binary data and expects its arguments in that form. As a result the values you provide to OpenSSL are actually 16 bytes (128 bits) and 8 bytes (64 bits), not 256 bits and 128 bits as they should be (for CTR; for GCM an IV of 96 bits would be correct, but as noted GCM won't work here). openssl enc automatically pads -K -iv with (binary) zeros, but Java doesn't. Thus you would need something more like

 byte[] key = Arrays.copyOf( javax.xml.bind.DatatypeConverter.parseHexBinary("61616161616161616161616161616161"), 32);
 // Arrays.copyOf zero-pads when expanding an array
 // then use SecretKeySpec (key, "AES")
 // and IVParameterSpec (iv) instead of GCMParameterSpec

 // but after Java8 most of javax.xml is removed, so unless you
 // are using a library that contains this (e.g. Apache) 
 // or have already written your own, you need something like

 byte[] fromHex(String h){
   byte[] v = new byte[h.length()/2];
   for( int i = 0; i < h.length(); i += 2 ) v[i] = Integer.parseInt(h.substring(i,i+2),16);
   return v;
 }

Compare AES encrypt with openssl command line tool, and decrypt in Java and Blowfish encrypt in Java/Scala and decrypt in bash (the latter is the reverse direction, but the need to match is the same)