Bash replace multi-line pattern with an output of a command whose argument is the match

Question

I have a string in Bash and I want to replace parts of it that match a certain pattern that matches multi-line substrings with an output of a command/function executed with each matched substring as an argument.

The string that I have is the output of this command:

openssl s_client -showcerts -connect example.net:443

It looks like this (heavily edited for brevity and clarity):

…stuff…
-----BEGIN CERTIFICATE-----
…Base64
multiline
string…
-----END CERTIFICATE-----
…stuff…
-----BEGIN CERTIFICATE-----
…Base64
multiline
string…
-----END CERTIFICATE-----
…stuff…

I want to replace the sections starting with -----BEGIN CERTIFICATE----- and ending with -----END CERTIFICATE----- with the output of the following command:

openssl x509 -in certificate.crt -text -noout

This command takes an input which is the Base64 string in between the above two markers (including the markers) and outputs a textual representation of the Base64.

My desired substitution is one where the Base64 lines get replaced by the output of the above command.

This means I need:

• An ability to specify a multi-line pattern (the two markers and lines between them)
• An ability to use a Bash function as the replacer predicate
  • The function writes the matched substring to a file
  • The function then executes the above command with the temporary file at input
  • The function deletes the file
  • The function returns the output of the command

The desired output looks like this:

…stuff…
-----BEGIN CERTIFICATE-----
…Textual
multiline
output…
-----END CERTIFICATE-----
…stuff…
-----BEGIN CERTIFICATE-----
…Textual
multiline
output…
-----END CERTIFICATE-----
…stuff…

I have found a few solutions on how to do one and the other using sed but I was not able to combine them. Eventually, I managed to get this mix of Bash and Python, but I am interested in a pure Bash solution:

echo | openssl s_client -showcerts -connect example.net:443 | python3 -c "
import fileinput
import re
import subprocess
import os

stdin=''.join(fileinput.input())

def replace(match):
  with open('temp.crt', 'w') as text_file:
    print(match.group(), file=text_file)

  process = subprocess.run(['openssl', 'x509', '-in', 'temp.crt', '-text', '-noout'], capture_output=True, text=True)
  os.remove('temp.crt')

  return '-----BEGIN CERTIFICATE-----\n' + process.stdout + '\n-----END CERTIFICATE-----'

stdout=re.sub(r'-----BEGIN CERTIFICATE-----(\n(.*\n)+?)-----END CERTIFICATE-----', replace, stdin)

print(stdout)
"

I am trying to get something similar to this pseudocode to work:

echo \
| openssl s_client -showcerts -connect example.net:443 \
| sed 's/-----BEGIN CERTIFICATE-----\n.*?-----END CERTIFICATE-----/$(openssl x509 -in \0 -text -noout)/e'

I am not precious about using sed for this, but I also tried with awk and perl and did not get anywhere. Maybe OpenSSL can do this and I don't even need the substitution? Haven't found anything on that.

Is there a Bash one-line which can do all this?

Answer 1

The awk command was designed for the scenario you have outlined: complex, conditional, input massage.

Below is a generalized example addressing your problem.

#!/bin/sh

### QUESTION:   https://stackoverflow.com/questions/74295101/bash-replace-multi-line-pattern-with-an-output-of-a-command-whose-argument-is-th

### Script to parse input file for match on conditions to start/stop ignoring input
### for placement of output from custom batch command at appropriate places in the input.

TEMP="/tmp/tmp.$$.job"
COM_BATCH="${TEMP}.ssl"
COM_OUTPUT="${TEMP}.output"
TEST_INPUT="${TEMP}.input"

if [ "${1}" = "--debug" ] ; then  DBG=1 ; else  DBG=0 ; fi

cat >"${TEST_INPUT}" <<-!EnDoFiNpUt
…stuff…
…more stuff…
…extra stuff…
-----BEGIN CERTIFICATE-----
…Base64
multiline
string…
-----END CERTIFICATE-----
…stuff…
…more stuff…
…extra stuff…
-----BEGIN CERTIFICATE-----
…Base64
multiline
string…
-----END CERTIFICATE-----
…stuff…
…more stuff…
…extra stuff…
!EnDoFiNpUt


#cat >"${COM_BATCH}" <<-!EnDoFiNpUt
#rm -f '${COM_OUTPUT}'
#openssl s_client -showcerts -connect example.net:443 >'${COM_OUTPUT}'
#!EnDoFiNpUt

### Dummy command instead of openssl command identified above
cat >"${COM_BATCH}" <<-!EnDoFiNpUt
echo '\
------------------------------------
Output from oppenssl command
------------------------------------' >'${COM_OUTPUT}'
!EnDoFiNpUt

StartCondition="BEGIN CERTIFICATE"
EndCondition="END CERTIFICATE"

awk -v dbg="${DBG}" \
    -v subStrt="${StartCondition}" \
    -v subEnd="${EndCondition}" \
    -v sslCmd="${COM_BATCH}" \
    -v sslRes="${COM_OUTPUT}" '\
function external_action(command,result){
    doPrint=0 ;
    system( "chmod 700 "command" ; "command ) ;
    if( dbg == 1 ){ system( "ls -l "command" >&2 ; ls -l "result" >&2" ) ; } ;
    system( "cat "result ) ;
    #print sslCmd ;
}

BEGIN{
    doPrint=1 ;
}

{
    if( dbg == 1 ){ printf("\n\n\t\t INPUT LINE: %s\n", $0 ) ; };
    if( dbg == 1 ){ printf("\t\t Print FLAG: %s\n", doPrint ) ; };

    posS=index( $0, subStrt ) ;
    if ( posS > 0 ){
        if( dbg == 1 ){ printf("\t posS = %s\n", posS ) ; };
        external_action( sslCmd, sslRes ) ;
    } ;

    if( doPrint == 1 ){
        print $0 ;
    }else{
        posE=index( $0, subEnd ) ;
        if( posE > 0 ){
            if( dbg == 1 ){ printf("\t posE = %s\n", posE ) ; };
            doPrint=1 ;
        } ;
    } ;
}' <"${TEST_INPUT}"

Answer 2

Using GNU sed and bash:

openssl s_client -showcerts -connect example.net:443 |
sed '/^-----BEGIN CERTIFICATE-----$/!b
     :a
     N
     /\n-----END CERTIFICATE-----$/!ba
     s/.*/openssl x509 -in <(cat <<"EOF"\n&\nEOF\n) -text -noout/e
     s/.*/-----BEGIN CERTIFICATE-----\n&\n-----END CERTIFICATE----/'