How to embed a shell command into a sed expression?

Question

I have a text file with the following format:

keyword value
keyword value
...

Where keyword is a single word and value is everything else until the end of line. I want to read the file from a shell script, in a way that the values (but not the keywords) undergo shell expansion.

With sed it's easy to match the keywords and value parts

input='
keyword value value
keyword "value  value"
keyword `uname`
'

echo "$input"|sed -e 's/^\([^[:space:]]*\)[[:space:]]\(.*\)$/k=<\1> v=<\2>/'

which produces

k=<keyword> v=<value value>
k=<keyword> v=<"value  value">
k=<keyword> v=<`uname`>

but then the question is how can I embed a shell command into the replacement part of the sed expression. In this case I would like the replacement to be \1 `echo \2`.

Answer 1

Having GNU sed you can use the following command:

sed -nr 's/([^ ]+) (.*)/echo "\1" \2\n/ep' input

Which outputs:

keyword value value
keyword value  value
keyword Linux

with your input data.

Explanation:

The sed command suppresses regular output using the -n option. -r is passed to use extended regular expressions which saves us some escaping of special chars in the pattern but it is not required.

The s command is used to transfer the input line into the command:

echo "\1" \2

The keyword gets quoted, and the value not. I pass the option e - which is GNU specific - to the s command, which tells sed to execute the result of the substitution as a shell command and read its results into the pattern buffer (Even multiple lines). Using the option p after(!) e makes sed printing the pattern buffer after the command has been executed.

Answer 2

Standard sed can't call a shell (GNU sed has an extension to do it, if you only care about non-embedded Linux), so you'll have to do some of the processing outside sed. There are several solutions; all require careful quoting.

It's not clear exactly how you want the values to be expanded. For example, if a line is

foo hello; echo $(true)  3

which of the following should the output be?

k=<foo> value=<hello; echo   3>
k=<foo> value=<hello; echo   3>
k=<foo> value=<hello; echo 3>
k=<foo> value=<foo hello
  3>

I'll discuss several possibilities below.

pure shell

You can get the shell to read the input line by line and process it. This is the simplest solution, and also the fastest for short files. This is the closest thing to your requirement “echo \2”:

while read -r keyword value; do
  echo "k=<$keyword> v=<$(eval echo "$value")>"
done

read -r keyword value sets $keyword to the first whitespace-delimited word of the line, and $value to the rest of the line minus trailing whitespace.

If you want to expand variable references, but not execute commands outside command substitutions, put $value inside a here document. I suspect that this is what you were really looking for.

while read -r keyword value; do
  echo "k=<$keyword> v=<$(cat <<EOF
$value
EOF
)>"
done

sed piped into a shell

You can transform the input into a shell script and evaluate that. Sed is up to the task, though it's not that easy. Going with your “echo \2” requirement (note that we need to escape single quotes in the keyword):

sed  -e 's/^ *//' -e 'h' \
     -e 's/[^ ]*  *//' -e 'x' \
     -e 's/ .*//' -e "s/'/'\\\\''/g" -e "s/^/echo 'k=</" \
     -e 'G' -e "s/\n/>' v=\\</" -e 's/$/\\>/' | sh

Going with a here document, we still need to escape the keyword (but differently).

{
  echo 'cat <<EOF'
  sed -e 's/^ */k=</' -e 'h' \
      -e 's/[^ ]*  *//' -e 'x' -e 's/ .*//' -e 's/[\$`]/\\&/g' \
      -e 'G' -e "s/\n/> v=</" -e 's/$/>/'
  echo 'EOF'
 } | sh

This is the fastest method if you have a lot of data: it doesn't start a separate process for each line.

awk

The same techniques we used with sed work with awk. The resulting program is considerably more readable. Going with “echo \2”:

awk '
  1 {
      kw = $1;
      sub(/^ *[^ ]+ +/, "");
      gsub(/\047/, "\047\\\047\047", $1);
      print "echo \047k=<" kw ">\047 v=\\<" $0 "\\>";
  }' | sh

Using a here document:

awk '
  NR==1 { print "cat <<EOF" }
  1 {
      kw = $1;
      sub(/^ *[^ ]+ +/, "");
      gsub(/\\\$`/, "\\&", $1);
      print "k=<" kw "> v=<" $0 ">";
  }
  END { print "EOF" }
' | sh

Answer 3

You could try this approach:

input='
keyword value value
keyword "value  value"
keyword `uname`
'

process() {
  k=$1; shift; v="$*"
  printf '%s\n' "k=<$k> v=<$v>"
}

eval "$(printf '%s\n' "$input" | sed -n 's/./process &/p')"

(if I get your intention right). That is insert "process" at the beginning of each non-empty line to make it a script like:

process keyword value value
process keyword "value  value"
process keyword `uname`

to be evaluated (eval) where process is a function that prints the expected message.

Answer 4

If a non-sed solution is acceptable, this PERL snippet will do the job:

$ echo "$input" | perl -ne 'chomp; /^\s*(.+?)\s+(.+)$/ && do { $v=`echo "$2"`; chomp($v); print "k=<$1> v=<$v>\n"}'

Answer 5

ONLY KISS SHORT PURE SED

I will do this

echo "ls_me" | sed -e "s/\(ls\)_me/\1/e" -e "s/to be/continued/g;"

and it working.