bash escaping quotes

Question

I have an interesting challenge for how to escape quotes in a bash script.

My bash script has a long curl call with a large -d json structure passed.

#!/bin/bash

Value4Variable=Value4

curl -s -X POST -H "Content-Type: application/json" -H "Accept: application/json" -d  \
'{"Variable1":"Value1", 
"Variable2":"Value2\'s",  # escape of quote doesnt work because of the previous single quote and backslash
"Variable3":"Value3",
"Variable4":"'"$Value4Variable"'", 
"Variable5":"Value5"
}' \
https://www.hashemian.com/tools/form-post-tester.php

What's the best way to add a single quote into the json structure? I've tried various combinations but no success.

Answer 1

There are multiple ways to escape long strings with different quotes. The simplest is to end the single quote and escape the single quote:

'...'\''...'

But there are some nicer things you can do. Heredocs are a good way to avoid the quoting issue:

curl -s -X POST https://www.hashemian.com/tools/form-post-tester.php \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -d  @- <<EOF
{
    "Variable1":"Value1", 
    "Variable2":"Value2's",
    "Variable3":"Value3",
    "Variable4":"$Value4Variable", 
    "Variable5":"Value5"
}
EOF

@- tell curl to read from stdin and <<EOF starts the heredoc which will be feed into the stdin of curl. The nice thing with the heredoc is that you do not need to escape any quotes and can use bash variables inside them sidestepping the need to worry about how to escape the quotes and making the whole thing much more readable.

Answer 2

I would use --data @filename notation to read the data from file:

Sample json-data-file contents:

{"Variable1":"Value1", 
"Variable2":"Value2's",
"Variable3":"Value3",
"Variable4":"'$Value4Variable'", 
"Variable5":"Value5"
}

---

The request:

Value4Variable="Value4"
curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" \
-d @<(sed "s/\$Value4Variable/$Value4Variable/" json-data-file) \
https://www.hashemian.com/tools/form-post-tester.php/test123

The output:

The POSTed value is:

********************

{"Variable1":"Value1", "Variable2":"Value2's","Variable3":"Value3","Variable4":"'Value4'", "Variable5":"Value5"}

********************

Results saved to: test123
*Note: Results are purged periodically. 1 hr minimum life.

Answer 3

Consider using a JSON-aware tool to create JSON documents. Doing so would remove the need to manually escape special characters like tabs, newline, double quotes etc. A single quote is actually a character that doesn't need to be escaped in JSON. You just need to make sure you're using the correct quotes for the shell to recognize your string properly.

The jo tool from here is made to make creating JSON documents from the command line be really easy:

somevariable2='line1
line2
line3'

curl -s -X POST \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d "$( jo \
        variable1="Value1" \
        variable2="$somevariable2" \
        variable3='String with "quotes"' \
        variable4="It's raining!" \
        variable5='\/\/\/\/\/' )" \
    'https://www.hashemian.com/tools/form-post-tester.php'

or

somevariable2='line1
line2
line3'

jo \
    variable1="Value1" \
    variable2="$somevariable2" \
    variable3='String with "quotes"' \
    variable4="It's raining!" \
    variable5='\/\/\/\/\/' |
curl -s -X POST \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d @- \
    'https://www.hashemian.com/tools/form-post-tester.php'

The response for either piece of code is

The POSTed value is:

********************

{"variable1":"Value1","variable2":"line1\nline2\nline3","variable3":"String with \"quotes\"","variable4":"It's raining!","variable5":"/\\/\\/\\/\\/"}

********************