bash script - sed - template file processing

Question

Using a bash script to process a file and pass it to AWS CLI. The script runs inside a docker container.

I'm using sed to do template string replace. When MY_VALUE contains special chars. (eg: MY_VALUE="http://aaa.com") the sed command fails.

This makes me question if sed is the right tool for this job, don't have a lot of control on the values that come in.

Because the template is a JSON file, I considered trying JQ or Python (something like this).

I haven't used bash or Unix tools a lot so I would like to hear your thoughts on:

1. Is sed the right tool for this type of job?
2. What else would you use?
3. What do you think about the script?

Simplified template (JSON)

{
 "networkMode": "bridge",
 "name": "[Placeholder.A]" 
}

Script

#!/bin/bash
MY_VALUE=${myvalue}
rm -f processed.json
cat /app/template.json | \
sed s/\\[Placeholder.A]/${MY_VALUE}/ > /app/processed.json
aws cli command -file processed.json

• fedorqui: Parsing a dictionary with sed is quite dangerous, many things can happen. Also, this is failing because your replacement contains slashes /, which are the character that sed uses to separate the blocks of a replacement: sed 's/find/replace/'. The first, easy, step is to change the separator into another one: sed 's_find_replace_' for example with _.
• thedev: I considered using a different separator, concerned I'll still end up with problematic replaces (the replacement values also contain _ )

Answer 1

You are parsing a JSON file using some UNIX tools. This is risky, since those can have so many edge cases that you will end up refactoring over and over.

For example, you mention that sed s/\\[Placeholder.A]/${MY_VALUE}/ is giving errors when $MY_VALUE has slashes. This is normal, since this is how it gets expanded:

sed s/\[Placeholder.A]/${MY_VALUE}/
 ↓ 
sed s/\[Placeholder.A]/http://aaa.com/
# ^ ^ ^^ ^

See? Too many slashes for a simple sed 's/find/replace/, thus it fails. You can solve this by using another separator:

$ sed 's/\[Placeholder.A\]/a/' file
{
 "networkMode": "bridge",
 "name": "a" 
}
$ MY_VALUE="http://aaa.com"
$ sed "s/\[Placeholder.A\]/$MY_VALUE/" file
sed: -e expression #1, char 27: unknown option to `s'
# Let's use ? as a separator, for example
$ sed "s?\[Placeholder.A\]?$MY_VALUE?" file
{
 "networkMode": "bridge",
 "name": "http://aaa.com" 
}

---

Instead, use jq, a command-line JSON processor. Given your file:

$ cat file
{
 "networkMode": "bridge",
 "name": "[Placeholder.A]" 
}

See how easy is to replace the value in the "name" key:

$ jq '.name="a"' file
{
 "networkMode": "bridge",
 "name": "a"
}

If you want to use a variable then you will need to do some quotes-opening-closing:

$ MY_VALUE="http://aaa.com"
$ jq '.name="'$MY_VALUE'"' file
{
 "networkMode": "bridge",
 "name": "http://aaa.com"
}

• janos: Escaping double-quotes inside double-quotes might be actually simpler in this case: jq ".name=\"$MY_VALUE\""