JSON array to bash variables using jq

Question

I've got a JSON array like so:

{
 "SITE_DATA": {
 "URL": "example.com",
 "AUTHOR": "John Doe",
 "CREATED": "10/22/2017"
 }
}

I'm looking to iterate over this array using jq so I can set the key of each item as the variable name and the value as it's value.

Example:

• URL="example.com"
• AUTHOR="John Doe"
• CREATED="10/22/2017"

What I've got so far iterates over the array but creates a string:

constants=$(cat ${1} | jq '.SITE_DATA' | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]")

Which outputs:

URL=example.com
AUTHOR=John Doe
CREATED=10/22/2017

I am looking to use these variables further down in the script:

echo ${URL}

But this echos an empty output at the moment. I'm guessing I need an eval or something in there but can't seem to put my finger on it.

Answer 1

Your original version isn't going to be evalable because the author name has spaces in it - it would be interpreted as running a command Doe with the environment variable AUTHOR set to John. There's also virtually never a need to pipe jq to itself - the internal piping & dataflow can connect different filters together.

All of this is only sensible if you completely trust the input data (e.g. it's generated by a tool you control). There are several possible problems otherwise detailed below, but let's assume the data itself is certain to be in the format you expect for the moment.

You can make a much simpler version of your jq program:

jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)'

which outputs:

URL='example.com'
AUTHOR='John Doe'
CREATED='10/22/2017'

There's no need for a map: .[] deals with taking each object in the array through the rest of the pipeline as a separate item, so everything after the last | is applied to each one separately. At the end, we just assemble a valid shell assignment string with ordinary + concatenation, including appropriate quotes & escaping around the value with @sh.

All the pipes matter here - without them you get fairly unhelpful error messages, where parts of the program are evaluated in subtly different contexts.

This string is evalable if you completely trust the input data and has the effect you want:

eval "$(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)' < data.json)"
echo "$AUTHOR"

---

As ever when using eval, be careful that you trust the data you're getting, since if it's malicious or just in an unexpected format things could go very wrong. In particular, if the key contains shell metacharacters like $ or whitespace, this could create a running command. It could also overwrite, for example, the PATH environment variable unexpectedly.

If you don't trust the data, either don't do this at all or filter the object to contain just the keys you want first:

jq '.SITE_DATA | { AUTHOR, URL, CREATED } | ...'

You could also have a problem in the case that the value is an array, so .value | tostring | @sh will be better - but this list of caveats may be a good reason not to do any of this in the first place.

---

It's also possible to build up an associative array instead where both keys and values are quoted:

eval "declare -A data=($(jq -r '.SITE_DATA | to_entries | .[] | @sh "[\(.key)]=\(.value)"' < test.json))"

After this, ${data[CREATED]} contains the creation date, and so on, regardless of what the content of the keys or values are. This is the safest option, but doesn't result in top-level variables that could be exported. It may still produce a Bash syntax error when a value is an array, or a jq error if it is an object, but won't execute code or overwrite anything.

• Charles Duffy: Use @sh to have JQ do eval-safe quoting. Adding \"s at the front and end is definitely not safe.
• Michael Homer: Indeed, this was not a good solution in the face of arbitrary data (and still isn't if the keys are untrusted), but it should certainly have been using @sh for the values.
• Michael Homer: Yes, you could write the (edited) comment's version inside a string literal expression, but I don't understand why you would. This is exactly the situation the formatter + interpolation construct is made for - add some formatted interpolations inside an otherwise-literal string (i.e. this is what "You can follow a @foo token with a string literal. The contents of the string literal will not be escaped. However, all interpolations made inside that string literal will be escaped" means). It certainly appears to work correctly.
• Charles Duffy: Ahh! Thank you for quoting the behavior above -- I now understand what this code is doing.
• user1274247: The @sh operation fails on multi-line values. Depending on other processing (see below), \n is either replaced by space or terminates the string.

Answer 2

Building on @Michael Homer's answer, you can avoid a potentially-unsafe eval entirely by reading the data into an associative array.

For example, if your JSON data is in a file called file.json:

#!/bin/bash
typeset -A myarray
while IFS== read -r key value; do
 myarray["$key"]="$value"
done < <(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + .value ' file.json)
# show the array definition
typeset -p myarray
# make use of the array variables
echo "URL = '${myarray[URL]}'"
echo "CREATED = '${myarray[CREATED]}'"
echo "AUTHOR = '${myarray[URL]}'"

Output:

$ ./read-into-array.sh 
declare -A myarray=([CREATED]="10/22/2017" [AUTHOR]="John Doe" [URL]="example.com" )
URL = 'example.com'
CREATED = '10/22/2017'
AUTHOR = 'example.com'

• Michael Homer: You could also indirect the assignment alone with declare -- "$key=$value" and have $AUTHOR etc work as in the original, without an array. It’s still safer than eval, though changing PATH or something is still possible so less so than this version.
• cas: yeah, the array nicely isolates the variables into a container of your choosing - no chance of accidentally/maliciously messing with important environment variables. you could make your declare -- version safe by comparing $key against a list of allowed variable names.
• milahu: to declare multiple associative arrays: echo '{"a":{"a1":1,"a2":2},"b":{"b1":1,"b2":2}}' | jq -r 'to_entries[] | "declare -A \(.key)=(\(.value | to_entries | map(@sh "[\(.key)]=\(.value)") | join(" ")))"'

Answer 3

Just realized that I can loop over the results and eval each iteration:

constants=$(cat ${1} | jq '.SITE_DATA' | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]")
for key in ${constants}; do
 eval ${key}
done

Allows me to do:

echo ${AUTHOR}
# outputs John Doe

Answer 4

I really like the @Michel suggestion. Sometimes, you may really just extract some variables value to execute a task in that specific server using Bash. So, if desired variables are known, using this approach is the way to avoid multiple calls to jq to set a value per variable or even using the read statement with multiple variables in which some can be valid and empty, leading to a value shift (that was my problem).

My previous approach that lead will lead to a value shift error – if .svID[ ].ID="", sv will get the slotID value.

-rd '\n' getInfo sv slotID <<< $(jq -r '(.infoCMD // "no info"), (.svID[].ID // "none"), (._id // "eeeeee")' <<< $data)

If you downloaded the object using curl, here is my approach to rename some variables to a friendly name as extract data from data arrays.

Using eval and filters will solve the problem with one line and will produce variables with the desired names.

eval "$(jq -r '.[0] | {varNameExactasJson, renamedVar1: .var1toBeRenamed, varfromArray: .array[0].var, varValueFilter: (.array[0].textVar|ascii_downcase)} | to_entries | .[] | .key + "=\"" + (.value | tostring) + "\""' <<< /path/to/file/with/object )" 

The advantage in this case, is the fact that it will filter, rename, format all the desired variables in the first step. Observe that in there is .[0] | that is very common to have if the source if from a RESTful API server using GET, response data as:

[{"varNameExactasJson":"this value", "var1toBeRenamed: 1500, ....}]

---

If your data is not from an array, i.e., is an object like:

{"varNameExactasJson":"this value", "var1toBeRenamed: 1500, ....}

just remove the initial index:

eval "$(jq -r '{varNameExactasJson, renamedVar1: .var1toBeRenamed, varfromArray: .array[0].var, varValueFilter: (.array[0].textVar|ascii_downcase)} | to_entries | .[] | .key + "=\"" + (.value | tostring) + "\""' <<< /path/to/file/with/object )"