Why doesn't setting IFS shell variable split string into words in a for-loop, unless $(echo ...) is used?

Question

I would assume that setting IFS='X' would cause bash to split the string fooXbarXbaz into three words: foo, bar, and baz.

However, it only works if the string is provided to the for-loop through command substitution: $(echo fooXbarXbaz):

$ IFS='X'; for x in fooXbarXbaz; do echo Y${x}Z; done
Yfoo bar bazZ
$ IFS='X'; for x in $(echo fooXbarXbaz); do echo Y${x}Z; done
YfooZ
YbarZ
YbazZ

Can someone explain why the first example command fails to split fooXbarXbaz into three words, while the second example is successful?

• Hauke Laging: You can also use IFS='X'; vars=fooXbarXbaz; for x in $vars; do echo Y${x}Z; done

Answer 1

$IFS is only used for word splitting after unquoted expansions. There is no expansion in for x in fooXbarXbaz. There is however an unquoted expansion in echo Y${x}Z, which means that echo is called with the three arguments Yfoo, bar and bazZ. The echo utility prints each of its arguments with a space in-between, so you get Yfoo bar bazZ.

In the second example, the string is the output of a command substitution, which is an expansion, so the string is split for the loop.

The IFS variable is also used for splitting the input to the read utility.

• Shuzheng: Does IFS-based word splitting only occur after expansions in general?
• Kusalananda: @Shuzheng Yes, correct. I have corrected the text to say "unquoted expansions". Quoted expansions are not split. Compare echo $s and echo "$s" with IFS=X; s=helloXworld
• fra-san: @Shuzheng (If I'm correctly reading your question) the (only, AFAIK) use of IFS for splitting that may not be occurring after an expansion is with the read utility. Try playing with (IFS=X; read x y; echo "x: $x"; echo "y: $y").
• Kusalananda: @fra-san That is a correct reading of the manual. I did not mention this as it was not applicable on this particular question, the way I read it. That's splitting input rather than words in a variable or a string.
• fra-san: @Kusalananda I agree. Actually, my "If I'm correctly reading your question" only referred to the question in the OP's comment to your answer - I haven't been really clear, sorry.

Answer 2

Can someone explain why the first example command fails to split fooXbarXbaz into three words

Because shells aren't supposed to work like that any more.

In some ancient times, word splitting happened for plain words too, e.g. with heirloom-sh:

heirloom-sh$ IFS=o
heirloom-sh$ for foo in 123o456o789; do echo $foo; done
ech: not found
ech: not found
ech: not found

Oops, I mean:

heirloom-sh$ for foo in 123o456o789; do "echo" $foo; done
123
456
789

(Even there the keywords and variable names didn't get split on the os.)

While using o or X in IFS is a bit extreme, it's probably for the best that splitting now only happens for expansions.

If you use splitting on purpose, remember that unquoted expansions are also processed for globbing, so things like * may cause surprises. Arrays usually work better for handling multiple values.

Answer 3

The GNU bash documentation clearly explains this behavior under Command Substitution

If the substitution appears within double quotes, word splitting and filename expansion are not performed on the results.

Since the second case has a a unquoted substitution, the resultant string fooXbarXbaz is subjected to word splitting by the shell by the value of IFS set, which splits the word into multiple words.

In the first case for x in fooXbarXbaz;, the string is subjected to pathname expansion (aka glob expansion). The shell performs this at a stage much later than word-splitting, after finishing all possible expansions (see Shell Expansions), so the resultant string is treated literally.

To add further take the case of variable expansion $var, which happens before pathname expansion, undergoes same process like your OP

IFS='X'; var=fooXbarXbaz; for x in $var; do echo Y${x}Z; done

An unquoted variable expansion like command substitution undergoes word splitting by the value of IFS. That's why generally we insist on quoting all expansions to avoid unforseen word splitting done by the shell.

• Stéphane Chazelas: Also remember that an unquoted expansion is split + glob. If you only want the splitting, you need to remove the quotes, but also to disable globbing. So: IFS=X; set -o noglob; for x in $var; do printf '%s\n' "Y${x}Z"; done (also remember: don't use echo to display arbitrary content).

Answer 4

An explanation for why IFS behaves the way it does is given in an article by Bash maintainer Chet Ramey. It confirms that IFS was originally used to split all words on the command line, and it was changed for security reasons.

Bash has closed a long-standing shell security hole by not using the $IFS variable to split each word read by the shell, but splitting only the results of expansion (ksh and the 4.4 BSD sh have fixed this as well).

Source: What's GNU: Bash - The GNU Shell

• muru: Well, yes, that was already noted in unix.stackexchange.com/a/587877/70524
• michaeljsalo: Some may find helpful clarification in the authoritative source provided.
• Community: Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.