How to loop over the lines of a file?

Question

Say I have this file:

hello
world
hello world

This program

#!/bin/bash
for i in $(cat 1ドル); do
 echo "tester: $i"
done

outputs

tester: hello
tester: world
tester: hello
tester: world

I'd like to have the for iterate over each line individually ignoring whitespaces though, i.e. the last two lines should be replaced by

tester: hello world

Using quotes for i in "$(cat 1ドル)"; results in i being assigned the whole file at once. What should I change?

Answer 1

With for and IFS:

#!/bin/bash
IFS=$'\n' # make newlines the only separator
set -f # disable globbing
for i in $(cat < "1ドル"); do
 echo "tester: $i"
done

Note however that it will skip empty lines as newline being an IFS-white-space character, sequences of it count as 1 and the leading and trailing ones are ignored. With zsh and ksh93 (not bash), you can change it to IFS=$'\n\n' for newline not to be treated specially, however note that all trailing newline characters (so that includes trailing empty lines) will always be removed by the command substitution.

Or with read (no more cat):

#!/bin/bash
while IFS= read -r line; do
 echo "tester: $line"
done < "1ドル"

There, empty lines are preserved, but note that it would skip the last line if it was not properly delimited by a newline character.

• Tobias Kienzler: thanks, I didn't know one could < into a whole loop. Although it makes perfectly sense now I saw it
• Grzegorz Wierzowiecki: I see IFS \ read -r line' in second example. Is really IFS=` needed ? IMHO it enough to say : while read -r line; do echo "tester: $line"; done < "1ドル"
• Gilles 'SO- stop being evil': @GrzegorzWierzowiecki IFS= turns off the stripping of leading and trailing whitespace. See In while IFS= read.., why does IFS have no effect?
• fra-san: @BenMares To prevent globbing expressions possibly appearing in the text we are reading from being expanded to matching file names. Try, for instance, printf '%s\n' '*o' 'bar' >afile; touch foo; IFS=$'\n'; for i in $(cat afile); do echo "$i"; done.
• user232326: A while IFS= read -r line || [ "$line" ]; do will process a trailing line not properly delimited by a newline character (but it will be added back).

Answer 2

(9 years later:)
Both provided answers would fail on files without a newline at the end, this will effectively skip the last line, produce no errors, would lead to disaster (learned hard way:).

The best concise solution I found so far that "Just Works" (in both bash and sh):

while IFS='' read -r LINE || [ -n "${LINE}" ]; do
 echo "processing line: ${LINE}"
done < /path/to/input/file.txt

For more in-depth discussion see this StackOverflow discussion: How to use "while read" (Bash) to read the last line in a file if there’s no newline at the end of the file?

Beware: this approach adds an additional newline to the last line if there is none already.

• Stéphane Chazelas: files with characters after the last newline are not text files, and those characters don't constitute a line. In many cases those bogus characters are better left ignored or removed, though there are cases where you may want to treat it as a extra line, so it's good you show how.
• Stéphane Chazelas: Note that files with NUL characters in them are also not text files, and except in zsh that loop would also fail if the intention was to keep them as if they were allowed in text files.

Answer 3

If you can avoid it, don't especially if it's to process text.

Most text utilities are already designed to process text one line at a time, and, at least for the GNU implementations, do it efficiently, correctly and handle error conditions nicely. Piping one to another which runs them in parallel also means you can leverage more than one processor to do the job.

Here:

<input.txt sed 's/^/tester /' > output.txt

Or:

<input.txt awk '{print "tester", 0ドル}' > output.txt

More on that at: Why is using a shell loop to process text considered bad practice?

If it's not about text processing and you do need to run some command per line of a file, also note GNU xargs where you can do:

xargs -rd'\n' -I@ -a input.txt cp -- @ @.back

for instance.

With the bash shell, you can get each line of a file into an array with the readarray builtin:

readarray -t lines < input.txt &&
 for line in "${lines[@]}"; do
 do-some-non-text-processing-you-cannot-easily-do-with-xargs "$line" || break
 done

POSIXly, you can use IFS= read -r line to read one line off some input, but beware that if you redirect the whole while read loop with the input file on stdin, then commands inside the loop will also have their stdin redirected to the file, so best is to use a different fd which you close inside the loop:

while
 IFS= read -r line <&3 ||
 [ -n "$line" ] # to cover for an unterminated last line.
do
 {
 do-some-non-text-processing-you-cannot-easily-do-with-xargs "$line" ||
 break # abort upon failure if relevant
 } 3<&-
done 3< input.txt > output.txt

read -r line removes leading and trailing whitespace characters from the line that it reads provided they are in the $IFS variable, though only the yash shell honours that POSIX requirement. With most shells, that's limited to space and tab. ksh93 and recent versions of bash do it for all single-byte characters considered as whitespace in the locale.

So to read a line and also strip leading and trailing blanks, you can do: IFS=$' \t' read -r line. With ksh93, yash1 or recent versions of bash. IFS=$' \t\r' would also strip the trailing CR character found in text files from the Microsoft world.

---

^{1 though yash doesn't support the $'...' syntax yet, you'd need IFS=$(printf ' \t\r') there.}

Answer 4

For what it is worth, I need to do that quite often, and can never remember the exact way of using while IFS= read..., so I defined the following function in my bash profile:

# iterate the line of a file and call input function
iterlines() {
 (( $# < 2 )) && { echo "Usage: iterlines <File> <Callback>"; return; }
 local File=1ドル
 local Func=2ドル
 n=$(cat "$File" | wc -l)
 for (( i=1; i<=n; i++ )); do
 "$Func" "$(sed "${i}q;d" "$File")"
 done
}

This function first determines the number of lines in the file, then uses sed to extract line after line, and passes each line as a single string argument to any given function. I suppose this might get really inefficient with large files, but that hasn't been a problem for me so far (suggestions on how to improve this welcome of course).

The usage is pretty sweet IMO:

>> cat example.txt # note the use of spaces, whitespace, etc.
a/path
This is a sentence.
"wi\th quotes"
$End
>> iterlines example.txt echo # preserves quotes, $ and whitespace
a/path
This is a sentence.
"wi\th quotes"
$End
>> x() { echo "$#"; }; iterlines example.txt x # line always passed as single input string
1
1 
1
1
1

Answer 5

To read all the lines, regardless of whether they are ended with a new line or not:

cat "somefile" | { cat ; echo ; } | while read line; do echo $line; done

Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/

• Kamil Maciorowski: It is not acceptable to post identical answers to multiple questions.
• Nathan S.R.: Hi Kamil Maciorowski, Answers are identical because Questions are identical too !

Answer 6

There is an answer here that may be useful to people who this doesn't work for

loop through file by row in tcsh

• muthuh: Not an answer as these usually contain some sort of explanation and/or a procedure to follow. A comment at best but not even a good one?