grep files from list

Question

I am trying to run grep against a list of a few hundred files:

$ head -n 3 <(cat files.txt)
admin.php
ajax/accept.php
ajax/add_note.php

However, even though I am grepping for a string that I know is found in the files, the following does not search the files:

$ grep -i 'foo' <(cat files.txt)
$ grep -i 'foo' admin.php
The foo was found

I am familiar with the -f flag which will read the patterns from a file. But how to read the input files?

I had considered the horrible workaround of copying the files to a temporary directory as cp seems to support the <(cat files.txt) format, and from there grepping the files. Shirley there is a better way.

Answer 1

You seem to be grepping the list of filenames, not the files themselves. <(cat files.txt) just lists the files. Try <(cat $(cat files.txt)) to actually concatenate them and search them as a single stream, or

grep -i 'foo' $(cat files.txt)

to give grep all the files.

However, if there are too many files on the list, you may have problems with number of arguments. In that case I'd just write

while read filename; do grep -Hi 'foo' "$filename"; done < files.txt

• dotancohen: Thank you! I did not realize that while could receive the lines of file.txt as such.
• Stéphane Chazelas: You'll want to disable the glob part of that split+glob operator here (unless the shell is zsh).
• PM 2Ring: while isn't exactly receiving the lines from the file, read is doing that;while just lets us do that in a loop. The loop ends when read fails (i.e. returns a non-zero return code), normally due to the End Of File being reached.
• Stéphane Chazelas: To read a (text) line, the syntax is IFS= read -r filename, read filename is something else.
• Stéphane Chazelas: Note that -H is a GNU extension. You're missing some --.

Answer 2

xargs grep -i -- foo /dev/null < files.txt

assuming files are blank or newline delimited (where quotes or backslashes can be used to escape those separators). With GNU xargs you can specify the delimiter with -d (which then disables the quoting handling though).

(unset -v IFS; set -f; grep -i -- foo $(cat files.txt))

assuming files are space, tab or newline separated (no way to escape those though you can choose a different separator by assigning it to IFS). That one will fail if the file list is too big on most systems.

Those also assume that none of the files are called -.

• jimmij: It is better/faster to use $(< file) instead of $(cat file), at least in bash and zsh.
• Znik: for limited files number $(anything) is ok, but remember. this expands command line. for many files it will exceed this one. For that situation only xargs will do properly, because it can split files into sensible groups. btw. of course, $(< file) works better, because you ommit unnesesary cat call. Unfortunately saving is extremaly little.

Answer 3

To read a list of file names from stdin you can use xargs. E.g.,

cat files.txt | xargs -d'\n' grep -i -- 'foo'

By default, xargs reads items from the standard input, delimited by blanks. The -d'\n' tells it to use newline as the argument delimiter, so it can handle file names containing blanks. (As Stéphane Chazelas points out, that's a GNU extension). However, it won't cope with file names containing newlines; we'd need a slightly more complicated approach to handle those.

FWIW, this approach is somewhat faster than a while read loop, as bash's read command is very slow - it reads its data character by character, whereas xargs reads its input more efficiently. Also, xargs only invokes the grep command as many times as it needs to, with each invocation receiving multiple file names, and that's more efficient than invoking grep individually for each file name.

See the xargs man page and the xargs info page for further details.

• Znik: this is traditional use, when file list is separated by new line char. then -d'\n' is not usable in this context. But if file names contains special characters and or new lines, consider use null char separator. It is supported by find and xargs commands.

Answer 4

xargs can read items from a file (like your files.txt list) with it's option:

 --arg-file=file
 -a file
 Read items from file instead of standard input. If you use this
 option, stdin remains unchanged when commands are run. Other‐
 wise, stdin is redirected from /dev/null.

So this should work too:

xargs -a files.txt grep -i 'foo'

or for spaces in filenames

xargs -d'\n' -a files.txt grep -i 'foo'
xargs -I{} -a files.txt grep -i 'foo' {}