How can I handle raw binary data in a bash pipe?

Question

I have a bash function that takes a file as a parameter, verifies the file exists, then writes anything coming off stdin to the file. The naive solution works fine for text, but I am having problems with arbitrary binary data.

echo -n '' >| "$file" #Truncate the file
while read lines
do # Is there a better way to do this? I would like one...
 echo $lines >> "$file"
done

Answer 1

Your way is adding line breaks to every thing that it write in space of whatever separator ($IFS) is using to split up the read. Instead of breaking it up into newlines just take the whole thing and pass it along. You can reduce the entire bit of code above to this:

 cat - > $file

You don't need the truncate bit, this will truncate and write the whole STDIN stream out to it.

Edit: If you are using zsh you can just use > $file in place of the cat. You are redirecting to a file and truncating it, but if there is anything hanging out there waiting for something to accept STDIN it will get read at that point. I think you can do something like this with bash but you would have to set some special mode.

• David Souther: I couldn't get the stdin redirect example to work, but changing the cat example to >| (I have noclobber set) works like a charm. Thanks for making my day ^.^
• rozcietrzewiacz: +1 for the cat-less version. Always avoid useless cats ;)
• Caleb: @rozcietrzewiacz: True, except it was an afterthought and I was wrong. This might not be a useless use of cat. The only thing you might be able to do is > $file. This only works as the first thing that looks for stdin in the parent shell script. Basically all of David's code can be reduced to a single character, but I think the cat - is more elegant and less trouble prode because it's understood on sight.
• Michael Mrozek: Sometimes I string four or five cats together, just to annoy UUOC fanatics
• Caleb: @MichaelMrozek: Sometimes I name my data files cat just so people who insist on using it necessarily have to do mental gymnastics to read the code. Named pipes are also good targets.

Answer 2

To read a text file literally, don't use plain read, which processes the output in two ways:

• read interprets \ as an escape character; use read -r to turn this off.
• read splits into words on characters in $IFS; set IFS to an empty string to turn this off.

The usual idiom to process a text file line by line is

while IFS= read -r line; do ...

For an explanation of this idiom, see Why is while IFS= read used so often, instead of IFS=; while read..?.

To write a string literally, don't just use plain echo, which processes the string in two ways:

• On some shells, echo processes backslash escapes. (On bash, it depends whether the xpg_echo option is set.)
• A few strings are treated as options, e.g. -n or -e (the exact set depends on the shell).

A portable way of printing a string literally is with printf. (There's no better way in bash, unless you know your input doesn't look like an option to echo.) Use the first form to print the exact string, and the second form if you want to add a newline.

printf %s "$line"
printf '%s\n' "$line"

This is only suitable for processing text, because:

• Most shells will choke on null characters in the input.
• When you've read the last line, you have no way to know if there was a newline at the end or not. (Some older shells may have bigger trouble if the input doesn't end with a newline.)

You can't process binary data in the shell, but modern versions of utilities on most unices can cope with arbitrary data. To pass all input through to the output, use cat. Going on a tangent, echo -n '' is a complicated and non-portable way of doing nothing; echo -n would be just as good (or not depending on the shell), and : is simpler and fully portable.

: >| "$file"
cat >>"$file"

or, simpler,

cat >|"$file"

In a script, you usually don't need to use >| since noclobber is off by default.

• David Souther: thanks for pointing out xpg_echo, that's actually a problem I was having somewhere else in my code and didn't even realize. Re noclobber, I am in the habit of turning it on in my bashrc.

Answer 3

This will do exactly what you want:

( while read -r -d '' ; do
 printf %s'0円' "${REPLY}" ;
 done ;
 # When read hits EOF, it returns non-zero which exits the while loop.
 # That data still needs to be output:
 printf %s "${REPLY}"
) >> ${file}

Do note the memory usage though. This reads input in a null-delimited fashion.

If there are no 0円 null bytes in the input then bash will first need to read the entire contents of input into memory, and then output it.

Regarding your truncate step:

echo -n '' >| "$file" #Truncate the file

a much simpler and equivalent is:

> ${file} #Truncate the file