Flush 'tail -f' buffer?

Question

I'm using the following statement (simplified version):

tail -f -c+1 <filename>

in order to pipe a file to a process.

What I've found, though, is that there is a number of lines at the end which are not piped.

A specific example is piping a mysql file, and stopping when the end is reached:

tail -f -c+1 mysqdump.sql | sed '/^-- Dump completed/ q0'

This doesn't work - the last line of the dump -- Dump completed [...] is not piped to sed.

My guess is that the tail -f buffer, in this case, is flushed only when it's full.

Does anybody know how can I workaround this?

=================

I've found the cause - the description is not complete (and the code doesn't exhibit the behavior).

The problem happens when piping from a compressed (lzma) file:

tail -f -c+1 <filename.lzma> | lzma -d | sed '/^-- Dump completed/ q0'

Most likely, tail is not sending the last compressed block, because it doesn't detect any new line, as the input is binary.

• jofel: See e.g. stackoverflow.com/questions/5295430/… for an answer.
• PersianGulf: I don't understand your means.before tail and after tail what do you do?
• cas: and especially read the BashFAQ entry linked to in that stackoverflow question, mywiki.wooledge.org/BashFAQ/009

Answer 1

tail -f flushes after every input line. You can confirm this with strace (or truss or whatever your unix variant provides to trace processes' system calls).

If there is an incomplete line, tail -f keeps waiting for the next newline. It's a tool designed for text files. If you need to tail a binary file (e.g. if -- Dump completed is not followed by a newline), you'll have to use a custom tool.

If you've redirected sed's output away from the terminal, it will be doing its own buffering. Try stdbuf or unbuffer.

• Marcus: I was in fact executing on a binary file being remotely copied (I purposely don't want to use "ssh cat"). I worked the problem around using scp on a named pipe.
• agate: For sed you can use -u if you don't want buffer

Answer 2

Example using stdbuf -o0. Reference: https://superuser.com/a/1123674/126847

tail -F cpu.log | stdbuf -o0 tr '=' ' ' | stdbuf -o0 awk '6ドル > 1'

Answer 3

One problem with the command invoked in the question is that sed is invoked in buffering mode.

tail -f -c+1 <filename.lzma> | lzma -d | sed '/^-- Dump completed/ q0'

You need to add the -u flag:

tail -f -c+1 <filename.lzma> | lzma -d | sed -u '/^-- Dump completed/ q0'

For more info, read about the -u switch on the sed man page.