Can't Break the while read loop even if condition is met

Question

This should be simple but I am missing something, need some help. My requirement is to read the log file via tail to get latest logs, grep Download Config & Copying all files of and write it in MyOwnLogFile.log but I want this to stop as soon as .myworkisdone file appears in /usr/local/FOLDER/ One thing is sure that .myworkisdone will be generated at the last when all logs are done... but the script just continues to read the log file and never comes out of it, even if the file is created.

while [[ ! -e /usr/local/FOLDER/.myworkisdone ]];
do
 sudo tail -f -n0 /var/log/server22.log | while read line; do echo "$line" | grep -e 'Downloading Config’ -e ‘Copying all files of' ; done >> /var/tmp/MyOwnLogFile.log
done

I also tried until instead of while to check the file but still the script cant break the spell of reading the log file. Thank you in advance.

• ilkkachu: tail -f will follow the file indefinitely, the condition in the outer while is pretty much checked only once. If you want to terminate the tail when the .myworkisdone file appears, you'll have to arrange for it to be killed explicitly. But that sounds a bit awkward. Would it be possible to to first wait for the file to appear, and then grep through the complete log file, without tail -f?
• JamesHumam: Thanks, the purpose is to show log to the support team members so showing logs live is important. :)
• cas: shell is, as usual, exactly the wrong tool for this job. Use perl's File::Tail module - it's made for exactly this (monitoring a log file and performing actions when patterns are matched). BTW, if you're using a while/read loop in some variant of sh, you're almost certainly doing it wrong. See Why is using a shell loop to process text considered bad practice?

Answer 1

As said by @ikkachu, tail -f is an infinite process. However you may use the inner loop to initiate the breaking:

while :
do
 tail -f server.log | while IFS= read -r line
 do
 printf '%s\n' "$line" | grep 'word' >> ownlog.log
 if [ -e killer.file ] ; then break ; fi
 done
 break 
done

• Why does break 2 in the if-loop not work?

The pipe initiates a subshell, the break command cannot exit this subshell. Hence another break in the outer loop.

• When does the script stop?

Once the killer.file is existing, the pending | while read ... still awaits another entry and holds open the whole thing. I.e. you need the next line, i.e. (with dummy files):

 echo test >> server.log
 #"normal" output - nothing happens
 echo word >> server.log
 #matching output - logged to ownlog.log
 
 touch killer.file
 #killer file appears - breaks cannot take effect
 #as "read" still awaits signal.
 echo test2 >> server.log
 #break takes effect

This is especially important if the killer.file would also indicate that there are no more logs written to server.log, so the process remains constantly open.

• Chris Davies: For bash you could use read with a timeout
• JamesHumam: hi there, this solution made much sense to me so I tried this first. Works as expected. Thanks a lot. This doesnt mean other solutions don't work. I just didnt try those. Please try and see what suits you best. :)

Answer 2

You need to separate the checking for the "stop file" from the reading of the log file, because tail -f will not exit. The following shell script does that by having a background task check for the existence of the "stop file" once a second. Whenever the file appears, it kills the tail -f process with a PIPE signal (the same signal it would have received if grep had stopped reading). When tail -f terminates due to the PIPE signal, the grep process terminates too since its input stream closes.

#!/bin/sh
stopfile=/usr/local/FOLDER/.myworkisdone
set -- tail -f /var/log/server22.log 
"$@" | {
 while true; do
 if [ -e "$stopfile" ]; then
 pkill -PIPE -fx "$*"
 break
 fi
 sleep 1
 done &
 grep -e 'Downloading Config' \
 -e 'Copying all files of'
} >/var/tmp/MyOwnLogFile.log

This sets the positional parameters to the tail -f command. This makes it a bit easier to refer to it with pkill later. The pkill utility is called with both -f and -x to be able to kill exactly the command that we started earlier.

The value "$*" is a single string consisting of all positional parameters (the tail -f command) with a space in-between them (by default).

Using bash and a named array for the tail -f command:

#!/bin/bash
stopfile=/usr/local/FOLDER/.myworkisdone
talicmd=( tail -f /var/log/server22.log )
"${tailcmd[@]}" | {
 while true; do
 if [ -e "$stopfile" ]; then
 pkill -PIPE -fx "${tailcmd[*]}"
 break
 fi
 sleep 1
 done &
 grep -e 'Downloading Config' \
 -e 'Copying all files of'
} >/var/tmp/MyOwnLogFile.log

(I'm also noticing that a couple of the single quotes around your regular expressions used with grep in the question are not ordinary single quotes but "fancy quotes".)

• JamesHumam: Hi, thanks for help. This sounds promising but I couldnt try, I think in future releases, I'd have time to test and try this. Thanks for help, again!

Answer 3

perl has a module called File::Tail designed to monitor log file(s), so your program can take actions if conditions are met (e.g. printing the line if it matches a pattern).

 #!/usr/bin/perl
 use File::Tail;
 tie *FH,"File::Tail",(name=>shift,nowait=>1);
 while (<FH>) {
 print if (m/Downloading Config|Copying all files of/);
 exit if ( -e '/usr/local/FOLDER/.myworkisdone')
 }

save as, e.g., watch.pl, make it executable with chmod +x watch.pl and run it as:

$ ./watch.pl /var/log/server22.log

or just embed it as a one-liner in a shell script, same as you would for an awk or sed script:

perl -MFile::Tail -e 'tie *FH,"File::Tail",(name=>shift,nowait=>1);
 while (<FH>) {
 print if (m/Downloading Config|Copying all files of/);
 exit if ( -e '/usr/local/FOLDER/.myworkisdone')
 }' /var/log/server22.log

This monitors the log file named as the first arg to the script (name=>shift). It prints any logfile lines that match the pattern, and exits if the .myworkisdone file exists.

nowait=>1 prevents the script from blocking while reading the input file. It's less than ideal, but OK for a quick-and-dirty script. A better solution would be to use the File::Tail's select() method. See man File::Tail.

BTW, the select() method also makes it easy to monitor multiple log files at once.

• JamesHumam: Thanks a lot for help. The 2nd solution worked at the moment so I'd go ahead with that but in my nxt release for script I'll try what you've suggested. Thanks again. :)