Looping through lines in file using bash and passing to variable. Resulting variable is not the same value as file, why?

Question

I have a bash script (just doing some simple log file pattern matching) that I have had running in a cron for a few years. Recently it broke and started returning odd results. Upon digging into the script and doing some debugging on it, I found that the problem seems to be with a while loop I do on a file.

To illustrate the problem I did a cat on the file 'cat /var/tmp/file' and get what I expect to get which is a time stamp and some (unformatted) IDs:

15:56:14,965 [,PCC12345678(PSI12345678),,]
18:08:43,706 [,PCC23456789(PSI23456789),,]
12:01:49,233 [,PCC34567891(PSI34567891),,]

When I put this in a while loop I would expect it to remain the same, but it does not. When I echo the line in the loop the second field (the IDs) is always changed to '1', like this:

cat /var/tmp/file | while read line
do
echo $line
done

It gives me an output of:

15:56:14,965 1
18:08:43,706 1
12:01:49,233 1

Which is obviously completely different to what is in the file and has left me baffled.

Things I've tried so far:

• I thought maybe the $line variable got stuck in memory so tried clearing that, didn't work.
• I tried the classic 'turn it off and on again' to the server for lack of better ideas thinking maybe it would clear out whatever might have got stuck.
• Tried running the script on two other servers, got the same problem.

Currently I am thinking maybe it doesn't like the format of the file, possibly something to do with the square brackets or commas. Although I am not sure why that would be the case or why it would suddenly be happening.

Note: Nothing has changed on either the script or the log files it is running against since being written. It was working previously for two years.

Edit: After suggestions I have checked what may have changed in the environment as far as I can tell its the same. Unless something has changed with an OS upgrade (which I wouldn't know how to check):

• .bash_profile and .profile have not been edited in over three years.
• I have tried to run the script in other shells ksh, bash, csh. Same problem encountered in all of them.
• I have tried running the script with other users including root. Again same issue with all of them.

Thanks Matt

• jai_s: I get the expected output when I run it (bash) - something else has changed in your environment. could you try it with ksh if you're on bash (or the other way around) ?
• MelBurslan: my regex fu is not strong enough to decipher what the string between square brackets are resolving to but in general, any string in between square brackets are treated as a selection list, if they are separated by commas. For instance a[1,2,3] is expanded into any one of a1 or a2 or a3. Since you are reading the whole line into a variable, line, bash is taking the square brackets as delimiters of a regex.
• MattM: Thanks @jai_s that is something I didn't consider. I did try your suggestions and also a few other things see my edits in my original posting however, it appears not to be the case.
• MattM: @MelBurslan sounds like you could be onto something there. I will try edit out the square brackets and commas to see if that helps the situation. However, it would be nice to know why this has suddenly started happening now and why it was working before. Also, if there is a way to get it working without having the edit the brackets and commas out.
• MattM: @MelBurslan Removing the square brackets did make the problem go away. Would be good to get an explanation of the full 'science' behind this and if there is a way around it or to override the behavior.

Answer 1

You have a file named 1 in the directory the script is running in.

As MelBurslan commented, [] has a special meaning for the shell, but it doesn't have much to do with regexes: it just means "a single character taken from any of the characters between the brackets". So when you run

echo 15:56:14,965 [,PCC12345678(PSI12345678),,]

the shell looks for a file named ,, or P, or C, or 1... If at least one file matches, [,PCC12345678(PSI12345678),,] is replaced with all the matching file names in the output; otherwise it's reproduced as is.

If you remove the 1 file the old behaviour should be restored. You can fix the script by protecting $line:

cat /var/tmp/file | while read line
do
echo "$line"
done

• MattM: Thanks @stephen that was exactly the issue I was having. I wonder if you are able to point me in the direction of some further reading on the square bracket's special meanings and their uses? I have never encountered this behavior before, I guess because I have always previously been in the habit of protecting my variables.
• Stephen Kitt: @MattM, check out the relevant section in the bash manual, and the examples in the globbing section of the Advances Bash-Scripting Guide.