How does grep decide that a file is binary?

Question

I have a large utf-8 text file which I frequently search with grep. Recently grep began reporting that it was a binary file. I can continue to search it with grep -a, but I was wondering what change made it decide that the file was now binary.

I have a copy from last month where the file is no longer detected as binary, but it's not practical to diff them since they differ on> 20,000 lines.

file identifies my file as

UTF-8 Unicode English text, with very long lines

How can I find the characters/lines/etc. in my file which are triggering this change?

---

The similar, non-duplicate question 19907 covers the possibility of NUL but grep -Pc '[\x00-\x1F]' says that I don't have NUL or any other ANSI control chaarcters.

• ott--: I would try this in this order: 1. Run it with strace/ltrace to check what input causes that 'binary' message 2. Check out grep's source and read it
• Charles: @muru: I'm using gnu grep, but if you have the answer for some other version I'd be interested as well.
• muru: Odd. I have a file which I know contains a nul and some Escs. I tried grepping for them. I could find the escs (\x1B), but the nul never showed up. The test given above showed 1, for the line containing Escs, but nothing for any range that didn't contain \x1B. I wouldn't trust that test. Try grep -zc . instead (should be one more than the number of nuls in your file). (Also, you might be better off using [[:cntrl:]].)
• muru: Also try: sed -z 's/.*\(....\)$/1円/' foo | od -c to see a few characters before the NUL (if there is one), which might lead you to the problem.
• Charles: @muru: My sed doesn't have a -z option: sed: invalid option -- 'z'.

Answer 1

It appears to be the presence of the null character in the file.(displayed ^@ usually) I entered various control characters into a text file(like delete, ^?, for example), and only the null character caused grep to consider it a binary. This was only tested for grep. The less and diff commands, for instance, may have different methods. Control characters in general don't appear except in binaries. The exceptions are the whitespace characters: newline(^M), tab(^I), formfeed(^L), vertical tab(^K), and return(^J).

However, foreign characters, like arabic or chinese letters, are not standard ascii, and perhaps could be confused with control characters. Perhaps that's why it's only the null character.

You can test it out for yourself by insterting control characters into a text file using the text editor vim. Just go to insert mode, press control-v, and then the control character.

Answer 2

A typical modern grep implementation should only declare a file "binary" if there are nul bytes inside. Anything else should be OK.

I cannot speak for the grep implementation you use...

Answer 3

An encoding error according to mbrlen() also makes GNU grep 2.24 consider it as binary

E.g.:

export LC_CTYPE='en_US.UTF-8'
printf 'a\x80' | grep 'a'

because \x80 cannot be the first byte of an UTF-8 Unicode point: https://en.wikipedia.org/wiki/UTF-8#Description

This is the only other possibility besides NUL.

GNU grep source code interpretation that leads to this conclusion: What makes grep consider a file to be binary?

• Thomas Guyot-Sionnest: That wasn't the case in v2.20, it would print a regardless of LANG and LC_*. Now grep v3.1 prints a when not using an UTF-8 locale, else it detects as binary. I also suspect 2.20 checked only once on a chunk of the file as I could grep without issue files that had null chunks in them, v3.1 now stops printing matches with the "Binary file ... matches" message. This is annoying as these chunks are common on files written concurrently by multiple processes (ex. ~/.bash_history) and lone nulls don't cause any issue on a terminal... :(