ls --color | grep doesn't work

Question

I'm trying to list all hidden dirs with the following command

ls -lhAF1 | grep -E '^d.*[0-9]{2}:[0-9]{2} \.'

which works perfectly fine

Explanation for Regex: I'm trying to get all rows that have the following Format: d, then some text, then the timestamp, then a space, then the dot and then more text

However when I try to color the ls output with this command:

ls --color -lhAF1 | grep -E '^d.*[0-9]{2}:[0-9]{2} \.'

It gives zero results, the output without --color is:

drwxr-xr-x 1 User Group 4096 Feb 1 08:48 .invisible

Why does ls / grep behave this way?

• Anthon: where is the difference between the two invocations of ls ...?
• danielr1996: @Anthon sorry, just copy pasted and forgot to add --colors in second code
• Anthon: I thought so, but didn't want to guess wrong and edit that in myself in case there was some commandline issue.
• Manu H: You may control if you call the shell built in comands or the system-wide one by giving the full path. e.g. '/usr/bin/ls -lhAF1' instead of 'ls -lhAF1'. They may behave differently
• Skaperen: ls is at /bin/ls.

Answer 1

--color adds escape sequences for the color. You can see this if you redirect the output (of ls --color) to a file.

This is what it looks like:

 drwxr-xr-x 6 root root 4.0K Jan 9 08:23 ^[[01;34m.cabal^[[0m/

To account for this, try this instead:

 ls -lhAF1 --color | grep -E '^d.*[0-9]{2}:[0-9]{2} .*\.'

• danielr1996: Thanks, that as exactly the problem, but it only works for dirs that do not contain a dot, but dirs like hello.world would also be accepted, I tried it with ^d.*[0-9]{2}:[0-9]{2} \^\[\[.*\., which works in RegExr, but not in my shell
• danielr1996: I came one step closer to the solution: grep $'\x1b' matches the ASCII Escape, but this doesn't work with double quotes around

Answer 2

Parsing ls is often a bad idea. Often, but not always. Here's another suggestion for you, which collects the required directories together before passing the set to ls.

find .* -maxdepth 0 -type d \( -name '.[^.]' -o -name '.??*' \) -exec ls -ld --color=always {} +

It's been pointed out that the original code actually limits the list of directories to those modified in the last six months. This can be handled with the following alternative solution.

find .* -maxdepth 0 -type d -mtime -180 -mtime +0 \( -name '.[^.]' -o -name '.??*' \) -exec ls -ld --color=always {} +

_{As ever, if your find doesn't understand the trailing + replace it with \; at the cost of efficiency.}

• Stéphane Chazelas: Presumably with his grep for HH:MM, the OP wanted only the files last modified between 6 months ago and now, for which you'd also need a -mtime -180 -mtime +0
• Chris Davies: @StéphaneChazelas ah, I missed that, thank you. It would have been easier if this had been included in the description of the requirement, of course. (I like the -mtime +0.)
• Stéphane Chazelas: Likely it was an oversight of the OP, see the discussion on his own answer.

Answer 3

The reason is that ls always colorizes its output even if it is connected to a terminal. From man ls:

 --color[=WHEN]
 colorize the output. WHEN defaults to 'always' or can be
 'never' or 'auto'. More info below

Many other tools such as grep do not retain colors when standard output is terminal but for some reasons ls was designed to act differently. Colorizing output is achieved by using ANSI escape codes that are interpreted by your terminal. Redirect ls output to a file and open it in editor:

$ ls --color -lhaF1 > /tmp/RESULT
$ less /tmp/RESULT
total 12K
drwxr-xr-x 3 ja users 4.0K Feb 2 09:47 ESC[0mESC[01;34m.ESC[0m/
drwxrwxrwt 12 root root 4.0K Feb 2 09:51 ESC[30;42m..ESC[0m/
drwxr-xr-x 2 ja users 4.0K Feb 2 09:47 ESC[01;34m.invisibleESC[0m/
-rw-r--r-- 1 ja users 0 Feb 2 08:35 a|a

If you have $LESS variable set you probably need to unset it before running less in order to see raw escape codes instead of color. So, to sum up, when using --color what grep gets is not what you see but a bunch of escape codes together with it. To fix it either don't use --colors at all or set it to --auto:

$ ls --color=auto -lhaF1 | grep -E '^d.*[0-9]{2}:[0-9]{2} \.'

Answer 4

Ok I finally figured out how to list all hidden dirs while preserving the Colors and not including dirs like "hello.world":

 ls -lhAF1 --color | grep -E "^d[rwx-]{9}.*[0-9]{2}:[0-9]{2} "$'\x1b'"\[([0-9]{1,2}m?)(;[0-9]{1,2}m?)?\."

• Stéphane Chazelas: Note that rwx are not the only permissions, especially for directories where t and s are common (see /tmp for instance). Only files last modified between 6 months ago and now have their time displayed with HH:MM (in the C and most English locales at least).
• danielr1996: @StéphaneChazelas well the permissions can be included easily, but how is the timestamp displayed when they were modified more than 6 month ago?
• Stéphane Chazelas: more than 6 months ago or in the future. Check ls -l /bin for examples. -rwxr-xr-x 1 root root 31240 Mar 14 2015 uname in a en_GB.UTF-8 locale on a GNU system,

Answer 5

Rather than applying grep to the output of ls which is not post-processable reliably anyway, I'd rather filter on the list of files passes to ls, like:

ls -lFhd --color -- .*(/)

In zsh. Or as your own approach suggests you only want the directories last modified between 6 months ago and now and not the ones with special permissions:

ls -lFhd --color -- .*(/mM-6mM+0f-7000)

• danielr1996: if I excute this it throws the error "Syntax error near unexpected token (" Note: I'm actually using Git Bash / Cygwin on Windows
• Stéphane Chazelas: @danielr1996, yes, that's for zsh, not bash. Check cygwin for a better Unix-like environment for Windows (includes zsh and bash).
• danielr1996: Oh well, you wrote that it's zsh, the Problem is that in our Company we can only use git bash and i was just playing arround. But at home I'm actually using zsh, can you tell me how the feature is called you're using Just as a side note: isn't git bash built on cygwin?
• Stéphane Chazelas: @daniel1996, they are glob qualifiers. Not sure about git-bash. I had heard it was outdated versions of some GNU utilities built for MS-Windows. It may be different now.