Tuning ls, ls -l, find ., find . -ls performance for very large dirs (60000+ files/dir)

Wed Apr 9 07:04:02 GMT 2025

On Tue, Apr 08, 2025 at 10:36:20PM -0600, Brian Inglis via Cygwin wrote:
> On 08/04/2025 19:05, Mark Liam Brown via Cygwin wrote:
> > Are there tuning variables to improve ls, ls -l, find ., find . -ls
> > performance for very large dirs?
> > 
> > If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
> > minutes (22+mins), while cmd.exe dir just floods the terminal with
> > results immediately.
>> I just remembered noticing my /var/log/ entries jumped looking at my last
> Cygwin upgrade log, so before I clean it up into subdirs, let's try some
> stuff!
>> If you just want a directory name dump, use \ls -U|--sort=none or \ls -f
> which is like -aU; command prefix \ overrides the usual aliases like -CF
> --color=auto:
>> $ time \ls -1U /var/log/ | \wc -lwmcL
> 54049 54049 1885787 1885787 38
>> real	0m0.336s
> user	0m0.046s
> sys	0m0.295s
> $ time \ls -1f /var/log/ | \wc -lwmcL
> 54051 54051 1885792 1885792 38
>> real	0m0.347s
> user	0m0.140s
> sys	0m0.232s
>> even with -l, it's only a few seconds:
>> $ time \ls -lU /var/log/ | \wc -lwmcL
> 54050 486455 4966716 4966716 102
>> real	0m15.891s
> user	0m1.421s
> sys	0m14.560s
> $ time \ls -lf /var/log/ | \wc -lwmcL
> 54052 486473 4966835 4966835 102
>> real	0m15.858s
> user	0m1.405s
> sys	0m14.374s
>> and skipping user and group lookup does not save anything:
>> $ time \ls -gloU /var/log/ | \wc -lwmcL
> 54050 378357 3777638 3777638 80
>> real	0m15.699s
> user	0m1.562s
> sys	0m14.155s
> $ time \ls -gGlf /var/log/ | \wc -lwmcL
> 54052 378371 3777713 3777713 80
>> real	0m15.908s
> user	0m1.389s
> sys	0m14.498s
>> -- 
> Take care. Thanks, Brian Inglis Calgary, Alberta, Canada

That reminds me:
OP should run cygserver to cache the AD LDAP queries.
That will likely have a measurable impact on performance.
https://cygwin.com/cygwin-ug-net/using-cygserver.html
Cheers, Glenn