Fastest way searching specific files

Question

I got thousands of files with a specific file extension in thousands of sub folders. Now, what is the fastest way to search with a pattern? I tried the method DirectoryInfo.GetFiles(rootfolder) (~8 minutes) and a recursive custom method (~5 minutes).

 private void WalkDirectoryTree(DirectoryInfo dr, string searchname)
 {
 System.IO.FileInfo[] files = null;
 System.IO.DirectoryInfo[] subDirs = null;
 try
 {
 files = dr.GetFiles(searchname + ".*");
 }
 catch (Exception ex)
 {
 MessageBox.Show(ex.Message);
 }
 if (files != null)
 {
 foreach (FileInfo fi in files)
 {
 allFiles.Add(fi);
 }
 subDirs = dr.GetDirectories();
 foreach (DirectoryInfo di in subDirs)
 {
 WalkDirectoryTree(di, searchname);
 }
 }
 }

Is there any faster way to do it?

• Heslacher: What type is allFiles ? Do you need FileInfo or the filename ?
• greenhoorn: It's a List<FileInfo>. I need the FileInfo, not only the filename.
• Heslacher: An how do you use the List<> ? Do you just iterate over the entries and process them in some way?
• greenhoorn: Yes that's it. It's not really relevant I think.
• Heslacher: yes it is relevant

Answer 1

Try parallelization. Instead of:

foreach (DirectoryInfo di in subDirs)
{
 WalkDirectoryTree(di, searchname);
}

Do

Parallel.ForEach(subDirs, dir => WalkDirectoryTree(dir, searchname));

Notice that by doing this allFiles will be accessed concurrently so change your collection to a ConcurrentBag.

• Bruno Costa: @greenhoorn I would be glad to hear the improvements you had after the replacement.
• greenhoorn: I will share it as soon as I test it. Could take 2 weeks :-)
• greenhoorn: Thanks for your answer! It still takes 2 minutes but that's significantly faster than before. I would say 2 minutes for millions of files is acceptable :-)

Answer 2

You can just use DirectoryInfo.EnumerateFiles() method which returns an IEnumerable<FileInfo> and therefor if you access them by the enumerator they will be evaluated when they are accessed.

private void WalkDirectoryTree(DirectoryInfo dr, string searchname)
{
 foreach (FileInfo file in FindFiles(dr, searchname + ".*"))
 {
 // process file
 allFiles.Add(file);
 }
}
public IEnumerable<FileInfo> FindFiles(DirectoryInfo startDirectory, string pattern)
{
 return startDirectory.EnumerateFiles(pattern, SearchOption.AllDirectories);
}

• greenhoorn: So the method GetFiles() does some more processing than EnumerateFiles(), do I get this right? I will give this a try too, thanks.
• Heslacher: No, but EnumerateFiles returns the single FileInfo's only if they are accessed (foreach... or calling .ToList())
• Bruno Costa: @Heslacher Lazyness does not solve slowness. And in this scenario it will be the same as a non lazy algorithm because you are evaluating all items imediatly and putting them in a list (wich can be done. like you said, with the ToList() method).