optimize this java code?

Question

I have a quartz scheduler job which scans a directory every 10 seconds and convert all pcl files to pdf files.

Here I put part of the code which scans the directory and loads all the files and for each of it it calls some methods to convert it.

 FileExtensionFilter extFilter = new FileExtensionFilter();
 extFilter.setAllowAll(false);
 extFilter.addExtension(_fExtension);
 // Filter out all files that match.
 File[] filteredFileList = fInbox.listFiles(extFilter);
 for (File fSrc : filteredFileList) {
 try {
 //call the methods to convert fSrc file
 }catch (Exception e){
 //threat exception's code
 }

All works nice if I have 30 pcl files but if I have 50 for instance, the processing goes step by step slower.

The question is: is there any way to improve this loop? Basically to not "wait" for the file to be processed but go further and take another one and so one?

I'm thinking about multithreading but I am not sure if this will work on this filesystem scan conversion task... Can you suggest something?

Thanks.

Ps: Please note that If I throw 30 files, and after that another 30 all works great. The performance is affected when I throw at once more files...

Answer 1

Throttle it so it only picks 20 files to do each 10 seconds. If you throw 50 files at it, it will do 20, and remove those, then do 20 more, then remove those, then do 10 more and remove those.

It may take a little longer (some downtime) but you won't screw up your scheduler.

• Cristian Boariu: Well..that's a nice approach!
• corsiKa: You could also keep track of your time. In the loop, stop after 9500 milliseconds, and let the next "run" take care of it.
• Byron Whitlock: +1. Also filesystem conversion tasks usually don't benenifit from multithreading. This is because the bottleneck isn't cpu, it is IO.

Answer 2

I'd suggest taking a look at jNotify. It will allow you to get notified at the exact time when a file is appears/is modified/deleted in a directory. This way, you can start processing when the file appears, and this could be better than processing all files at once.

This will be a part of JDK core in Java 7. This article will tell you more about it. The java.nio.file package has a WatchService API to support this.

Although, as others have said, your problem is more I/O bound than CPU bound. If you're doing a lot of conversions, a faster hard drive, possibly a SSD, will be of greater help.