A high-performance duplicate file finder
- Java 100%
|
Tad
a02ad1ba10
Make builds more reproducible
https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives Signed-off-by: Tad <tad@spotco.us> |
||
|---|---|---|
| gradle | Enable strict dependency verification | |
| src | Don't try to read the whole file at once | |
| .gitignore | Do the hashing | |
| .gitlab-ci.yml | Do the CI | |
| build.gradle | Make builds more reproducible | |
| gradle.properties | Enable strict dependency verification | |
| LICENSE | Add the LICENSE | |
| README.md | Note the prebuilts | |
jensame
A high-performance duplicate file finder. Meant for use with btrfs/xfs + duperemove utility.
Use
- gradle assemble
- java -jar -XX:+UseShenandoahGC jensame.jar $fdupesOutput $pathsToRecurse
- cat $fdupesOutput | sudo duperemove -d -b32768 --fdupes -v
- Example output:
- Found 10049077 files, hashed 802450 files, totalling 388119MB, and identified 691490 duplicates in 656920ms at 590MBps
Prebuilts
TODO
- User defined thread count
- Default thread count determined by storage medium
- Alternate hash algorithms
Credits
- Zero-Allocation Hashing (Apache-2.0), https://github.com/OpenHFT/Zero-Allocation-Hashing