Duplik

Duplik is a small tool to detect duplicate files. Duplicate detection is done based on Blake 3 hashes of files combined with their sizes, ignoring empty files. Duplicates will be output as JSON; the idea being that this tool can easily integrate with jq as part of a more sophisticated processing pipeline. This tool is only tested with *Nix based systems

duplik:main λ duplik -D 4 -H . | jq
[
 {
 "hash": "18d1af8d48726adde098078c26107358e0f3af5c609804ac5592c2ee1e7e4b23",
 "bytes": 451592,
 "paths": [
 ".zig-cache/o/91cbe7e9e527f5f6363b5565027d7d98/duplik",
 "zig-out/bin/duplik"
 ]
 },
 {
 "hash": "c09afee0c9f361fb61e5ff28a7739893de766fb470c5fa82b4e5e31de27fbad4",
 "bytes": 9,
 "paths": [
 ".zig-cache/tmp/6eF86OTTwYx7SkkC/one.test",
 ".zig-cache/tmp/rcysl3VUWmjMS1G7/two.test",
 ".zig-cache/tmp/rcysl3VUWmjMS1G7/one.test",
 ".zig-cache/tmp/9aONBAv5sGdqnu_Z/two.test",
 ".zig-cache/tmp/9aONBAv5sGdqnu_Z/one.test",
 ".zig-cache/tmp/mqCXxbRJhZ6zzHSo/two.test",
 ".zig-cache/tmp/mqCXxbRJhZ6zzHSo/one.test",
 ".zig-cache/tmp/L8nDmQLRwWZ-IVc3/two.test",
 ".zig-cache/tmp/L8nDmQLRwWZ-IVc3/one.test"
 ]
 }
]

Installation

1. Install Zig (0.16 is supported currently)
2. Clone this repo and cd into the repo root.
3. Run zig build -Doptimize=ReleaseFast to compile the executable
4. Copy zig-out/bin/duplik to somewhere on your PATH (e.g., ~/.local/bin).

Usage

usage: duplik [options] <directory path>
options:
 -D/--max-depth <n> sets maximum recursion depth; default: 0
 -A/--absolute-paths outputs the paths as absolute paths; default: False
 -F/--exclude-dir <dir> exclude some directory paths (exact match) from the recursive walk; can be specified multiple times
 -W/--workers <n> set number of workers (min 1); default: # of cores available
 -H/--include-hidden include hidden files; excluded by default
 -S/--min-size <bytes> minimum size in bytes for a file to be evaluated
 -M/--max-size <bytes> maximum size in bytes for a file to be evaluated

Some recipes:

# Recurse through the home directory, and output duplicate files larger than 1MB, skipping the file with the shortest path (assumed to be the original)
duplik -A -D 99 ~ | jq -r '.[] | select(.bytes >= 1048576) | .paths | sort_by(length) | .[1:] | .[]'
# Note that instead of the jq-based size post-filter above, you can also use the -S/--min-size option to let duplik ignore files 
# below a byte threshold. This is more performant (because duplik will hash fewer files), but less flexible.
duplik -A -D 99 -S 1048576 ~ | jq -r '.[] | .paths | sort_by(length) | .[1:] | .[]'
# Only consider files that are smaller than a megabyte (note: -M rather than -S).
duplik -A -D 99 -M 1048576 ~ | jq -r '.[] | .paths | sort_by(length) | .[1:] | .[]'
# Delete all duplicates up until a depth of 2, keeping only the first path in the list of duplicates (careful!)
duplik -A -D 2 ~ | jq -r '[.[].paths[1:]] | flatten | .[]' | xargs rm -f

Performance

Based on local benchmarking, duplik performs in a similar order of magnitude as best-in-class duplication detection tools like fclones and rmlint for most workloads.

Development philosophy

Duplik is developed without the use of LLMs. LLM-assisted contributions of any kind will not be accepted.