option to create file hash and store in exported json file. #271

sakerk commented 2026年02月18日 04:35:52 +01:00 (Migrated from code.blicky.net)

Hello I have built a pretty slick web interface front end that is built on ncdu for the backend scan component. I can display the tree scans and use to manage storage over time. Search for files, types etc. Its pretty amazing how fast ncdu is and we have tried everything.. from treesize pro to diskover. We have multiple petabyte isilon clusters that I am now using ncdu with to build scans and composite them into a single unified view. Once a project is complete we archive it to s3 or tape. A project can span multiple locations with people working on it. As a result there are lots of duplicate files. I was thinking I could use ncdu to create a scan and store a file hash inside the exported json file. Then I can build build some code ontop to compare scan and tell me where duplicate file paths are and then use that to consoldiate into a new single archive. Does a hash feature exist and I am overlooking it? Is this something that could be implemented? I know it would slow things down, but for long term archive scans I am not so concerned on how long the scans would take as once the scan is complete thats it. I do not need to scan again as the archive is read-only at that point but having the hash done with ncdu and stored within the json file woud be gold . Thank you !

Hello I have built a pretty slick web interface front end that is built on ncdu for the backend scan component. I can display the tree scans and use to manage storage over time. Search for files, types etc. Its pretty amazing how fast ncdu is and we have tried everything.. from treesize pro to diskover. We have multiple petabyte isilon clusters that I am now using ncdu with to build scans and composite them into a single unified view. Once a project is complete we archive it to s3 or tape. A project can span multiple locations with people working on it. As a result there are lots of duplicate files. I was thinking I could use ncdu to create a scan and store a file hash inside the exported json file. Then I can build build some code ontop to compare scan and tell me where duplicate file paths are and then use that to consoldiate into a new single archive. Does a hash feature exist and I am overlooking it? Is this something that could be implemented? I know it would slow things down, but for long term archive scans I am not so concerned on how long the scans would take as once the scan is complete thats it. I do not need to scan again as the archive is read-only at that point but having the hash done with ncdu and stored within the json file woud be gold . Thank you !

sakerk commented 2026年02月18日 20:03:32 +01:00 (Migrated from code.blicky.net)

Thinking about it.. I am just going to run this as another job.

1. ncdu scan runs produces the JSON
2. A post-processor reads the JSON, finds duplicate candidates (same name + size)
3. Reads only those candidate files from the mounted scan-targets, computes hashes using something xxhash
4. Writes the hash field back into the matching entries in the same JSON file

Thinking about it.. I am just going to run this as another job. 1. ncdu scan runs produces the JSON 2. A post-processor reads the JSON, finds duplicate candidates (same name + size) 3. Reads only those candidate files from the mounted scan-targets, computes hashes using something xxhash 4. Writes the hash field back into the matching entries in the same JSON file