Export file from live view #174

mcint commented 2021年07月24日 01:55:47 +02:00 (Migrated from code.blicky.net)

Using ncdu in anger often comes with time pressure. I would be wonderful to be able to export the state already scanned into a live session, even if it meant some delay.

I assume ncdu does not use json format in memory. What challenges should I be aware if I want make easier: exporting state from a live session (and importing state into a live session, and rescanning, I recall some hiccup in that workflow)?

Using ncdu in anger often comes with time pressure. I would be wonderful to be able to export the state already scanned into a live session, even if it meant some delay. I assume ncdu does not use json format in memory. What challenges should I be aware if I want make easier: exporting state from a live session (and importing state into a live session, and rescanning, I recall some hiccup in that workflow)?

mcint commented 2021年07月24日 01:58:01 +02:00 (Migrated from code.blicky.net)

It seems that maybe I should look at the Zig version, although language instability means I should zealously timebox any efforts.

It seems that maybe I should look at the Zig version, although language instability means I should zealously timebox any efforts.

yorhel commented 2021年07月24日 08:57:43 +02:00 (Migrated from code.blicky.net)

I mentioned this feature in the ncdu 2 release announcement (under "Next steps"). A few things to consider:

• The main implementation challenge is the counting of directory sizes. The JSON dump expects to have non-cumulative raw dir sizes, but the in-memory tree only stores cumulative sizes. It should be possible to subtract all file sizes and get the raw dir sizes back, the Zig version makes this a little bit easier (though, I do still have a few ideas for improving hard link counting that may in turn make it harder to subtract file sizes, so let's await that...)
• Where should the export be written to? Ncdu currently doesn't have any sort of text input widget available to it, and I'm not eager to implement one. :/
• This feature should probably not be available in double-readonly mode (-rr).
• Exporting to stdout isn't really an option for interactive mode, so no way to pipe through a compression command. This feature would benefit a lot if transparent export compression were implemented as well.

I don't think importing or rescanning an import makes much sense in interactive mode.

(EDIT: Also, as per https://dev.yorhel.nl/contributing.html, I'm more interested in a discussion to figure out a good implementation strategy than in seeing actual code - that's the easy part, usually. :))

I mentioned this feature in the [ncdu 2 release announcement](https://dev.yorhel.nl/doc/ncdu2) (under "Next steps"). A few things to consider: - The main implementation challenge is the counting of directory sizes. The JSON dump expects to have non-cumulative raw dir sizes, but the in-memory tree only stores cumulative sizes. It should be possible to subtract all file sizes and get the raw dir sizes back, the Zig version makes this a little bit easier (though, I do still have a few ideas for improving hard link counting that may in turn make it harder to subtract file sizes, so let's await that...) - Where should the export be written to? Ncdu currently doesn't have any sort of text input widget available to it, and I'm not eager to implement one. :/ - This feature should probably not be available in double-readonly mode (-rr). - Exporting to stdout isn't really an option for interactive mode, so no way to pipe through a compression command. This feature would benefit a lot if transparent export compression were implemented as well. I don't think importing or rescanning an import makes much sense in interactive mode. (EDIT: Also, as per https://dev.yorhel.nl/contributing.html, I'm more interested in a discussion to figure out a good implementation strategy than in seeing actual code - that's the easy part, usually. :))

mcint commented 2021年07月26日 05:59:08 +02:00 (Migrated from code.blicky.net)

Thanks for a quick great response! I see you have previously shared some of your thinking about export and import workflows, to address my questions.

I'm interested in rescans from imports, and rescans to include sub-trees beyond what was specified at invocation...

because when I reach for ncdu, it's often "in anger," and I'm trying to address an issue quickly. For example, low free space on a server. I appreciate that ncdu's TUI interface makes it easy to compare relative sizes (and ages) and quick to drill down into a file heirarchy. However, the initial scan time is a limiting factor, and quickly hits a limit determined by hardware (even with careful and clever programming).

Between invocations of ncdu, most of my file system sub-trees haven't changed significantly, but ncdu requires a full rescan. I would like to be able to import a scan from the last time I used it, or even to run a cron job to save a most recent snapshot (and limit the depth of the recorded* scan entries, or limit the maximum file or directory size explicitly included). I appreciate how good ncdu is for space management workflows, but I would like to be able to rely on it even more, and address some questions more quickly, with cached information.

I suppose addressing this use-case would require adding a (meta)data field for last-scanned time. I've considered working around this by making a temp directory for symlinking out to subtrees of interest, although it doesn't follow symlinks.

Does any of this sound easy or interesting for you? Thanks for linking your contribution guide.

Thanks for a quick great response! I see you have previously shared some of your thinking about export and import workflows, to address my questions. I'm interested in rescans from imports, and rescans to include sub-trees beyond what was specified at invocation... because when I reach for ncdu, it's often "in anger," and I'm trying to address an issue quickly. For example, low free space on a server. I appreciate that ncdu's TUI interface makes it easy to compare relative sizes (and ages) and quick to drill down into a file heirarchy. However, the initial scan time is a limiting factor, and quickly hits a limit determined by hardware (even with careful and clever programming). Between invocations of ncdu, most of my file system sub-trees haven't changed significantly, but ncdu requires a full rescan. I would like to be able to import a scan from the last time I used it, or even to run a cron job to save a most recent snapshot (and limit the depth of the recorded* scan entries, or limit the maximum file or directory size explicitly included). I appreciate how good ncdu is for space management workflows, but I would like to be able to rely on it even more, and address some questions more quickly, with cached information. I suppose addressing this use-case would require adding a (meta)data field for last-scanned time. I've considered working around this by making a temp directory for symlinking out to subtrees of interest, although it doesn't follow symlinks. Does any of this sound easy or interesting for you? Thanks for linking your contribution guide.

yorhel commented 2021年07月28日 14:12:54 +02:00 (Migrated from code.blicky.net)

Quoting myself:

though, I do still have a few ideas for improving hard link counting that may in turn make it harder to subtract file sizes, so let's await that...

Implemented that new idea on the zig branch now, turns out that it instead simplifies extracting directory sizes, so that shouldn't be a big challenge anymore. The other points remain, though.

As for refreshing an imported dir, that's fairly easy, but something that should absolutely be opt-in through a special cli flag because it's very prone to errors (refreshing on the wrong system, when the fs isn't mounted, or refreshing a year old export, etc).

I suppose addressing this use-case would require adding a (meta)data field for last-scanned time

What would this be used for?

Quoting myself: > though, I do still have a few ideas for improving hard link counting that may in turn make it harder to subtract file sizes, so let's await that... Implemented that new idea on the zig branch now, turns out that it instead *simplifies* extracting directory sizes, so that shouldn't be a big challenge anymore. The other points remain, though. As for refreshing an imported dir, that's fairly easy, but something that should absolutely be opt-in through a special cli flag because it's very prone to errors (refreshing on the wrong system, when the fs isn't mounted, or refreshing a year old export, etc). > I suppose addressing this use-case would require adding a (meta)data field for last-scanned time What would this be used for?