Add "pack"/"unpack" #21

looking at this again i'm thinking maybe we want to move this to its own top-level command, sciop-cli pack

for consistency, --path ?

wording is a little unclear, maybe like "float from 0-1, skip directories that have more than this proportion of their files already compressed as archives (list of extensions detected)"

i think we probably want a --dry-run option here that just prints the directories it would compress with the given settings?

reading the impl, this should be easy to do with the Compressor.select_targets method :) nicely done

lets open as a secondary issue

we can just use pydantic models for this :)

i think the return type is _hashlib.HASH

let's use const.DEFAULT_TORRENT_CREATOR for this

base_dir: Path

mode: typing.Literal["heuristic", "leaf", "depth"] = "heuristic"

i'll show ya how pydantic models work for this :)

let's make the manifest a model too!

only hashing .txt files?

love a good progress bar

add an option for removing the compressed files?

opening as secondary issue

make level a param and then put in manifest?

opening as secondary issue

pbar on this would probably be good

use const.EXCLUDE_FILES to exclude system files

why not raise? seems like we'd want to do that.

for consistency, let's make this a classmethod, Compressor.from_args

make this a model :)

and this!

the ordering of this is a little odd to me - I would expect it to work like

• determine which directories to compress
• stats/hashes are computed for the files that are to be compressed
• it attempts to compress them, erroring if it fails for some reason and cleaning up temporary files
• it optionally removes the compressed files

so that the directories that are being removed are tied to the act of compression, eliminating the chance for e.g. the packmap to be out of date and making resume state unambiguous in the case of errors. would probably simplify the method here too, because we check if there is an archive present, but we don't know if it's correct here

see types.Size - i think we should have one way of stringifying sizes :)

i think we'll want to parameterize this, add more files programmatically with different sizes and quantities so that we can test the correctness and behavior of the compressors. I think for the sake of testing that we can just have this function create a single directory with n files of average_size and std_size with the filesize taking some normal distribution centered at the average with standard deviation? we use gamma distributions for this in test_torrent to check auto piece size

great start here! yes! I think we'll want to add some unit tests too,

• test roundtrip of the directory compression/decomp functions
• positive and negative controls for the different compressors - whether they correctly identify a directory that shoudl be compressed and reject a directory that shouldn't.
• test manifest creation
• test rejection of system files

and one end-to-end test that is basically this except for calling it with the CLI params so we make sure we don't break the cli part while leaving the library part working

Working on the test updates, when you say "system files", do you just mean the EXCLUDE_FILES?

Ya ya, I don't have an exhaustive list there, but just the ones ive seen. We can expand that as we go. Just want to exclude common files that wind up in datasets as a byproduct of the OS

let's raise here - if we have run out of disk space, we shouldn't try to continue

actually looking at the docs, this has platform availability notes so can sometimes fail. so we should probably do this like

try:
 usage = shutil.disk_usage(folder.parent)
except Exception:
 pass
else:
 if usage.free < total_bytes:
 raise OSError(f"Out of disk space, directory is {humanize.naturalsize(total_bytes)}, {humanize.naturalsize(usage.free)} available.")

hopefully == right?

len(Path(path).parents)

>>> len(Path("sup/hey/hello.txt").parents)
3

yeah for this one similar feelings, don't mean to bog you down with too much testing for this but i think we do want to cover the over and under for each of the heuristic rules - like if we set a mean filesize, does it correctly select a directory that is beneath it and reject a directory that above it and etc. for each of the rules in the heuristic. you already have the basic setup in place here so just need to add cases for each of these. you may want to split this off to its own testing function that works similarly to this one so you don't need to mess too much with the sample directory used in the rest of the non-heuristic cases, which only care about depth for the most part.

asking for this mostly for regression protection - want to make sure we don't accidentally break this, so want to at least cover the expected basic functionality even if we're not covering all the possible edge cases

this is a little odd behavior to me, i would expect that the depth parameter would behave like a "max depth" parameter, where we would compress everything below a certain depth. it seems odd to me to have an exact depth here, like e.g. if you had

a/z.gif
a/b/y.gif
a/b/c/x.gif

it would be surprising to me if i set depth: 2 (meaning a/b) and that caused me to compress a/b/y.gif into a/b.zip but then also still have a/b/c/x.gif since that's also in a/b. am i reading this test case right?

In our current implementation, we select the exact level to compress, then compress every folder at that exact level. Therefore, everything below that level will be included in the archives created at that level. At least, this is my intention. Isn't this basically max_depth? If not I'm happy to change it.

I'm adding new dummies to the fixture so I can expand the positive/negative cases anyway, but I think the selectors currently work with that in mind.

ok i see this now in the tests ya

another thing, i would usually think of "leaf" and "depth" as being the same thing, and one would think of "compress the leaf directories" as being depth: -1. not asking you to rewrite because i can see why they would be separated here, but we might want to add a specific note in the cli docs like "for negative depth (e.g. compress the bottommost directories that contain only files) see --leaf"

.... oh my god. Duh. Not sure how that didn't click for me. In that case we could just allow for a -1, and then throw in the skip_threshold check for the depth mode, to consolidate. Leaf mode is kinda my go-to for pack so it seemed natural to spin it off as it's own thing either way, and we could keep it around if we wanted but maybe also enable -1 depth, to be more intuitive?

i'm good with whatever as long as there is only one actual implementation :)

not really testing the behavior of interest here - we want to test at least four things:

• only files in leaf directories are compressed, even if they are below the threshold
• files in leaf directories are compressed if they are below the threshold
• files in leaf directories are not compressed if they are above the threshold
• zipped files are not counted against the threshold (i think? or however zip files are handled)

(or whichever direction the threshold goes).

so, here with the high skip threshold, we should test that we do choose to zip any directories that should be zipped, which loks like keep_me and parent/child and also tiny??, and then also add another file within parent that should not be zipped.

nice. splitting this out was the right call.