homepage

I want shutil.make_archive() to produce deterministic output when given identical data as inputs.
Right now there are two holes in this. One is that mtimes might not match. This can be fixed by the caller. The second is that the order that files in a subdirectory get added to the tarfile is not deterministic. This can't be fixed by the caller.
Attached is a trivial patch to sort the results of os.listdir() to ensure the output tarfile is stable.
This only applies to the 'tar' format.
I've attached my testcase for this, which creates 3 tarfiles in /tmp. When this patch is applied, the 3 tarfiles it creates are identical according to `sha1sum`. Without this patch, they are all different.

This would go beyond what the tar command itself does. I'm not sure we want to do that, as we are pretty much modeling our behavior on tar. However, that doesn't automatically mean we can't do it. We'll see what other people think. Personally I'm -0.
I've changed the issue title since your proposed patch is to tarfile, not shutil.

You don't need to patch the tarfile module. You could use os.walk() in shutil._make_tarball() and add each file with TarFile.add(recursive=False).

Thanks for the comments! Would you be happy for the patch to be merged if it was implemented by modifying shutil.make_archive() instead? I will rework it if so.

I don't see any downside for this simple patch and think there is some merit for wanting a reproducible archive.

The patch would change behaviour for all tarfile users by the back door, that's why I am a little reluctant. And if the same can be achieved by a reasonably simple change to shutil I think it's just as well.

I've discovered that this patch introduces a nasty failure case! If you have a relative symlink pointing to a directory that's alphabetically sorted after the symlink, and files inside the symlink, 'tar -x' won't be able to create those files because the symlink target won't exist yet.
I'll rework this to only affect shutil.make_archive(), and to avoid hitting this bug.

Having tested, the problem I described above doesn't happen with this patch. It's a mistake in some other code I wrote which is following symlinks when it should not do.

Here's a patch which does the same thing but only for shutil.make_archive().
Note that the final output will still be non-deterministic if you use format=gztar because time.time() and the base_name argument get added to the gzip header. Might be nice to add an option to make that deterministic too, as a separate thing. This patch is useful to me as-is though.