homepage

Patch in attachment uses platform specific zero-copy syscalls on Linux and Solaris (os.sendfile(2)), Windows (CopyFileW) and OSX (fcopyfile(2)) speeding up shutil.copyfile() and other functions using it (copy(), copy2(), copytree(), move()).
Average speedup for a 512MB file copy is +24% on Linux, +50% on OSX and +48% on Windows by copying file on the same partition (SSD disk was used).
Follows some benchmarks.
Setup
=====
Create 128K, 8M, 512M file:
 $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(128 * 1024))"
 $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(8 * 1024 * 1024))"
 $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(512 * 1024 * 1024))"
Benchmark:
 $ time ./python -m timeit -s 'import shutil; p1 = "f1"; p2 = "f2"' 'shutil.copyfile(p1, p2)'
Linux
=====
128K copy (+13%):
 without patch:
 1000 loops, best of 5: 228 usec per loop
 real 0m1.756s
 user 0m0.386s
 sys 0m1.116s
 with patch:
 1000 loops, best of 5: 198 usec per loop
 real 0m1.464s
 user 0m0.281s
 sys 0m0.958s
8MB copy (+24%):
 without patch:
 50 loops, best of 5: 10.1 msec per loop
 real 0m2.703s
 user 0m0.316s
 sys 0m1.847s
 with patch:
 50 loops, best of 5: 7.78 msec per loop
 real 0m2.447s
 user 0m0.086s
 sys 0m1.682s
512MB copy (+26%):
 without patch:
 1 loop, best of 5: 872 msec per loop
 real 0m5.574s
 user 0m0.402s
 sys 0m3.115s
 with patch:
 1 loop, best of 5: 646 msec per loop
 real 0m5.475s
 user 0m0.037s
 sys 0m2.959s
OSX
===
128K copy (+8.5%):
 without patch:
 500 loops, best of 5: 508 usec per loop
 real 0m2.971s
 user 0m0.442s
 sys 0m2.168s
 with patch:
 500 loops, best of 5: 464 usec per loop
 real 0m2.798s
 user 0m0.379s
 sys 0m2.031s
8MB copy (+67%):
 without patch:
 20 loops, best of 5: 32.8 msec per loop
 real 0m3.672s
 user 0m0.357s
 sys 0m1.434s
 with patch:
 20 loops, best of 5: 10.8 msec per loop
 real 0m1.860s
 user 0m0.079s
 sys 0m0.719s
512MB copy (+50%):
 without patch:
 1 loop, best of 5: 953 msec per loop
 real 0m5.930s
 user 0m1.021s
 sys 0m4.835s
 
 with patch:
 1 loop, best of 5: 480 msec per loop
 real 0m3.150s
 user 0m0.067s
 sys 0m2.740s
Windows
=======
128K copy (+69%):
 without patch:
 50 loops, best of 5: 6.45 msec per loop
 with patch:
 50 loops, best of 5: 1.99 msec per loop
8M copy (+64%):
 without patch:
 10 loops, best of 5: 22.6 msec per loop
 with patch:
 50 loops, best of 5: 7.95 msec per loop
512M copy (+48%):
 without patch:
 1 loop, best of 5: 1.21 sec per loop
 with patch:
 1 loop, best of 5: 629 msec per loop

PR: https://github.com/python/cpython/pull/7160 

Nice, I really like this.
Apart from the usual bit of minor style issues, I couldn't see anything inherently wrong with the PR, but I'll leave the detailed reviews to those who'd have to maintain the code in the future. :)

Regarding the benchmarks, just to be sure, did you try reversing the run order to make sure you don't get unfair caching effects for the later runs?

http://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html
That possibly should be used under Linux in order to really acheive zero-copying. Just like modern cp command.

Yes, I tried changing benchmarks order and zero-copy variants are always faster. As for instantaneous CoW copy, it is debatable. E.g. "cp" command does not do it by default: 
https://unix.stackexchange.com/questions/80351/why-is-cp-reflink-auto-not-the-default-behaviour
I think shutil should follow the same lead, and perhaps provide a cow=bool argument in the future.

New changeset 4a172ccc739065bb658c75e8929774a8e94af9e9 by Giampaolo Rodola in branch 'master':
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160)
https://github.com/python/cpython/commit/4a172ccc739065bb658c75e8929774a8e94af9e9

For future reference, as per https://github.com/python/cpython/pull/7160 discussion, we decided not to use CopyFileEx on Windows and instead increase read() buffer size from 16KB to 1MB (Windows only) resulting in a 40.8% speedup (instead of 48%). Also copyfileobj() has been optimized on all platforms by using readinto()/memoryview()/bytearray().
Updated benchmarks on Windows:
128KB copy (+27%)
 without patch:
 50 loops, best of 5: 7.69 sec per loop
 with patch:
 50 loops, best of 5: 5.61 sec per loop
8MB copy (+45.6%)
 without patch:
 10 loops, best of 5: 20.8 sec per loop
 with patch:
 20 loops, best of 5: 11.3 sec per loop
512MB copy (+40.8%)
 without patch:
 1 loop, best of 5: 1.26 sec per loop
 with patch:
 1 loop, best of 5: 646 msec per loop

Thanks Gianpaolo for pushing for this. Great job.

> Thanks Gianpaolo for pushing for this. Great job.
I concur: great job! Cool optimization.

New changeset c7f02a965936f197354d7f4e6360f4cfc86817ed by Giampaolo Rodola in branch 'master':
bpo-33671 / shutil.copyfile: use memoryview() with dynamic size on Windows (#7681)
https://github.com/python/cpython/commit/c7f02a965936f197354d7f4e6360f4cfc86817ed

New changeset 8fbbdf0c3107c3052659e166f73990b466eacbb0 by Victor Stinner in branch 'master':
bpo-33671: Add support.MS_WINDOWS and support.MACOS (GH-7800)
https://github.com/python/cpython/commit/8fbbdf0c3107c3052659e166f73990b466eacbb0

New changeset 937ee9e745d7ff3c2010b927903c0e2a83623324 by Victor Stinner in branch 'master':
Revert "bpo-33671: Add support.MS_WINDOWS and support.MACOS (GH-7800)" (GH-7919)
https://github.com/python/cpython/commit/937ee9e745d7ff3c2010b927903c0e2a83623324

New changeset 3b0abb019662e42070f1d6f7e74440afb1808f03 by Inada Naoki (Giampaolo Rodola) in branch 'master':
bpo-33671: allow setting shutil.copyfile() bufsize globally (GH-12016)
https://github.com/python/cpython/commit/3b0abb019662e42070f1d6f7e74440afb1808f03

shutil.COPY_BUFSIZE isn't documented. Is it a deliberate choice?
https://docs.python.org/dev/library/shutil.html 

Yes, it's deliberate, see PR-12016.

> Yes, it's deliberate, see PR-12016.
"I decided not to document `shutil.COPY_BUFSIZE` because I consider it a corner case."
Ok. I have no opinion on that, I just wanted to ask the question :-)

New changeset b08c48e61745666df9aeee97d8bddbf1e5550627 by Anthony Sottile in branch 'main':
bpo-33671 fix orphaned comment in shutil.copyfileobj (GH-27516)
https://github.com/python/cpython/commit/b08c48e61745666df9aeee97d8bddbf1e5550627