Remove directory tree without following symlinks

Sun Apr 24 15:42:29 EDT 2016

On Sun, Apr 24, 2016 at 5:42 AM, Albert-Jan Roskam
<sjeik_appie at hotmail.com> wrote:
> Aww, I kinda forgot about that already, but I came across this last
> year [1]. Apparently, shutil.rmtree(very_long_path) failed under Win 7,
> even with the "silly prefix". I believe very_long_path was a
> Python2-str.
> [1]
> https://mail.python.org/pipermail/python-list/2015-June/693156.html

Python 2's str branch of the os functions gets implemented on Windows
using the [A]NSI API, such as FindFirstFileA and FindNextFileA to
implement listdir(). Generally the ANSI API is a light wrapper around
the [W]ide-character API. It simply decodes byte strings to UTF-16 and
calls the wide-character function (or a common internal function).
IIRC, in Windows 7, byte strings are decoded using a per-thread buffer
with size MAX_PATH (260), so prefixing the path with "\\?\" won't
help. You have to use the wide-character API. Windows 10, on the other
hand, decodes using a dynamically allocated buffer, so you can usually
get away with using a long byte string. But not with Python 2
os.listdir(), which uses a stack-allocated MAX_PATH+5 buffer in the
str branch. For example:
Python 2 os.mkdir works:
 >>> path = os.path.normpath('//?/C:/Temp/long/' + 'a' * 255)
 >>> os.makedirs(path)
but os.listdir requires unicode:
 >>> os.listdir(path)
 Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 TypeError: must be (buffer overflow), not str
 >>> os.listdir(path.decode('mbcs'))
 []
Also, the str branch of listdir appends "/*.*", with a forward slash,
so it's incompatible with the "\\?\" prefix, even for short paths:
 >>> os.listdir(r'\\?\C:\Temp')
 Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 WindowsError: [Error 123] The filename, directory name, or volume
 label syntax is incorrect: '\\\\?\\C:\\Temp/*.*'
> It seems useful if shutil or os.path would automatically prefix paths
> with "\\?\". It is rarely really needed, though. (in my case it was
> needed to copy a bunch of MS Outlook .msg files, which automatically
> get the subject line as the filename, and perhaps the first sentence
> of the mail of the mail has no subject).

I doubt a change like that would get backported to 2.7. Recently there
was a lengthy discussion about adding an __fspath__ protocol to Python
3. Possibly this can be automatically handled in the __fspath__
implementation of pathlib.WindowsPath and the DirEntry type returned
by os.scandir.