homepage

Yeah, I very much agree with what Nick says -- we really want a way to expose what the platform provides. It's less important (though still the ideal), to expose that in a platform-independent way. Today the only way to get access to opendir/readdir on Linux and FindFirst/Next on Windows is by using a bunch of messy (and slowish) ctypes code. And yes, os.walk() would be the main cross-platform abstraction built on top of this.
Charles gave this example of code that would fall over:
size = 0
for name, st in scandir(path):
 if stat.S_ISREG(st.st_mode):
 size += st.st_size
I don't see it, though. In this case you need both .st_mode and .st_size, so a caller would check that those are not None, like so:
size = 0
for name, st in scandir(path):
 if st.st_mode is None or st.st_size is None:
 st = os.stat(os.path.join(path, name))
 if stat.S_ISREG(st.st_mode):
 size += st.st_size
One line of extra code for the caller, but a big performance gain in most cases.
Stinner said, "But as usual, a benchmark on a real platform would be more convicing". Here's a start: https://github.com/benhoyt/betterwalk#benchmarks -- but it's not nearly as good as it gets yet, because those figures are still using the ctypes version. I've got a C version that's half-finished, and on Windows it makes os.walk() literally 10x the speed of the default version. Not sure about Linux/opendir/readdir yet, but I intend to do that too.