homepage

> STINNER Victor added the comment:
>
>> scandir is slower on my machine:
>
> Please share more information about your config: OS, disk type (hard
> drive, SSD, something else), filesystem, etc.
>
Ubuntu 14.04, SSD, ext4 filesystem. Results for different python
versions are the same (scandir on pypy is even slower due to ctype
usage).
What other information could be useful?
> Ben Hoyt added the comment:
>
> Akira, note the "Using slower ctypes version of scandir" -- this is
> the older, ctypes implementation of scandir. On Linux, depending on
> file system etc, that can often be slower. You need to at least be
> using the "fast C version" in _scandir.c, which is then half C, half
> Python. But ideally you'd use the all-C version that I've provided as
> a CPython 3.5 patch.
Yes. I've noticed it, that is why I asked :) *"What commands should I
run to benchmark the attached patch (scandir-2.patch)?"*
I've read benchmark.py's source; It happens that it already supports
benchmarking of os.scandir. In my python checkout:
 cpython$ hg import --no-commit https://bugs.python.org/file36963/scandir-2.patch
 cpython$ make && cd ../scandir/
 scandir$ ../cpython/python benchmark.py -s /usr/
 Using Python 3.5's builtin os.scandir()
 Comparing against builtin version of os.walk()
 Priming the system's cache...
 Benchmarking walks on /usr/, repeat 1/3...
 Benchmarking walks on /usr/, repeat 2/3...
 Benchmarking walks on /usr/, repeat 3/3...
 os.walk size 7925376343, scandir.walk size 5534939617 -- NOT EQUAL!
 os.walk took 13.552s, scandir.walk took 6.032s -- 2.2x as fast
It seems os.walk and scandir.walk do a different work here.
I've written get_tree_size_listdir_fd() that mimics get_tree_size() 
(see get_tree_size_listdir.diff patch) to get the same size:
 scandir$ ../cpython/python benchmark.py -s /usr
 Using Python 3.5's builtin os.scandir()
 Comparing against builtin version of os.walk()
 Comparing against get_tree_size_listdir_fd
 Priming the system's cache...
 Benchmarking walks on /usr, repeat 1/3...
 Benchmarking walks on /usr, repeat 2/3...
 Benchmarking walks on /usr, repeat 3/3...
 os.walk size 5534939617, scandir.walk size 5534939617 -- equal
 os.walk took 5.697s, scandir.walk took 5.621s -- 1.0x as fast
scandir is not noticeably faster but scandir-based code is much nicer here.