homepage

PyPy had a longstanding issue :
ZipFile.extractall is very slow compared to CPython 2.6
https://bitbucket.org/pypy/pypy/issues/905/zipfileextractall-is-very-slow-compared-to
which has been fixed in the PyPy code base. The changes were entirely in zipfile.py (see the attached patch for PyPy)
The patch fixed a significant performance bottleneck in PyPy.

Could you provide a benchmarkin script that demonstrates a performance bottleneck?
The patch is against 2.7. Please provide a patch against the master branch.

Please assign this bug to me. I will submit a patch

serhiy sure I will attach proof of the performance bottle-neck on 2.7 and 3.7 before I submit a patch. Please assign this bug to me.

I assigned this issue to me as the maintainer of the zipfile module. This means that I undertake to examine the results of benchmarking, make the review of the proposed patch, and merge it if it improves the performance and doesn't have negative side effects. This shouldn't stop you from writing your patch.
Actually there was a reason why the code is written in that way. But maybe that reason no longer actual or there are stronger arguments for other ways. It is hard to say until I reproduce benchmarking results.

Serhiy yes what you said makes sense. Thanks for clarifying. Updates (benchmarking results) shortly...stay tuned.

Serhiy, I am curious why did you have chosen to compute the CRC32 table everytime? It is standard (the generator polynomial does not change) and always will output the same values. And it is also less computational intensive to loading a precomputed array vs calculating it each time a zip archive is extracted.

I'm not the initial author of zipfile. That initial code for decrypting was added in issue698833.
I like your idea about replacing the generating code with the precomputed table. Please open a separate issue for this.

Upon comparing the PyPy changes from attached diff to the latest cpython3 github, I don't find a need for improvement. Looks like cpython3 zipfile.py has the same changes and the read() method in class 
ZipExtFile(io.BufferedIOBase) is vastly improved compared to what pypy had. In fact it has been completely re-written.
The CPython 2.7 version of this bug was promptly closed as a duplicate:
http://bugs.python.org/issue30467
Perhaps the changes from CPython3 are not being backported to CPython2 ?
This bug can be closed because CPython3 has no issues.

The reason of using bytes concatenating rather than accumulating in the list, is that in most cases one of arguments is an empty bytes object (appending to the empty buffer or uncompressing a file with large compression block), and this case is optimized in CPython. In mos cases there is at most one nontrivial bytes concatenation per read operation, and using b''.join() is slower in that case.