homepage

I have a valid zlib compressed string, attached here as 'mat.bin' (1.7M), that cause and error on zlib.decompress decompression:
>>> import zlib
>>> data = open('mat.bin', 'rb').read()
>>> out = zlib.decompress(data)
Traceback (most recent call last):
 File "<ipython console>", line 1, in <module>
error: Error -5 while decompressing data
I know these data are valid, because I get the string I was expecting with:
>>> dc_obj = zlib.decompressobj()
>>> out = dc_obj.decompress(data)
As expected, there is no remaining data after this read:
>>> assert dc_obj.flush() == ''
>>> 
I believe that the behavior of zlib.decompress(data) and zlib.decompressobj().decompress(data) should be equivalent, and that the error for zlib.decompress(data) is therefore the symptom of a bug.

After a bit of debugging, it seems your data is not actually a complete zlib stream (*). What did you generate it with?
(*) in technical terms, the zlib never returns Z_STREAM_END when decompressing your data. The decompressobj ignores it, but the top-level decompress() function considers it an error.

Hi,
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> After a bit of debugging, it seems your data is not actually a complete zlib stream (*). What did you generate it with?
>
> (*) in technical terms, the zlib never returns Z_STREAM_END when decompressing your data. The decompressobj ignores it, but the top-level decompress() function considers it an error.
Thanks for the debugging. The stream comes from within a matlab 'mat'
file. I maintain the scipy matlab file readers; the variables within
these files are zlib compressed streams.
 Is there (should there be) a safe and maintained way to allow me to
read a stream that does not return Z_STREAM_END?

> Thanks for the debugging. The stream comes from within a matlab 'mat'
> file. I maintain the scipy matlab file readers; the variables within
> these files are zlib compressed streams.
So this would be a Matlab issue, right?
> Is there (should there be) a safe and maintained way to allow me to
> read a stream that does not return Z_STREAM_END?
Decompressor objects allow you to do that, but I cannot tell you how
"maintained" it is. If it has to be maintained, we could add an unit
test for it so that regressions get detected. It would be nice if you
could provide a very short zlib stream reproducing the issue.

I also think we should improve the zlib module's error messages. I've added a patch in issue8681 for that. With that patch, the message you'd've encountered would have been "Error -5 while decompressing data: incomplete or truncated stream", which is quite more informative.

>> Thanks for the debugging. The stream comes from within a matlab 'mat'
>> file. I maintain the scipy matlab file readers; the variables within
>> these files are zlib compressed streams.
>
> So this would be a Matlab issue, right?
Yes, except scipy and numpy aim in part to be an open-source
replacement for matlab, so we very much want to be able to read their
files.
>> Is there (should there be) a safe and maintained way to allow me to
>> read a stream that does not return Z_STREAM_END?
>
> Decompressor objects allow you to do that, but I cannot tell you how
> "maintained" it is. If it has to be maintained, we could add an unit
> test for it so that regressions get detected. It would be nice if you
> could provide a very short zlib stream reproducing the issue
This is the only .mat file stream I have yet come across that causes
the error. It is possible to knock a portion off the end of a valid
stream to reproduce the problem?

Ok, it turned out to be quite easy indeed. Here is a patch adding a test.

patch looks good.

The patch was committed in r81094 (2.7), r81095 (2.6), r81096 (3.2) and r81097 (3.1). Thank you!