homepage

Underlying zlib can determine when it has hit the end of a compressed
stream without reading past the end. Python zlib implementation requires
that one read past the end before it signals the end by putting data in
Decompress.unused_data. This complicates interfacing with mixed
compressed/uncompressed streams.

Here is a patch which adds a member called is_finished to decompression
objects that allows client code to know when it has reached the end of
the compressed stream.

Thanks for the patch!
Can you provide tests too?

What kind of tests did you have in mind?
Unit tests in python, or something else?

Yes, I think that the right place where to add the tests is
Lib/test/test_zlib.py 

Attaching unit test diff
Output of "diff -u test_zlib.py~ test_zlib.py"

Some comments about the patch:
- In zlibmodule.c, the is_finished member should be an int, and converted 
to a PyObject only when requested.
- The test should check that is_finished is False one byte before the 
compressed part, and becomes True when the decompressor reads the last 
compressed byte. I don't think that dco.flush() is necessary for the 
test.
- Also, the last check could be more precise: assertEquals(y1 + y2, 
HAMLET_SCENE) and assertEquals(dco.unused_data, HAMLET_SCENE)

zlibmodule.c.diff Implements all the suggested features, but I'm not
exactly sure whether it handles reference counts properly.

Diff to tests
Implements all suggested changes save one:
I wasn't sure how to test that is_finished is clear one byte before the
end of the compressed section. Instead, I test that it is clear before
I call the compression routine.

Figured out how to test is_finished attribute of the zlib module properly.

Hm, I tried a modified version of your first test, and I found another 
problem with the current zlib library;
starting with the input:
x = x1 + x2 + HAMLET_SCENE # both compressed and uncompressed data
The following scenario is OK:
dco.decompress(x) # returns HAMLET_SCENE
dco.unused_data # returns HAMLET_SCENE
But this one:
for c in x:
 dco.decompress(x) # will return HAMLET_SCENE, in several pieces
dco.unused_data # only one character, the last of (c in x)!
This is a bug IMO: unused_data should accumulate all the extra uncompressed 
data.

This bug (zlib not providing a way to detect end-of-stream) has already
been fixed - see issue 12646.
I've opened issue 16350 for the unused_data problem.