homepage

Python currently supports zlib, gzip and bzip2 compressors. What is missing is support 
for xz (http://tukaani.org/xz/). It comes with a C library.

Is zc really a C library? I could find a standalone program, but no
shared object.
Actually, it seems that zc is a file format based on the lzma algorithm.
The plan could be to first implement the lzma module (issue5689), then a
xzfile module in pure python.

What is xz compression and why is it important?
Skip

Yes, xz-utils contains a C library, though it still caries the name
"liblzma.so", probably for historic reasons.
You are right that xz is a file format based around the lzma algorithm.
It just uses a more advanced container format. (lzma-utils had no
container at all.)
xz is the successor of lzma, which provides a better compression than
bzip2, while decompression speed is comparable with gzip. It is used by
the GNU project for source tarball compression (replacing bzip2) and
supported by GNU tar. See http://en.wikipedia.org/wiki/Xz,
http://tukaani.org/xz/ and http://tukaani.org/lzma/ for reference.

Are xz and lzma formats compatible with each other? If not, which one is
the most popular?

As I understand it from the http://tukaani.org/xz/ page, .lzma and .xz
are two file formats bases on the lzma compression method.
- .lzma simply stores the compressed stream.
- .xz is more complex

.lzma is actually not a format. It is just the raw output of the LZMA1
coder. XZ instead is a container format for the LZMA2 coder, which
probably means LZMA+some metadata.
XZ is the official successor to .lzma, and GNU is using it already
(look at coreutils), and GNU tar officially supports it since 1.22.

Once Python gets native support for lzma/xz like it does for zlib, bzip2 it could switch to using it for bundles and remote transfers. See:
http://mercurial.selenic.com/bts/issue1463
With lzma/xz being able to compress so much better, it'd be really appreciated by users on especially slow links(!!).

Ugh, can't edit previous message. Meant to say:
"Once Python gets native support for lzma/xz like it does for zlib and bzip2, Mercurial could switch to using it for bundles and remote transfers."
For platforms with native support in-kernel (e.g. Linux) that could be used instead of the bundled version.
(Since Python is officially switching to Mercurial, arguably this issue even more important.)

About why xz is important. 
gnu.org, tug.org started publishing sources in xz format, quick grep:
autoconf/autoconf.spec:Source0: http://ftp.gnu.org/gnu/autoconf/%{name}-%{version}.tar.xz
coreutils/coreutils.spec:Source0: http://ftp.gnu.org/gnu/coreutils/%{name}-%{version}.tar.xz
libpng12/libpng12.spec:Source0: http://downloads.sourceforge.net/libpng/libpng-%{version}.tar.xz
libpng/libpng.spec:Source0: http://downloads.sourceforge.net/libpng/%{name}-%{version}.tar.xz
parted/parted.spec:Source0: http://ftp.gnu.org/gnu/parted/%{name}-%{version}.tar.xz
texlive-texmf/texlive-texmf.spec:Source0: ftp://tug.org/texlive/historic/%{year}/texlive-%{version}-texmf.tar.xz
xz is also supported by automake as dist target.

We all agree that lzma/xz is important, what is needed is a patch.

There's a Python binding for some lzma lib here: https://launchpad.net/pyliblzma 

Once xz is implemented, xz compatibility should be added to the tarfile library.

Ooops, I kinda should've commented on this issue here in stead, rather than in issue5689, so I'll just copy-paste it here as well:
I'm the author of the pyliblzma module, and if desired, I'd be happy to help out adapting pyliblzma for inclusion with python.
Most of it's code is based on bz2module.c, so it shouldn't be very far away from being good 'nuff.
What I see as required is:
* clean out use of C99 types etc.
* clean up the LZMAOptions class (this is the biggest difference from the bz2 module, as the filter supports a wide range of various options, everything related such as parsing, api documentation etc. was placed in it's own class, I've yet to receive any feedback on this decission or find any remote equivalents out there to draw inspiration from;)
* While most of the liblzma API has been implemented, support for multiple/alternate filters still remains to be implemented. When done it will also cause some breakage with the current pyliblzma API.
I plan on doing these things sooner or later anyways, it's pretty much just a matter of motivation and priorities standing in the way, actual interest from others would certainly have a positive effect on this. ;)
For other alternatives to the LGPL liblzma, you really don't have any, keep in mind that LZMA is "merely" the algorithm, while xz (and LZMA_alone, used for '.lzma', now obsolete, but still supported) are the actual format you want support for. The LZMA SDK does not provide any compatibility for this.

I will happily review any implementation, and I can help with inclusion into python trunk.
> ...the LGPL liblzma...
Can you check which licences cover the different parts of the module? I think that you will have to contribute your code under the Python Contributor Agreement; and I just grabbed some copy of the "xz-utils" source package, and it states that "liblzma is in the public domain".

ah, you're right, I forgot that the license for the library had changed as well (motivated by attempt of pleasing BSD people IIRC;), in the past the library was LGPL while only the 'xz' util was public domain..
For my code, feel free to use your own/any other license you'd like or even public domain (if the license of bz2module.c that much of it's derived from permits of course)!
I guess everyone should be happy now then. :)
Btw. for review, I think the code already available should be pretty much good 'nuff for an initial review. Some feedback on things not derived from bz2module.c would be nice, especially the LZMAOptions class would be nice as it's where most of the remaining work required for adding additional filters support. Would kinda blow if I did the work using an approach that would be dismissed as utterly rubbish. ;)
Oh well, it's out there available for anyone already, I probably won't(/shouldn't;) have time for it in a month at least, do as you please meanwhile. :)

Hello,
> I'm the author of the pyliblzma module, and if desired, I'd be happy
> to help out adapting pyliblzma for inclusion with python.
> Most of it's code is based on bz2module.c, so it shouldn't be very far
> away from being good 'nuff.
Well, I wouldn't say bz2module is the best module out there, but as you
say it's probably good enough :) And we can help you fix things if
needed.
Is pyliblzma compatible with Python 3.x? It's too late to incorporate
any new feature in Python 2.x now.
> * While most of the liblzma API has been implemented, support for
> multiple/alternate filters still remains to be implemented. When done
> it will also cause some breakage with the current pyliblzma API.
Hmm, then perhaps you should first fix the current API so that adding
new features doesn't force you to break the API again. There are strict
rules for API breakage in the standard library.
By the way, adding a new module to the stdlib probably requires writing
a PEP (Python Enhancement Proposal). I wouldn't expect this very
proposal to be controversial, but someone has to do it.
Finally, when a module is in the stdlib, it is expected that maintenance
primarily happens in the Python SVN (or Mercurial) tree. We have a
couple of externally-maintained modules, but they're a source of
problems for us.

Yeah, I guess I anyways can just break the current API right away to make it compatible with future changes, I've already figured since long ago how it should look like. It's not like I have to implement the actual functionality to ensure compatibility, no-op works like charm. ;)

[Replying to msg106566]
> if you're already looking at issue6715, then I don't get why you're
> asking.. ;)
Can you please submit a contributor form?
> Martin: For LGPL (or even GPL for that matter, disregarding linking
> restrictions) libraries you don't have to distribute the sources of
> those libraries at all (they're already made available by others, so
> that would be quite overly redundant, uh?;). LGPL actually doesn't
> even care at all about the license of your software as long as you
> only dynamically link against it.
Of course you do. Quoting from the LGPL
"You may convey a Combined Work ... if you also do each of the following:
 ...
 d) Do one of the following:
 0) Convey the Minimal Corresponding Source under the terms of this 
 License, and the Corresponding Application Code in a form 
 suitable for, and under terms that permit, the user to recombine 
 or relink the Application with a modified version of the Linked 
 Version to produce a modified Combined Work, in the manner 
 specified by section 6 of the GNU GPL for conveying 
 Corresponding Source.
 1) [not applicable to Windows]
"
> I don't really get what the issue would be even if liblzma were still
> LGPL, it doesn't prohibit you from distributing a dynamically linked
> library along with python either if necessary (which of course would
> be of convenience on win32..)..
Of course I can distribute a copy of an lzma DLL. However, I would have to provide ("convey") a copy of the source code of that DLL as well.

> Of course I can distribute a copy of an lzma DLL. However, I would
> have to provide ("convey") a copy of the source code of that DLL as
> well.
Can you tell me where you are currently providing the source code for
the readline library, or the gdbm library?
Oh, and by the way, you should probably shut down PyPI, since there are
certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed
one), and you aren't offering a link to download those libraries' source
code either.
You seem to have no problem "conveying" copies for the source code of
non-LGPL libraries such as OpenSSL. Why is that?

> Can you tell me where you are currently providing the source code for
> the readline library, or the gdbm library?
We don't, as they aren't included in the Windows distribution. The 
readline library doesn't work on Windows, anyway, and instead of gdbm, 
we had traditionally been distributing bsddb instead on Windows.
> Oh, and by the way, you should probably shut down PyPI, since there are
> certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed
> one), and you aren't offering a link to download those libraries' source
> code either.
This is off-topic for this bug tracker. Please remain objective and 
professional if you can manage to.
> You seem to have no problem "conveying" copies for the source code of
> non-LGPL libraries such as OpenSSL. Why is that?
Not sure what you are referring to. We don't provide the sources for the 
OpenSSL libraries along with the Windows installer, because the license 
of OpenSSL doesn't require us to. This is very convenient for our users.

> > Oh, and by the way, you should probably shut down PyPI, since there are
> > certainly Python wrappers for LGPL'ed libraries there (or even GPL'ed
> > one), and you aren't offering a link to download those libraries' source
> > code either.
> 
> This is off-topic for this bug tracker. Please remain objective and 
> professional if you can manage to.
This whole subthread was already off-topic (since it was pointed out,
before your previous message, that the underlying lib is in the public
domain).
Actually, I would argue that the whole idea of promoting a rigorous
interpretation of a license has no place on the bug tracker. It makes no
sense to do this on an ad hoc fashion, especially if you want lawyers to
be involved (they are certainly not reading this).
(of course, you will also have understood that I disagree with such a
rigorous interpretation)
> Not sure what you are referring to. We don't provide the sources for the 
> OpenSSL libraries along with the Windows installer, because the license 
> of OpenSSL doesn't require us to. This is very convenient for our users.
This was not about providing the sources together with the installer
(which even the GPL or the LGPL don't require to do), but providing them
as a separate bundle on the download site.
We do have a copy of the OpenSSL source tree somewhere, it is used by
the Windows build process.

I've ported pyliblzma to py3k now and also implemented the missing functionality I mentioned earlier, for anyone interested in my progress the branch is found at:
https://code.launchpad.net/~proyvind/pyliblzma/py3k
I need to fix some memory leakages (side effect of the new PyUnicode/Pybytes change I'm not 100% with yet;) and some various memory errors reported by valgrind etc. though, but things are starting to look quite nice already. :)

I've (finally) finalized the api and prepared pyliblzma to be ready for inclusion now.
The code can be found in the 'py3k' branch referred to earlier.
Someone else (don't remember who:p) volunteered for writing the PEP earlier, so I leave it up to that person to write the PEP, I won't be able to get around to do so myself in the near future..

As promised, here is a quick review of the module.
https://code.launchpad.net/~proyvind/pyliblzma/py3k looks ready for a new entry in the PyPI, but for inclusion in core python it needs some cleanup:
- I suppose that src/pyliblzma.c is the only interesting file. In CPython it will be named Modules/lzmamodule.c
- There are some calls to malloc(), calloc() and free() that are incorrect (there is even a call to free() on a PyObject*!); in any case, functions like PyMem_Malloc() are preferred.
- please don't use alloca (and it has another name on Windows)
- snprintf is not available everywhere; use PyOS_snprintf instead.
- The module does not compile on Windows: types like uint8_t don't exist, __attribute__((unused)) is GCC-specific.
- Don't use PyBytesObject*; PyObject* is enough and avoids many casts.
- PyLong_FromLong(LZMA_FILTER_LZMA1) truncates the value on 32bit platforms
- You could use LZMA_VERSION_STRING to fill the version attribute.
Also, I did not find any documentation.
Otherwise the code follows CPython conventions and is easy to read. Good job!
3.2beta1 is scheduled for November 13, 2010, and no new feature will be accepted after. Do you think you can update it before the limit? otherwise the package could live in PyPI before it is shipped with Python.

All fixed now. :)

> All fixed now. :)
Does that refer to msg119743? Please provide this module
as a patch for branches/py3k, otherwise it's difficult to see
what exactly you are contributing.
Also, please provide Doc/library/lzma.rst.

Here's a patch with the latest code generated against py3k branch, it comes with Doc/library/lzma.rst as well now.

here's Lib/test/teststring.lzma, required by the test suite.

here's Lib/test/teststring.xz, required by the test suite.

Could you upload the patch to http://codereview.appspot.com/? 

After applying the patch, it builds fine here and the test suite passes. However, it seems to leak quite a bit -- if I run regrtest with -R::, my system starts swapping heavily after the second run.
In lzmamodule, there are lots of API calls that aren't error-checked, e.g.
 PyDict_SetItemString(filter, "dict_size", PyLong_FromLong((long)lzma_options.dict_size));
General comments: please don't introduce tabs and trailing whitespace in C and Python files, and wrap your lines at 79 characters whenever possible. reST function/class directives need a blank line after the signature. (But don't worry about the markup too much, I'll review the new file anyway after commit.)
@pitrou: Why is this marked "Python 3.3"? If the error handling in the C module is corrected, it's in a good enough shape to be committed before 3.2b1, and the remaining bugs ironed out until final.

> @pitrou: Why is this marked "Python 3.3"? If the error handling in
> the C module is corrected, it's in a good enough shape to be committed
> before 3.2b1, and the remaining bugs ironed out until final.
I think it needs a real review before going in. Now if that review can
get done and the issues fixed before the beta, it's ok. By a quick look
at the code it seemed to need quite a bit of polish before being
acceptable for commit, but I might be mistaken.
(we should also avoid the multiprocessing syndrome where we rushed some
alpha-quality code just before the feature deadline and then had to fix
many issues in urgency. I do believe xz/lzma support is important in the
long term, though)
Oh, by the way, Per should agree to do maintenance directly against the
CPython repository. Please, no more externally-maintained modules (you
know what I'm talking about).

Yes, definitely no externally maintained modules.

I've uploaded a new version of the patch to http://codereview.appspot.com/2724043/ now.
I'd be okay on doing maintenance directly against the CPython repository btw. :)

After starting to review the code, I'm becoming skeptical whether this is the right approach. This does way to much action in C, and thus becomes complicated but also limited.
An alternative approach would be to just expose lzma_code to Python, and then integrate that into the io architecture, i.e. as a subclass of RawIOBase. I.e. LZMAFile would go; if you want to compress to a file, use FileIO instead, and wrap that with a LZMAIO (say). LZMACompressor and Decompressor could stay if desired, although it seems more liblzma-like to have a single object for both compression and decompression.
In addition, I find the options object too complicated. It seems to serve documentation purposes only, so I wonder whether it could be reduced in code size.

LZMAFile, LZMACompressor & LZMADecompressor are all inspired by and written to be as similar to bz2's for easier use & maintenance. I must admit that I haven't really put much thought into alternate ways to implement them beyond monkey see, monkey do.. ;)
LZMAOptions is a bit awkwardly written, but it doesn't serve documentation purposes only, it also exposes these values for max, min etc. to python (ie. as used by it's regression tests) and are also used when processing various compression options passed.
IMO it does serve a useful purpose, but certainly wouldn't hurt from being rewritten in some better way...

I feel guilty of having said, some months ago, that "Well, I wouldn't say bz2module is the best module out there, but as you say it's probably good enough" (my words).
It's true that bz2module is not awful in terms of coding style or quality; the main issue with it is that uses outdated ways of doing I/O compared to what Python 3 recommends (that is, compose various building blocks together to allow for better re-use).
A point of reference is gzip, where the core compression routines (mostly wrappers) are in zlibmodule (in C), but the high-level file object is written in pure Python (and is composeable with other building blocks for fast buffered IO).

Hehe, don't feel guily on my part at least, I had already implemented it like this long before. :p
I guess I could rewrite it following these suggestions, but I probably won't be able to finish it in time for 3.2 beta.

> LZMAFile, LZMACompressor & LZMADecompressor are all inspired by and
> written to be as similar to bz2's for easier use & maintenance. I
> must admit that I haven't really put much thought into alternate ways
> to implement them beyond monkey see, monkey do.. ;)
My concern really is with LZMAFile only. It uses stdio, and it really
shouldn't - we are trying to drop stdio in Python, as it's causing too
many problems. LZMACompressor/Decompressor may be fine.
> LZMAOptions is a bit awkwardly written, but it doesn't serve
> documentation purposes only, it also exposes these values for max,
> min etc. to python (ie. as used by it's regression tests) and are
> also used when processing various compression options passed.
> 
> IMO it does serve a useful purpose, but certainly wouldn't hurt from
> being rewritten in some better way...
If you are willing to do that, here is what I propose: rename the module
to _lzma, drop options and lzmafile from _lzma. Then write lzma.py,
which re-introduces this Options class in pure Python, as well as
reintroduces LZMAFile, wrapping io.FileIO. The various constants that
go into options should be exposed from _lzma using
PyModule_AddIntConstant (except perhaps for the ones that you define
yourself, such as LZMA_BEST_SPEED)
That assumes, of course, that there is a need to provide backwards
compatibility. I would personally be fine with *just* having the LZMA
symbolic constants exposed, leaving it to the user to determine which
constant serves which purpose (e.g. guessing what LZMA_DICT_SIZE_MIN
and LZMA_DICT_SIZE_MAX do when there is a dict_size keyword parameter
isn't too hard). But then, having the various options explained
in a consistent manner may be useful enough to preserve the Options
class.

> I guess I could rewrite it following these suggestions, but I probably won't be able to 
> finish it in time for 3.2 beta.
The more I think about it, the more I feel like -1 about this code as
long as it uses stdio, and does buffering. We really don't need another
implementation of splitting a stream of bytes into lines.

While I realize Python extension modules are pretty much the norm in CPython, it'd be pretty cool if xz support could be written overtop of ctypes. ctypes seems to be emerging as the way of doing FFI across different python implementations.
I've been working on a backup engine in my spare time, and I'd very much like to use a Python xz module in it. However, the code I have so far runs about 4x faster on pypy than CPython, even if I turn on psyco in CPython.
IOW, I'd be pretty pleased to see an xz module that works equally well on CPython, pypy (seems to have pretty good ctypes), jython (has the beginnings of ctypes in 2.5.2rc*), iron python (supposedly included recently)...

I’m afraid ctypes is too insecure for stdlib code. Even using it in some tests had to be discussed and decided at EuroPython.
I’m told that PyPy has CPyExt now, so this may solved your compat concern :)

I agree that ctypes is a bit more brittle - both ctypes and c extension modules can yield segfaults, but at least the c extension module is likely to give an error or warning when you rebuild it.
However, I'm getting the impression that:
1) In pypy, the C extension module layer isn't a complete emulation
2) In pypy, the C extension module is a bit disfavored compared to ctypes
3) In pypy, there's a performance expense for CPyExt compared to ctypes
3) (less important) In pypy, the C extension module layer is still too young to do much with at this time
I've been wondering if maybe there should be a way to check ctypes use against .h's at build time. That way, the brittleness should be about the same.

Careful: the ctypes module is much slower with pypy than with cpython.
In any case, a C extension module is preferred because the (platform-dependent) details of the liblzma library are defined in a .h header file, not as a ctypes description. Future versions of the library are likely to be source-compatible, but may change some struct member for example; this would break a ctypes-based interface.

> While I realize Python extension modules are pretty much the norm in
> CPython, it'd be pretty cool if xz support could be written overtop
> of ctypes. ctypes seems to be emerging as the way of doing FFI
> across different python implementations.
There is a clear policy in CPython that ctypes must not be used
to implement library modules.

no programmer or anything. Just adding that the wikipedia link given is not right. The correct link is https://secure.wikimedia.org/wikipedia/en/wiki/XZ_Utils
Just my 2 paise.

Having "Christophe Simonis" in the nosy list is precluding NOSY changes. I guess the space is causing problems. Should we forbid spaces in usernames (maybe provided by OpenID, who knows)???
What do we need to progress in this issue?.
A rewrite to use Python 3 IO facilities?.

> Having "Christophe Simonis" in the nosy list is precluding NOSY
> changes. I guess the space is causing problems. Should we forbid
> spaces in usernames (maybe provided by OpenID, who knows)???
Please report this in the meta-tracker (link on the left).
> What do we need to progress in this issue?.
> 
> A rewrite to use Python 3 IO facilities?.
See Nadeem's recent work on the bz2 module.

Reported to metatracker. Thanks for the kick :-).
As a reference, I think you are refering to #5863 and #10791. Pretty fine job, I must say.
Nadeem, what do you think?. Is xz in your list?.

> As a reference, I think you are refering to #5863 and #10791. Pretty fine job, I must say.
Thank you :)
> Nadeem, what do you think?. Is xz in your list?.
Yes, it's the next substantial thing I was planning on working on. I don't have
a lot of free time at the moment, though, so it might take a while. But I'll
definitely have it done in time for the release of 3.3 ;)
Looking at the pyliblzma code, I see that it provides the same thread-safety
guarantees as bz2 does. In rewriting, I would like to remove the locking code,
as it complicates the control flow logic of the C code, and doesn't seem like
the sort of thing a compression library should be concerned with. Are there any
objections to this? (Also, if anyone knows why bz2 was written to be thread-safe
in the first place, please enlighten me)

I don't know that much about compression, but I wonder if a threadsafe compression module would enable parallel forms of compression? If yes, then multithreaded might be a big benefit, in light of multicore taking off.
EG: http://www.compression.ca/pbzip2/
It might be worthwhile to check with the authors (of bzip2 and pyliblzma modules) about why they went with thread safety.

I think doesn't makes sense for two threads using at the same time the same compress/decompress object. But I could guess ony reason for the locking code is to avoid crashing the interpreter just having two threads racing each other on an inherently mutable object.
Dan, parallel compression simply split the input data and compress each fragment independly. The actual compression object is not shared, each block is compressed with its own self-contained compression context.

Interesting thing to consider: maybe it'd be better to add support for libarchive, which includes xz support among other things.
http://code.google.com/p/libarchive/ 

> maybe it'd be better to add support for libarchive
That wouldn't be better. We want the C dependency as small
and low-level as possible, so that the dependency is small
and the functionality flexible to implement stuff in Python
on top of it.

Also, if I'm reading the documentation correctly, there isn't any
support for incremental in-memory coding, so it wouldn't be possible
to implement classes analogous to BZ2Compressor/BZ2Decompressor.

Attached is a patch (f3cf187208ea.diff) containing my work so far on
reimplementing the lzma module. So far I've just done the LZMACompressor
and LZMADecompressor classes, but I'm hoping to implement LZMAFile this
weekend.

In uint32_converter, I'm not sure the if statement comparing val and UINT32_MAX makes sense. val was defined to be unsigned long, which is 32bit on 32-bit compiler.

> In uint32_converter, I'm not sure the if statement comparing val and
> UINT32_MAX makes sense. val was defined to be unsigned long, which is
> 32bit on 32-bit compiler.
Yes, on a 32-bit system, this comparison will always be false. However,
on a 64-bit system, an unsigned long will (usually) be 64 bits wide. In
this case, we do need to check that the value fits into a uint32_t.

I've attached another patch (fe2c9998f329.diff) with a more complete
implementation of the lzma module. All that's left now is to write the
documentation, and make sure that the module can build on Windows.

I've attached another patch that includes proper ReST documentation, a
few more tests, and some fixups following Ezio and Amaury's reviews.

Today I played around with lzma support for tarfile based on your last patch (see issue5689). There are a few minor issues that I just wanted to mention, as they break the tarfile testsuite:
- LZMAFile does not expose a name attribute. BZ2File doesn't either (not in 3.x anyway), but GzipFile does.
- LZMAFile does not allow a 'b' in the mode argument, unlike GzipFile and BZ2File.
- The bz2 module exposes many error conditions as standard Python exceptions, e.g. IOError, EOFError. The lzma module uses LZMAError for all errors without distinction.

Thanks for getting the ball rolling with that. I'll want to try and
familiarize myself with the code for tarfile module before I start
changing stuff, but those points shouldn't be hard to fix, one way
or the other.

Nadeem, I started reviewing this path, but now stopped since the patch I reviewed isn't available anymore. Please let us know when the patch has a state where you don't plan to make more changes.

Ah, sorry about that. It probably wasn't necessary to regenerate the
patch after I addressed the comments in each review. The patch should be
stable now; I don't plan to make any further changes for a while.
(At some point I'll want to add a few more tests, and get the module to
compile on Windows. But that can wait.)

I'm a bit worried by the Windows version:
- liblzma can't be compiled by Visual Studio: too many C99 isms, mostly variables declared in the middle of a block. It's doable for sure, but it's a lot of work.
- liblzma is normally compiled with mingw, but we have to be sure that is uses the correct MSCRT C runtime, and what about debug builds?
- All extension modules use static libraries: zlib, expat, sqlite. But a gcc static library can't be used by Visual Studio.
- The way recommended by XZ is to use a precompiled liblzma.dll; Then it should be easy to build an extension module, but its would be the first time that we distribute an extension module which needs a non-system DLL. Is it enough to copy it next to _lzma.pyd? Is there some work to do in the installer?
Too many "but"s :)

Thanks for investigating the Windows situation.
> - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly
> variables declared in the middle of a block. It's doable for sure, but it's a
> lot of work.
I don't think that creating our own MSVC-friendly fork of liblzma is really an
option. Over and above the work of porting it in the first place (and all the
opportunities for bugs to creep in along the way), we'd also have to worry about
keeping up to date with upstream changes. I believe we currently do something
similar with libffi (for ctypes), and the impression I've gotten is that it's
caused a lot of trouble.
> - The way recommended by XZ is to use a precompiled liblzma.dll; Then it
> should be easy to build an extension module, but its would be the first time
> that we distribute an extension module which needs a non-system DLL. Is it
> enough to copy it next to _lzma.pyd? Is there some work to do in the
> installer?
I would guess that this is sufficient, but my knowledge of how Windows DLLs work
is minimal. Could someone with more platform knowledge weigh in on whether this
would work (and if there are any problems it might cause)?

On Sun, Oct 2, 2011 at 3:49 PM, Nadeem Vawda <report@bugs.python.org> wrote:
>
> Nadeem Vawda <nadeem.vawda@gmail.com> added the comment:
>
> Thanks for investigating the Windows situation.
>
> > - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly
> > variables declared in the middle of a block. It's doable for sure, but
> it's a
> > lot of work.
>
> I don't think that creating our own MSVC-friendly fork of liblzma is really
> an
> option. Over and above the work of porting it in the first place (and all
> the
> opportunities for bugs to creep in along the way), we'd also have to worry
> about
> keeping up to date with upstream changes. I believe we currently do
> something
> similar with libffi (for ctypes), and the impression I've gotten is that
> it's
> caused a lot of trouble.
It's much better to contribute patches upstream.
> > - The way recommended by XZ is to use a precompiled liblzma.dll; Then it
> > should be easy to build an extension module, but its would be the first
> time
> > that we distribute an extension module which needs a non-system DLL. Is
> it
> > enough to copy it next to _lzma.pyd? Is there some work to do in the
> > installer?
>
> I would guess that this is sufficient, but my knowledge of how Windows DLLs
> work
> is minimal. Could someone with more platform knowledge weigh in on whether
> this
> would work (and if there are any problems it might cause)?
I've not done much with windows dll's, but I've heard they're pretty similar
to AIX shared libraries which I've done some work with. AIX shared
libraries don't deal with versioning well - if you have two version of the
same library on a system, you have to pop them into two different loader
domains, or suffer unresolved externals at runtime. Then your applications
are in of the two loader domains, and if they're in the wrong one, you again
suffer unresolved externals at runtime.

Based on Amaury's report, I would suggest going forward integrating the xz module for configure-based systems, and letting someone else handle Windows integration later if a solution is found.

> - liblzma can't be compiled by Visual Studio: too many C99 isms,
> mostly variables declared in the middle of a block. It's doable for
> sure, but it's a lot of work.
I'd be in favor of doing so, and then feeding patches upstream.
Hopefully, eventually, the code would compile out of the box on VS.
> - liblzma is normally compiled with mingw, but we have to be sure
> that is uses the correct MSCRT C runtime, and what about debug
> builds?
In principle, it's not necessary to use the same CRT, as long as
we aren't passing CRT objects across DLL boundaries (memory
blocks managed by malloc/free would be candidates). I haven't reviewed
the module to find out whether the liblzma interface involves CRT
objects.
> - The way recommended by XZ is to use a precompiled liblzma.dll; Then
> it should be easy to build an extension module, but its would be the
> first time that we distribute an extension module which needs a
> non-system DLL. Is it enough to copy it next to _lzma.pyd? Is there
> some work to do in the installer?
It wouldn't actually be the first time. We also ship Tcl DLLs. But
it's a pain, so it would be much better if the sources were actually
referenced in the VS project - so we would not need a library at all.

> Based on Amaury's report, I would suggest going forward integrating
> the xz module for configure-based systems, and letting someone else
> handle Windows integration later if a solution is found.
-1000. I feel quite strongly that this should not be added unless there
is also support in the Windows build process for it. I'll see what I
can do, but it may take some time - until then, I urge to stall this
issue, i.e. not proceed with checking it in.

> > Based on Amaury's report, I would suggest going forward integrating
> > the xz module for configure-based systems, and letting someone else
> > handle Windows integration later if a solution is found.
> 
> -1000. I feel quite strongly that this should not be added unless there
> is also support in the Windows build process for it.
Why?

Am 04.10.11 19:08, schrieb Antoine Pitrou:
>
> Antoine Pitrou<pitrou@free.fr> added the comment:
>
>>> Based on Amaury's report, I would suggest going forward integrating
>>> the xz module for configure-based systems, and letting someone else
>>> handle Windows integration later if a solution is found.
>>
>> -1000. I feel quite strongly that this should not be added unless there
>> is also support in the Windows build process for it.
>
> Why?
This module is only useful in the standard library if it is available
on all systems. If it is not available on all systems, it may just as
well be available from PyPI only. Now, as it needs to be available on
Windows, I want to see the actual Windows support, else we would have
to block the release until Windows support is available, possibly then
reverting the module because no Windows support is forthcoming. So it's
easier not to commit in the first place.

> This module is only useful in the standard library if it is available
> on all systems.
Not really. xz is becoming a defacto standard under Linux (and perhaps
other free Unices) while I guess it is marginal under Windows. 
We have other system-specific functionality, and nobody sees it as a bad
thing.

> Not really. xz is becoming a defacto standard under Linux (and perhaps
> other free Unices) while I guess it is marginal under Windows.
> We have other system-specific functionality, and nobody sees it as a bad
> thing.
That's because all system-specific functionality that we have really 
depends on system features which just can't be available elsewhere.
For all functionality that in principle works on all systems, it also
actually works on all systems for Python. In cases where stuff was
only available on Linux even though it could work on other systems,
people *did* see it as a bad thing.

> That's because all system-specific functionality that we have really 
> depends on system features which just can't be available elsewhere.
> For all functionality that in principle works on all systems, it also
> actually works on all systems for Python. In cases where stuff was
> only available on Linux even though it could work on other systems,
> people *did* see it as a bad thing.
Agreed, but it's a probably with the external library. That's like
saying we are responsible if libffi fails building with the AIX
compiler.

> > That's because all system-specific functionality that we have really 
> > depends on system features which just can't be available elsewhere.
> > For all functionality that in principle works on all systems, it also
> > actually works on all systems for Python. In cases where stuff was
> > only available on Linux even though it could work on other systems,
> > people *did* see it as a bad thing.
> 
> Agreed, but it's a probably with the external library. That's like
> saying we are responsible if libffi fails building with the AIX
> compiler.
s/probably/problem/ ;)

I agree with Martin here. We should *NOT* have first and second class OS support, if we can avoid it.
That said, I wonder what happens in Windows with the BZ2 module, for instance :-?. Do we include the BZ2 sourcecode to compile it under windows?.
I know, for instance, that Windows 2.* builds include Berkeley DB.

> I agree with Martin here. We should *NOT* have first and second class
> OS support, if we can avoid it.
The key word being "if we can avoid it".
Jesus, if you are a Windows expert, your contribution is welcome.

For bz2, Tools/buildbot/external-common.bat has code to download bz2 source, and PCbuild/_bz2.vcproj include and compile these files together with _bz2.pyd.
The _ssl module does a similar thing, except that libeay32.lib and libssleay32.lib are built in a separate step.

> That said, I wonder what happens in Windows with the BZ2 module, for instance :-?.
External code currently lives at 
http://svn.python.org/projects/external/. The build process gets it from 
there, and we may
have local modifications to libraries where necessary.

> I agree with Martin here. We should *NOT* have first
> and second class OS support, if we can avoid it.
Ok but who will do the job? If nobody is motivated to fix compiler issues, it would be a pity to not add the module for that.

Antoine, I am a Linux/Solaris/Illumos guy. I only use Windows (virtualized) to sync my iPhone with iTunes :)

Wow, this discussion has gotten quite busy while I've been travelling...
Martin, could you explain what the problems are with bundling a precompiled DLL
for Windows? I am willing to do the work of getting liblzma to compile with VS
if necessary, but I don't know how receptive the upstream maintainer will be to
the changes. If I can explain how lack of VS support is a problem for us, the
request should carry more weight.

Am 06.10.11 13:01, schrieb Nadeem Vawda:
>
> Nadeem Vawda<nadeem.vawda@gmail.com> added the comment:
>
> Wow, this discussion has gotten quite busy while I've been travelling...
>
> Martin, could you explain what the problems are with bundling a precompiled DLL
> for Windows?
Off-hand, it's only minor issues: e.g. when running Python out of its 
build directory, the DLL must be in the same directory. Now, since
the output directory differs depending on build option, getting the
DLL there might be tricky.
Things I wonder about and couldn't quickly answer from the web:
where exactly is the DLL that we would use?
is there a AMD64 version of it?
Does it come with import libraries usable by VS?

On http://tukaani.org/xz, I downloaded the file named xz-5.0.3-windows.zip.
It contains precompiled dlls for both platforms: bin_i486/liblzma.dll and bin_x86_64/liblzma.dll
Unfortunately, there is no import library for VS. It should not be too difficult to make one, though: the provided headers are C89, so it's enough to write stubs for the functions used by the extension module.

Hmm... according to http://git.tukaani.org/?p=xz.git;a=blob;f=windows/README-Windows.txt;hb=HEAD#l80,
the MinGW-compiled static libs *can* be used with MSVC. Not sure how
reliable the information is, but it's worth a try at least.

Ah indeed, the zip archive contains a doc/liblzma.def which can be used to build a liblzma.lib

Hey, today I learnt something about mingw!
"""Rename liblzma.a to e.g. liblzma_static.lib and tell MSVC to link against it."""
Apparently mingw can generate COFF libraries. This may simplify things *a lot*.

*Very* good news for lzma on windows: The precompiled static library liblzma.a works very well with MSVC (tested with VS2008 on Windows XP, 32bit). This was a surprise for me...
Here is a patch for the win32 build files, to be applied after Nadeem's.
I did not update the svn "external" repository, for my local copy I simply extracted xz-5.0.3-windows.zip.
Tests pass in release and debug builds, and depends.exe shows no special requirements for _lzma.pyd. I could not test win64.
Nadeem, in test.support precisionbigmemtest was recently renamed to bigmemtest, could you update your patch?

Awesome stuff! I'll post an updated patch during the course of the day.
Martin: I've been having problems with Rietveld lately, so I'm posting
my replies to your comments here instead.
>> Modules/_lzmamodule.c:115: return _PyBytes_Resize(buf, size + BIGCHUNK);
> This has quadratic performance.
Correct. I copied the algorithm from _io.FileIO, under the assumption
that there was a reason for not using a simpler O(n log n) doubling
strategy. Do you know of any reason for this? Or is it safe to ignore it?
>> Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS
> It seems that the Windows version at least is not thread-safe. If so, you
> would need an LZMA lock when releasing the GIL.
Does the class need to be thread-safe, though? ISTM that there isn't any
sensible use case for having two threads feeding data through the same
compressor concurrently.
(If we *do* want thread-safety, then it doesn't matter whether the
underlying lib is internally thread-safe or not. We would still need to
guard against the possibility of the _lzmamodule.c code in one thread
modifying the lzma_stream's input or output pointer while lzma_code is
operating on the stream's data in another thread.)

> >> Modules/_lzmamodule.c:115: return _PyBytes_Resize(buf, size + BIGCHUNK);
> > This has quadratic performance.
> 
> Correct. I copied the algorithm from _io.FileIO, under the assumption
> that there was a reason for not using a simpler O(n log n) doubling
> strategy. Do you know of any reason for this? Or is it safe to ignore it?
I don't know, but I'd say it's safe to ignore it.
Maybe open a separate bug about that.
Generally we use a less-than-doubling strategy, to conserve memory (see
e.g. bytearray.c).
(note: it is O(n), not O(n log n))
> >> Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS
> > It seems that the Windows version at least is not thread-safe. If so, you
> > would need an LZMA lock when releasing the GIL.
> 
> Does the class need to be thread-safe, though? ISTM that there isn't any
> sensible use case for having two threads feeding data through the same
> compressor concurrently.
We certainly want to avoid crashes so, if that's what one risks from
using lzma from several threads, we should protect against it.

>>> Modules/_lzmamodule.c:364: Py_BEGIN_ALLOW_THREADS
>> It seems that the Windows version at least is not thread-safe. If so, you
>> would need an LZMA lock when releasing the GIL.
>
> Does the class need to be thread-safe, though?
As a matter of principle, Python code must not be able to crash the
interpreter or corrupt memory. There are known bugs in this area,
but if it's known in advance that an issue exists, we should avoid
it.
> ISTM that there isn't any
> sensible use case for having two threads feeding data through the same
> compressor concurrently.
Right. So having a per-compressor mutex lock would be entirely
reasonable. I could also accept a per-module lock. I could even
accept the GIL, and if no other code is forth-coming, I would
prefer to keep holding the GIL during comprssion over risking
crashes.
> (If we *do* want thread-safety, then it doesn't matter whether the
> underlying lib is internally thread-safe or not. We would still need to
> guard against the possibility of the _lzmamodule.c code in one thread
> modifying the lzma_stream's input or output pointer while lzma_code is
> operating on the stream's data in another thread.)
I haven't reviewed the module in this respect. If you say that it
wouldn't be thread-safe even if LZMA was compiled as thread-safe,
then this definitely must be fixed.
To elaborate on the policy: giving bogus data in cases of race
conditions is ok; crashing the interpreter or corrupting memory
is not. If bogus data is given, it would be useful if the bogosity
can be specified (e.g. when multiple threads read from the same
POSIX file concurrently, they also get bogus data, but in a manner
where each input byte is given to exactly one thread).

>> Correct. I copied the algorithm from _io.FileIO, under the assumption
>> that there was a reason for not using a simpler O(n log n) doubling
>> strategy. Do you know of any reason for this? Or is it safe to ignore it?
>
> I don't know, but I'd say it's safe to ignore it.
To elaborate: ISTM that it's actually a bug in FileIO. I can imagine
where it's coming from (i.e. somebody feeling that overhead shouldn't
grow unbounded), but I think that's ill-advised - *if* somebody really
produces multi-gigabyte data (and some people eventually will), they
still deserve good performance, and they will be able to afford the
memory overhead (else they couldn't store the actual output, either).
> Generally we use a less-than-doubling strategy, to conserve memory (see
> e.g. bytearray.c).
Most definitely. In case it isn't clear (but it probably is here):
any constant factor > 1.0 will provide amortized linear complexity.

> To elaborate on the policy: giving bogus data in cases of race
> conditions is ok; crashing the interpreter or corrupting memory
> is not. If bogus data is given, it would be useful if the bogosity
> can be specified (e.g. when multiple threads read from the same
> POSIX file concurrently, they also get bogus data, but in a manner
> where each input byte is given to exactly one thread).
OK, that makes sense. My next patch will include per-compressor locks for
LZMACompressor and LZMADecompressor.
> To elaborate: ISTM that it's actually a bug in FileIO.
I've filed issue 13159 proposing that FileIO be fixed.

Patch update time! Noteworthy changes:
* Windows build support, thanks to Amaury
* Thread safety for LZMACompressor and LZMADecompressor, by means of per-instance locks
* O(n) buffer growth strategy
I've tried running the tests on Windows, but I've been getting failures
that don't look like they're related to the new code. I think they are
due to issue 7443. I'll try and tweak my Windows setup a bit over the
weekend to see if I can get things working smoothly.

I have imported xz-5.0.3 into the externals repository now.

After disabling a bunch of services and skipping the tests that are
failing on the Windows buildbots, it seems that the failures I was
getting last week have disappeared. It looks like they were due to the
Windows search indexer preventing temp directories from being deleted... >_>
> I have imported xz-5.0.3 into the externals repository now.
I tried to test this out, but it looks like liblzma.a is missing from the
SVN repo. There should be files with that name in the bin_i486 and
bin_x86-64 directories.
Apart from this, is there anything else that needs to be done before this
change can be committed to the main repo?

> Apart from this, is there anything else that needs to be done before this
> change can be committed to the main repo?
Things look fine to me, apart from a couple of minor comments I've made on the review tool.

What is the status on this, Nadeem? It would be lovely to get the feature in the stdlib.

I just added the missing files liblzma.a to the "externals" repository.
If someone can quickly check that it works on win32, I don't have anything else to add to this change.

Everything is ready to go, as far as I'm concerned. Once I've tested the
Windows build using the binary from the externals repository, I'll be ready
to commit it to the main repository.

I've tested the Windows build, and it works fine. If there are no objections,
I'll commit on Saturday.
Btw, what is the preferred channel for communicating with buildbot owners?
I'll want to send out an email asking that all the *nix bots have
liblzma-dev installed, so that they can actually run the tests...

I've tested under 64-bit Windows and it worked fine.

I wrote a short review on http://bugs.python.org/review/6715/show 

Victor: Thanks for the review; I've replied to your comments and updated
the patch. Let me know what you think of the changes.

The last patch (9276fc685c05.diff) looks good to me. Go ahead for the commit!

New changeset 74d182cf0187 by Nadeem Vawda in branch 'default':
Issue #6715: Add module for compression using the LZMA algorithm.
http://hg.python.org/cpython/rev/74d182cf0187 

The patch has been committed. I'm leaving this issue open for a while
longer, until more of the buildbots are building and testing the module.

Not meaning to sound petty, but wouldn't it be common etiquette to retain some original copyright notice from original code intact..?

> Not meaning to sound petty, but wouldn't it be common etiquette to
> retain some original copyright notice from original code intact..?
It seemed to me that Nadeem had rewritten everything from scratch. Is
there any code of yours in the current module?

I've opened Issue13507 to track adding liblzma to the OS X installer builds.

Ah, I thought that he had reused most of the original C code in _lzmamodule.c not replaced by python code, but I see that not being the case now (only slight fragments;).
Oh well, I thought that I'd still earned a note with some slight credit at least, but I guess I won't go postal or anything in lack of either.. :p

> Oh well, I thought that I'd still earned a note with some slight credit
at least
I completely agree. Sometimes people get credit for simple bug fixes (count me among them) so the author of the first working implementation deserves some recognition IMO.

Nadeem: Instead of duplicating the list of archiving/compression modules in each doc, what about only linking to the shutil doc for archives and the archiving.rst file? (I can make the patch, just wanted feedback first)

>> Not meaning to sound petty, but wouldn't it be common etiquette to
>> retain some original copyright notice from original code intact..?
>
> It seemed to me that Nadeem had rewritten everything from scratch. Is
> there any code of yours in the current module?
That is correct. Based on my experience with the bz2 module, rewriting
from scratch seemed to make more sense.
>> Oh well, I thought that I'd still earned a note with some slight
> credit at least
>
> I completely agree. Sometimes people get credit for simple bug fixes
> (count me among them) so the author of the first working implementation
> deserves some recognition IMO.
Of course. How does this look?
 diff --git a/Misc/ACKS b/Misc/ACKS
 --- a/Misc/ACKS
 +++ b/Misc/ACKS
 @@ -502,6 +502,7 @@
 Peter van Kampen
 Rafe Kaplan
 Jacob Kaplan-Moss
 +Per Øyvind Karlsen
 Lou Kates
 Hiroaki Kawai
 Sebastien Keim
 diff --git a/Misc/NEWS b/Misc/NEWS
 --- a/Misc/NEWS
 +++ b/Misc/NEWS
 @@ -400,6 +400,7 @@
 -------
 
 - Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm.
 + Thanks to Per Øyvind Karlsen for the initial implementation.
 
 - Issue #13487: Make inspect.getmodule robust against changes done to
 sys.modules while it is iterating over it.

> Instead of duplicating the list of archiving/compression modules in
> each doc, what about only linking to the shutil doc for archives and
> the archiving.rst file?
Sure, go ahead. I actually hadn't realized that each section of the
library docs had a "sub-index" page like that until now...

--- a/Misc/NEWS
 +++ b/Misc/NEWS
 @@ -400,6 +400,7 @@
 -------
 
 - Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm.
 + Thanks to Per Øyvind Karlsen for the initial implementation.
The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message should be enough. Lately I’ve noticed some attributions in NEWS, but it’s usually not done (as redundant).

what about a mention in lzmamodule.c?

> I actually hadn't realized that each section of the library docs had a "sub-index" page like that
That’s multiple-entry navigation :) You can jump to a module page, or use the index, or use the search, or browse and see the sub-indexes.
As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the archiving subsection, there’s already a link to the subsection index, so I just removed the "See also zlib, etc." lines (except for the link from zlib to gzip). I added a link from archiving.rst to shutil and more links and reST targets in shutil.

> The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message
> should be enough. Lately I’ve noticed some attributions in NEWS, but
> it’s usually not done (as redundant).
I generally add attributions in NEWS since that's where most people
learn about changes (for changes which aren't in whatsnew).

New changeset 6cde416ef03b by Nadeem Vawda in branch 'default':
Credit Per Øyvind Karlsen for the initial implementation of the lzma module (issue #6715).
http://hg.python.org/cpython/rev/6cde416ef03b 

> As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the
> archiving subsection, there’s already a link to the subsection index,
> so I just removed the "See also zlib, etc." lines (except for the link
> from zlib to gzip). I added a link from archiving.rst to shutil and
> more links and reST targets in shutil.
Looks good to me.
> The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message
> should be enough. Lately I’ve noticed some attributions in NEWS, but
> it’s usually not done (as redundant).
According to Doc/whatsnew/3.3.rst:
 "The maintainer will go through Misc/NEWS periodically and add
 changes; it's therefore more important to add your changes to
 Misc/NEWS than to this file."
> what about a mention in lzmamodule.c?
Done and committed.

I’ll commit my doc patch to all branches later.
I checked the 4 red 3.3 builbots and they can’t build _lzma.

> I’ll commit my doc patch to all branches later.
OK, great.
> I checked the 4 red 3.3 builbots and they can’t build _lzma.
For the record, the 3.x buildbots that currently aren't able to build the
_lzma module are:
* x86 Ubuntu Shared
* sparc solaris10 gcc
* AMD64 Snow Leopard 2
* x86 debian parallel
* ARM Ubuntu
* x86 Tiger
* AMD64 OpenIndiana
* x86 OpenIndiana
* x86 FreeBSD 7.2
* x86 FreeBSD 6.4
Additionally, the AMD64 debian bigmem, PPC Tiger and PPC Leopard builders
have been offline/broken since before changeset 74d182cf0187 was committed.

Le jeudi 08 décembre 2011 à 17:40 +0000, Nadeem Vawda a écrit :
> Nadeem Vawda <nadeem.vawda@gmail.com> added the comment:
> 
> > I’ll commit my doc patch to all branches later.
> 
> OK, great.
> 
> > I checked the 4 red 3.3 builbots and they can’t build _lzma.
> 
> For the record, the 3.x buildbots that currently aren't able to build the
> _lzma module are:
Ok, I've e-mailed the owners.
Will someone write an entry in the "what's new" file for 3.3?
(Doc/whatsnew/3.3.rst)
Regards
Antoine.

New changeset 66df5ace0eee by Nadeem Vawda in branch 'default':
What's New in Python 3.3: Add entry for lzma module (issue #6715).
http://hg.python.org/cpython/rev/66df5ace0eee 

> Ok, I've e-mailed the owners.
Thanks. I was just thinking I should send a reminder, in case they
missed the note in my announcement on python-dev.
> Will someone write an entry in the "what's new" file for 3.3?
Done

Is there any reason why this issue is still open?

Yes, almost half of the buildbots still don't have the xz-utils headers
installed, and thus are not building/testing the lzma module. I've kept
the issue open as a reminder to myself to follow up on this.

New changeset 11bd2d32b4e8 by Éric Araujo in branch '3.2':
Improve interlinking of archiving/compression modules docs.
http://hg.python.org/cpython/rev/11bd2d32b4e8
New changeset 1cb9b8126534 by Éric Araujo in branch 'default':
Merge edits from 3.2 (#13716, #1040439, #2945, #13770, #6715)
http://hg.python.org/cpython/rev/1cb9b8126534 

BTW, any plans on a PyPI backport for fun and profit?

> BTW, any plans on a PyPI backport for fun and profit?
At present, no. I'll look into it later in the year, but at the moment
I don't have the time to work on it - I suspect the parts written in C
will require substantial changes to work under 2.x.

New changeset 7c22281e967c by Éric Araujo in branch '2.7':
Improve interlinking of archiving/compression modules docs.
http://hg.python.org/cpython/rev/7c22281e967c 

Why is this issue still open?

I had intended to wait until the code had been tested on all of the
buildbots before closing it. However, it's taking a while to get xz-utils
installed on the last few bots, so it doesn't make sense to keep the
issue open for this.

Apologies for noise, but since a backport was discussed, I'm mentioning this here.
I've started implementing a backport, currently working and tested on Mac OS X and Linux, back to Python 3.0 - supporting Python 2 would be nice but probably significantly more work (assistance welcome - please get in touch on github):
https://github.com/peterjc/backports.lzma
Assuming Nadeem has no objections, I intend to publish this on PyPI (I have tried to email directly but perhaps I'm using an old email address or he has been busy, another reason for commenting here). Thanks!

Apologies again for the noise, but I've just made the first public release of the lzma backport at http://pypi.python.org/pypi/backports.lzma/ with the repository as mentioned before at https://github.com/peterjc/backports.lzma
I have tested this on Python 2.6, 2.7 and 3.0 through 3.3 under Linux and Mac OS. I've not tried building this on Windows yet, but it should be possible and I will be reviewing Amaury's notes on this thread.
My thanks to Nadeem and Per for looking over this with helpful feedback, and agreeing to the use of the 3-clause BSD license for the backport.
If anyone has any further questions about the backport, please get in touch via GitHub or email me.