homepage

While the netloc/path parts of URLs are scheme-specific, and urlparse can be forgiven for refusing to parse them for unknown schemes, the query and fragment parts are standardized, and should be parsed for unrecognized schemes.
According to Wikipedia:
------------------
Internet standard STD 66 (also RFC 3986) defines the generic syntax to be used in all URI schemes. Every URI is defined as consisting of four parts, as follows:
<scheme name> : <hierarchical part> [ ? <query> ] [ # <fragment> ]
------------------
http://en.wikipedia.org/wiki/URI_scheme#Generic_syntax
Here is a demonstration of what urlparse currently does:
>>> urlparse.urlsplit('myscheme://netloc/path?a=b#frag')
SplitResult(scheme='myscheme', netloc='', path='//netloc/path?a=b#frag', query='', fragment='')
>>> urlparse.urlsplit('http://netloc/path?a=b#frag')
SplitResult(scheme='http', netloc='netloc', path='/path', query='a=b', fragment='frag')

New changeset 79e6ff3d9afd by Senthil Kumaran in branch '2.7':
Issue9374 - Generic parsing of query and fragment portion of urls for any scheme
http://hg.python.org/cpython/rev/79e6ff3d9afd
New changeset a9d43e21f7d8 by Senthil Kumaran in branch '3.2':
Issue9374 - Generic parsing of query and fragment portion of urls for any scheme
http://hg.python.org/cpython/rev/a9d43e21f7d8
New changeset 152c78b94e41 by Senthil Kumaran in branch 'default':
Issue9374 - Generic parsing of query and fragment portion of urls for any scheme
http://hg.python.org/cpython/rev/152c78b94e41 

Thanks for raising this issue, Nick. Yes, I verified in both RFC 3986 and 2396 and realized we can safely adopt a generic parsing system for query and fragment portions of the urls for any scheme. Since it was supported in earlier versions too, I felt it was good move to backport too.
Fixed in all versions. 
Thanks!

Removing the module attributes causes third-party code to break. See one example here: http://lists.idyll.org/pipermail/testing-in-python/2012-July/005082.html 

Better link: https://github.com/pypa/pip/issues/552 

this breaks the following upstream builds:
createrepo, linkchecker, gwibber, pegasus-wm
there is no need to remove is_hierarchical on the branches. it's not used by urlparse at all.
is it safe to just keep the uses_query and uses_fragment lists on the branches as well?
raising to a release blocker, I consider this as a regression for the 2.7 and 3.2 release series.

Senthil, either the module globals should be re-added for compatibility, or the commits should be reverted, IMO.

New changeset a0b3cb52816e by Georg Brandl in branch '3.2':
Closes #9374: add back now-unused module attributes; removing them is a backward compatibility issue, since they have a public-seeming name.
http://hg.python.org/cpython/rev/a0b3cb52816e
New changeset c93fbc2caba5 by Georg Brandl in branch 'default':
Closes #9374: merge with 3.2
http://hg.python.org/cpython/rev/c93fbc2caba5
New changeset a43481210964 by Georg Brandl in branch '2.7':
Closes #9374: add back now-unused module attributes; removing them is a backward compatibility issue, since they have a public-seeming name.
http://hg.python.org/cpython/rev/a43481210964 

Oops. I had not seen Eric and Mattiahs comment to this issue, which
pointed out to the problem. Sorry for not acting on this.
Thanks Georg for adding those module attributes back.
On Fri, Aug 24, 2012 at 9:17 AM, Roundup Robot <report@bugs.python.org> wrote:
>
> Roundup Robot added the comment:
>
> New changeset a0b3cb52816e by Georg Brandl in branch '3.2':
> Closes #9374: add back now-unused module attributes; removing them is a backward compatibility issue, since they have a public-seeming name.
> http://hg.python.org/cpython/rev/a0b3cb52816e
>
> New changeset c93fbc2caba5 by Georg Brandl in branch 'default':
> Closes #9374: merge with 3.2
> http://hg.python.org/cpython/rev/c93fbc2caba5
>
> New changeset a43481210964 by Georg Brandl in branch '2.7':
> Closes #9374: add back now-unused module attributes; removing them is a backward compatibility issue, since they have a public-seeming name.
> http://hg.python.org/cpython/rev/a43481210964
>
> ----------
> resolution: remind -> fixed
> status: open -> closed
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue9374>
> _______________________________________

After encountering an instance of people relying on fragment not being parsed for "irc://" URLs, with resulting breakage, I don't think we should change this in point releases. IOW, it's fine for 3.3.0, but not for 2.7.x or 3.2.x.
It may be fixing a bug, but the bug is not obvious and the fix is not backward compatible. I therefore suggest to roll back the commits to 3.2 and 2.7.

If there is a list of known protocols that don't use the fragment, can't we include it in urlparse as we already do in Lib/urlparse.py:34?
If #channel in irc://example.com/#channel should not be parsed as fragment, then this can be considered as a regression. This doesn't necessary mean that the whole change is a regression though.

People make up URL schemes all the time, irc:// is not a special case. This change will mean breakage for them, unwarranted.

One would hope that people making up URI schemes would follow the generic syntax (and thus irc would be an exception), but as the risk exists I agree we should not break code in bugfix releases.

New changeset 950320c70fb4 by Georg Brandl in branch 'default':
Add a versionchanged note for #9374 changes.
http://hg.python.org/cpython/rev/950320c70fb4 

> It may be fixing a bug, but the bug is not obvious and the fix is not
> backward compatible. I therefore suggest to roll back the commits to
> 3.2 and 2.7.
Well, the bug is quite obvious to me :-) (just hit it here)
The fix for those who want the old behaviour is obvious: just pass `allow_fragments=False` to urlparse(). OTOH, if you revert the fix, patching things manually is quite cumbersome.

Reviewed the issue and correct rollbacks and commits were applied.
This ticket should be closed. Thanks!