homepage

This bug is still present in Python 3.3. Some people use urlsafe strings *a lot*. And they find that the (obvious) implementation in base64.py is way too slow. In fact, I found myself writing the following a few times in App Engine libraries:
 # This is 3-4x faster than urlsafe_b64decode() 
 urlsafe = base64.b64encode(<arg>)
 return urlsafe.rstrip('=').replace('+', '-').replace('/', '_')
Someone from YouTube told me they patch the stdlib for the same reason (they claim their solution is 10x faster).
IIUC the cause of the slowness is the (mostly irrelevant) generality in b64encode and the _translate() function.

This is a feature request, not a bug. Marking for 3.3.

Guido, would your patch be enough?. Is a 3x improvement enough?. What improvement could be acceptable?. Implemented in C in "binascii" module?

Why would it be a feature? There is no change in semantics, only in speed. Do we have a policy that forbids performance enhancements in bugfix releases? (It doesn't really matter, I'm just curious. Also, if it really was a feature request, it couldn't be introduced once the 3.3 beta has started, which would strike me as silly.)
Anyway, I don't think my hack should be applied literally; I was more thinking of memoizing the translation table and inlining the _translate call.

> Do we have a policy that forbids performance enhancements in bugfix
> releases?
Yes, except when it fixes a regression. I don't know where that policy is codified, but it seems quite ingrained.
It also means the window for getting it in 3.3 is getting narrow, the beta being so soon :)

Here's a patch. It makes urlsafe_b64{en,de}code about 4x faster on my machine. The tests in base64_test.py still pass.

$ ./python -m timeit -s "from base64 import urlsafe_b64encode; url=bytes(range(256))" "urlsafe_b64encode(url)"
10000 loops, best of 3: 73.5 usec per loop
$ ./python -m timeit -s "from base64 import b64encode; url=bytes(range(256))" "b64encode(url).replace(b'+', b'-').replace(b'/', b'_')"
100000 loops, best of 3: 18.9 usec per loop
$ ./python -m timeit -s "from base64 import b64encode; url=bytes(range(256)); tr=bytes.maketrans(b'+/', b'-_')" "b64encode(url).translate(tr)"
100000 loops, best of 3: 15.6 usec per loop

Heh. I'd forgotten about maketrans. Here's a patch that kills the _translate() helper function completely.

You're faster than me.
I think that asserts should be replaced by if/raise.

I didn't touch that assert.

Antoine thinks so: http://bugs.python.org/issue15139 :-)
I think we should have an official policy, though.

So, can I check this into 3.3?

I haven't tested the patch, but it looks fine to me.

Submitted.

Here's a backport of the patch to 2.7. It's pretty rad, and basically identical to how YouTube monkeypatches base64.
Not sure what will happen to this patch. According to recent discussion on the list (e.g. https://mail.python.org/pipermail/python-dev/2015-May/140380.html ), performance improvements are open for inclusion in 2.7 if anyone wants to bother with merging this in and taking on the review / maintenance burden.
I'm OK with just publishing it for others to merge in with their own private versions of Python. It is only relevant if you use base64 a lot. :)

I'm going to apply the 2.7 patch, except I'm going to leave the (now unused) _translate() function in place, in case it's used anywhere (minimizing the risk of breaking anything).

New changeset 309cbbc32491 by Guido van Rossum in branch '2.7':
Issue #15138: Speed up base64.urlsafe_b64* considerably (2.7 backport).
https://hg.python.org/cpython/rev/309cbbc32491 

And just in case someone objects to my breaking the policy stated by Antoine in msg163432: see http://bugs.python.org/issue4753 -- it's not actually a real policy.