[Python-3000] Lazy strings (was Re: Py3k release schedule worries)

Fri Jan 12 18:48:25 CET 2007

Guido van Rossum wrote:
>> As discussed on that page, the current version of the patch could cause
>> crashes in low-memory conditions. I welcome suggestions on how best to
>> resolve this problem. Apart from that fly in the ointment I'm pretty
>> happy with how it all turned out.
> What kind of crashes? The right thing to do is to raise MemoryError.
> Is there anything besides sheer will power that prevents that?
Nothing *has* prevented that; the relevant code already calls 
PyErr_NoMemory(). The problem is that *its* callers currently won't 
notice, and continue on their merry way.
My patch adds a new wrinkle to the API: PyUnicode_AS_UNICODE() can now 
fail. And currently when it fails it returns NULL. (Why could it 
fail? Under the covers, PyUnicode_AS_UNICODE() may attempt to allocate 
memory.)
Without the patch PyUnicode_AS_UNICODE() always works. Since no caller 
ever expects it to fail, code looks like this:
 static
 int fixupper(PyUnicodeObject *self)
 {
 Py_ssize_t len = self->length;
 Py_UNICODE *s = PyUnicode_AS_UNICODE(self);
 int status = 0;
 while (len-- > 0) {
 register Py_UNICODE ch;
 ch = Py_UNICODE_TOUPPER(*s);
 ...
And there you are; when s is NULL, Python crashes.
In the patch comments I proposed four possible solutions for this 
problem, listed in order of least-likely to most-likely. I just came up 
with a fifth one, and I'll include it here.
 1. Redefine the API such that PyUnicode_AS_UNICODE() is allowed to
 return NULL, and fix every place in the Python source tree that
 calls it to check for a NULL return. Document this with strong
 language for external C module authors.
 2. Pre-allocate the str buffer used to render the lazy string
 objects. Update this buffer whenever the size of the string
 changes. That moves the failure to a better place for error
 reporting; once again PyUnicode_AS_UNICODE() can never fail. But
 this approach also negates a healthy chunk of what made the patch
 faster.
 3. Change the length to 0 and return a constant empty string. 
 Suggest that users of the Unicode API ask for the pointer *first*
 and the length *second*.
 4. Change the length to 0 and return a previously-allocated buffer of
 some hopefully-big-enough-size (4096 bytes? 8192 bytes?), such
 that even if the caller iterates over the buffer, odds are good
 they'll stop before they hit the end. Again, suggest that users
 of the Unicode API ask for the pointer *first* and the length
 *second*.
 5. The patch is not accepted.
(You see what an optimist I am.)
I'm open to suggestions (and patches!) of other approaches to solve this 
problem.
Cheers,
/larry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-3000/attachments/20070112/56f21a03/attachment.html