homepage

> I think it would be a mistake to make the stdlib use this for most 
> notions of what a "character" is, as I said this notion is also 
> inaccurate. Having an iterator library somewhere that you can use and 
> compose is great, changing the internal workings of string operations 
> would be a major change, and not entirely productive.
Agreed. 
I won't pretend to be able to predict what Python 5.0 will bring *wink* 
but there's too much history around the "code point = character" notion 
for the language to change now.
If the language can expose a grapheme iterator, then people can 
experiment with grapheme-based APIs in libraries.
(By grapheme I mean "extended grapheme cluster", but that's a mouthful. 
Sorry linguists.)
What do you think of these as a set of grapheme primitives?
(1) is_grapheme_break(string, i)
Return True if a grapheme break would occur *before* string[i].
(2) graphemes(string, start=0, end=len(string))
Iterate over graphemes in string[start:end].
(3) graphemes_reversed(string, start=0, end=len(string))
Iterate over graphemes in reverse order.
I *think* is_grapheme_break would be enough for people to implement 
their own versions of graphemes and graphemes_reversed. Here's an 
untested version:
 def graphemes(string, start, end):
 cluster = []
 for i in range(start, end):
 c = string[i]
 if is_grapheme_break(string, i):
 if i != start:
 # don't yield the empty cluster at Start Of Text
 yield ''.join(cluster)
 cluster = [c]
 else:
 cluster.append(c)
 if cluster:
 yield ''.join(cluster)
Regarding is_grapheme_break, if I understand the note here:
https://www.unicode.org/reports/tr29/#Testing
one never needs to look at more than two adjacent code points to tell 
whether or not a grapheme break will occur between them, so this ought 
to be pretty efficient. At worst, it needs to look at string[i-1] and 
string[i], if they exist.