python unicode regular expressions

Question

I read a file with the code below and then I want to find words in the file using re library. The file contains Turkish characters. So I decode file using utf-8. re library doesn't know Turkish character. Below code isn't working.

 text= unicodedata.normalize("NFKD",codecs.open(os.path.abspath("texts/kopru1.txt"),"rb").read().decode("utf-8"))
 text=text.replace("\r\n"," ").lower()
 aa= re.findall(ur"[a-zçşıöü]+", text,re.UNICODE) 

Although "ayşe" is a word, this word seems as of "ays" and "e".

• kqr: Could you give some example data and tell us what you want to do?
• hinzir: example string is "ayşe kulin köprü". I want to find words in this string.
• PurityLake: If you want to split by word why not use text.split(" ")?

Answer 1

Use the escape sequence \w which means "a letter of any kind." Just getting an example sentence from wikipedia:

>>> text = u'Türkî-i çin (güzel güneş) terkiplerinde de gördüğümüz'
>>> re.findall(r'\w+', text, re.UNICODE)
['Türkî', 'i', 'çin', 'güzel', 'güneş', 'terkiplerinde', 'de', 'gördüğümüz']

• hinzir: I had done before. And again I did. But the code isn't still working.
• kqr: @hinzir what does your text variable look like before you try to match on it?
• kqr: @hinzir Oh, right. I've updated my reply with additional hints. See if it helps.
• hinzir: @kgr you are wrong about replace method. docs.python.org/2/library/string.html#string.replace "Return a copy of string s with all occurrences of substring old replaced by new. "
• kqr: @hinzir That's really, really weird. When I tested yesterday I could swear it wouldn't return a new string, but now that I try it does.