I have an unicode encoded (with BOM) source file and some string that contains unicode symbols. I want to replace all characters that not belong to a defined character set with an underscore.

# coding: utf-8 
import os
import sys
import re
t = "🙂 [°] \n € dsf $ ¬ 1 Ä 2 t34円Ú";
print re.sub(r'[^A-Za-z0-9 !#%&()*+,-./:;<=>?[\]^_{|}~"\'\\]', '_', t, flags=re.UNICODE)

output:

____ [__] _ ___ dsf _ __ 1 __ 2 t3__4__

But each character is replaced by a number of its underscores that may be equal to the bytes in its unicode representation.

I have an unicode encoded (with BOM) source file and some string that contains unicode symbols. I want to replace all characters that not belong to a defined character set with an underscore.

# coding: utf-8 
import os
import sys
import re
t = "🙂 [°] \n € dsf $ ¬ 1 Ä 2 t34円Ú";
print re.sub(r'[^A-Za-z0-9 !#%&()*+,-./:;<=>?[\]^_{|}~"\'\\]', '_', t, flags=re.UNICODE)

output: ____ [__] _ ___ dsf _ __ 1 __ 2 t3__4__
expected: _ [_] _ _ dsf _ _ 1 _ 2 t3_4_

But each character is replaced by a number of its underscores that may be equal to the bytes in its unicode representation.

Maybe an additional problem:

In the actual problem the strings is read from a unicode file by another python module and I do not know if it handles the unicodeness correctly. So may be the string variable is marked as ascii but contains unicode sequences.