Return to Revisions

I have an unicode encoded (with BOM) source file and some string that contains unicode symbols. I want to replace all characters that not belong to a defined character set with an underscore.

# coding: utf-8 
import os
import sys
import re
t = "🙂 [°] \n € dsf $ ¬ 1 Ä 2 t34円Ú";
print re.sub(r'[^A-Za-z0-9 !#%&()*+,-./:;<=>?[\]^_{|}~"\'\\]', '_', t, flags=re.UNICODE)

output:

____ [__] _ ___ dsf _ __ 1 __ 2 t3__4__

But each character is replaced by a number of its underscores that may be equal to the bytes in its unicode representation.