I have a strange problem. A friend of mine sent me a text file. If I copy/paste the text and then paste it into my text editor and save it, the following code works. If I choose the option to save the file directly from the browser, the following code breaks. What's going on? Is it the browser's fault for saving invalid characters?

This is an example line.

When I save it, the line says

What�s going on?

When I copy/paste it, the line says

What’s going on?

This is the code:

import codecs
def do_stuff(filename):
 with codecs.open(filename, encoding='utf-8') as f:
 def process_line(line):
 return line.strip()
 lines = f.readlines()
 for line in lines:
 line = process_line(line)
 print line
do_stuff('stuff.txt')

This is the traceback I get:

Traceback (most recent call last):
 File "test-encoding.py", line 13, in <module>
 do_stuff('stuff.txt')
 File "test-encoding.py", line 8, in do_stuff
 lines = f.readlines()
 File "/home/somebody/.python/lib64/python2.7/codecs.py", line 679, in readlines
 return self.reader.readlines(sizehint)
 File "/home/somebody/.python/lib64/python2.7/codecs.py", line 588, in readlines
 data = self.read()
 File "/home/somebody/.python/lib64/python2.7/codecs.py", line 477, in read
 newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 4: invalid start byte

What can I do in such cases?

How can I distribute the script if I don't know what encoding the user who runs it will use?

Fixed:

codecs.open(filename, encoding='utf-8', errors='ignore') as f: