UnicodeDecodeError message in python

Question

I am relatively new to Python programming. I am using Python 3.3.2 on Windows XP.

My program was working and then all of a sudden I got a UnicodeDecodeError error message.

The exec.py file looks like this:

import re
import os,shutil 
f=open("C:/Documents and Settings/hp/Desktop/my_python_files/AU20-10297-2_yield_69p4_11fails_2_10_14python/a1.txt","a")
for r,d,fi in os.walk("C:/Documents and Settings/hp/Desktop/my_python_files/AU20-10297-2_yield_69p4_11fails_2_10_14python"):
for files in fi:
 if files.startswith("data"):
 g=open(os.path.join(r,files))
 shutil.copyfileobj(g,f)
 g.close()
f.close()
keywords = ['FAIL']
pattern = re.compile('|'.join(keywords))
inFile = open("a1.txt")
outFile =open("failure_info", "w")
keepCurrentSet = False
for line in inFile:
if line.startswith(" Test Results"):
 keepCurrentSet = False
if keepCurrentSet:
 outFile.write(line)
if line.startswith("Station ID "):
 keepCurrentSet = True
#if 'FAIL' in line in inFile:
 # outFile.write(line)
if pattern.search(line):
 outFile.write(line)
inFile.close()
outFile.close()

Now, a1.txt is initially an empty seed text file used for collecting data from the data files. I got the following error messages:

Traceback (most recent call last):
 File "C:\Documents and Settings\hp\Desktop\my_python_files\AU20-10297-2_yield_69p4_11fails_2_10_14python\exec.py", line 8, in <module>
 shutil.copyfileobj(g,f)
 File "C:\Python33\lib\shutil.py", line 68, in copyfileobj
 buf = fsrc.read(length)
 File "C:\Python33\lib\encodings\cp1252.py", line 23, in decode
 return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 754: character maps to <undefined>

Can anyone help me fix the code so it is more robust?

Answer 1

You have opened the files in text mode, which means Python will try and decode the contents to Unicode. You'd normally need to specify the correct codec for the file (or Python will use your platform default), but are just copying files across with shutil.copyfileobj() here, and decoding is not needed.

Open the files in binary mode instead.

f = open(..., 'ab')
for r,d,fi in os.walk(...): 
 for files in fi: 
 if files.startswith("data"):
 g = open(os.path.join(r, files), 'rb')
 shutil.copyfileobj(g,f)

Note the addition of the b to the filemode.

You probably want to use the file objects as context managers so they are closed for you, automatically:

with open(..., 'ab') as outfh:
 for r,d,fi in os.walk(...): 
 for files in fi: 
 if files.startswith("data"):
 with open(os.path.join(r, files), 'rb') as infh:
 shutil.copyfileobj(infh, outfh)

• user3313975: Thanks for solving my problem!