Capturing the bad codes that raise UnicodeError exceptions during decoding

Thu Aug 4 17:03:06 EDT 2016

Wow!!! A huge thank you to all who replied to this thread!
Chris: You gave me some ideas I will apply in the future.
MRAB: Thanks for exposing me to the extended attributes of the UnicodeError object (e.start, e.end, e.object).
Mike: Cool example! I like how _cleanlines() recursively calls itself to keep cleaning up a line after an error is handled. Your code solved the mystery of how to recover from a UnicodeError and keep decoding.
Random832: Your suggestion to write a custom codecs handler was great. Sample below for future readers reviewing this thread.
# simple codecs custom error handler
import codecs
def custom_unicode_error_handler(e):
 bad_bytes = e.object[e.start:e.end]
 print( 'Bad bytes: ' + bad_bytes.hex())
 return ('<?>', e.end)
codecs.register_error('custom_unicode_error_handler',
custom_unicode_error_handler)
Malcolm