0

I am having trouble with unicode in a script I am writing. I have scoured the internet, including this site and I have tried many things, and I still have no idea what is wrong.

My code is very long, but I will show an excerpt from it:

raw_results = get_raw(args)
write_raw(raw_results)
parsed_results = parse_raw(raw_results)
write_parsed(parsed_results)

Basically, I get raw results, which is in XML, encoded in UTF-8. Writing the RAW data has no problems. But writing the parsed data is. So I am pretty sure the problem is inside the function that parses the data.

I tried everything and I do not understand what the problem is. Even this simple line gives me an error:

def parse_raw(raw_results)
 content = raw_results.replace(u'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>', u'')

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 570: ordinal not in range(128)

Ideally I would love to be able to work with unicode and have no problems, but I also have no issue with replacing/ignoring any unicode and using only regular text. I know I have not provided my full code, but understand that it's a problem since it's work-related. But I hope this is enough to get me some help.

Edit: the top part of my parse_raw function:

from xml.etree.ElementTree import XML, fromstring, tostring
def parse_raw(raw_results) 
 raw_results = raw_results.decode("utf-8")
 content = raw_results.replace('<?xml version="1.0" encoding="UTF-8" standalone="yes"?>', '')
 content = "<root>\n%s\n</root>" % content
 mxml = fromstring(content)

Edit2:: I think it would be a good idea to point out that the code works fine UNLESS there are special characters. When it's 100% English, no problem; whenever any foreign letters or accented letters are involved is when the issues arise.

dda
6,2212 gold badges27 silver badges37 bronze badges
asked Aug 26, 2012 at 8:45
16
  • I suspect your problem is in get_raw() - how are you reading the file? Commented Aug 26, 2012 at 8:48
  • get_raw() is no problem because write_raw() is working fine, the problem is always in write_parsed() which gets the data after it goes through the parse_raw() function Commented Aug 26, 2012 at 8:49
  • This is most probably where you're wrong. The problem occurs when you're mixing encoded and non-encoded strings. As long as you read and write using the same kind of string, you won't notice the problem. As soon as you start using the data as in your parse_raw() function, you will. Commented Aug 26, 2012 at 8:51
  • so how do i edit and work with the data so that it stays in the same encoding? Commented Aug 26, 2012 at 8:51
  • As I said, you need to show us how you're reading the file. All else is guesswork (as you can see below). Commented Aug 26, 2012 at 9:05

2 Answers 2

3

raw_results is probably a str object, not a unicode object.

raw_results.replace(u'...', ...) causes Python to first decode the str raw_results into a unicode. Python2 uses the ascii codec by default. raw_results contains the byte '\xd7' at position 570, which is not decodeable by the ascii codec (i.e., it is not an ascii character).

Here is a demonstration of how this error might occur:

In [27]: '\xd7'.replace(u'a',u'b') 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)

Whereas if raw_results were unicode, there would be no silent decoding with ascii, and therefore no error would occur:

In [28]: u'\xd7'.replace(u'a',u'b')
Out[28]: u'\xd7'

You can fix this problem by decoding raw_results explicitly, provided you know the appropriate codec:

raw_results = raw_results.decode('latin-1')

latin-1 is just a guess. It might be correct if the character at position 570 is a multiplication symbol:

In [26]: print('\xd7'.decode('latin-1'))×ばつ
answered Aug 26, 2012 at 8:55
Sign up to request clarification or add additional context in comments.

7 Comments

its utf-8 . and doing that didnt help. UnicodeEncodeError: 'ascii' codec can't encode character u'\u2605' in position 5620: ordinal not in range(128)
Can you post a snippet of the raw_results str around characters 570 and 5620, such as repr(raw_results[565:575]+raw_results[5615:5625])?
cant show you the data - sorry, its sensitive. but the error is raw_results = raw_results.decode("utf-8")
Hmmm. If you are getting a UnicodeEncodeError there, then raw_results must already be unicode...
Are you sure posting 20 bytes of your super sensitive data for us is not possible?
|
0

Thank you everyone for the input and the nudges. I have subsequently solved my own problem by going over my code for the millionth time with a fine-toothed comb, and I have found the culprit. And I have solved all my problems now.

For anyone with a similar problem, I have the following information that could help you:

  • Use the codecs module for writing your files.
  • Do not try to handle it all along your code, your code should ignore any type of character set throughout methods, and should have specific methods or calls to methods where only you modify the charset. (this helped me find the problem)

My problem was that at a certain point I was trying to turn unicode into unicode. And in another place I was trying to turn normal ASCII into ASCII again. So whenever I solved one issue, another arose and I figured it was the same problem.

Break your issue into sections... and then you might find your problem!

dda
6,2212 gold badges27 silver badges37 bronze badges
answered Aug 26, 2012 at 12:35

Comments

Your Answer

Draft saved
Draft discarded

Sign up or log in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

Post as a guest

Required, but never shown

By clicking "Post Your Answer", you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.