Edit - Stack Overflow

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Rev

Required fields*

Utf-8 on windows python

I have html file to read parse etc, it's encode on unicode (I saw it with the notepad) but when I tried

infile = open("path", "r") 
infile.read()

it fails and I had the famous error :

UnicodeEncodeError: 'charmap' codec can't encode characters in position xx: character maps to undefined

So for test I tried to copy paste the contain of the file in a new one and save it in utf-8 and then tried to open it with codecs like this :

inFile = codecs.open("path", "r", encoding="utf-8")
outputStream = inFile.read()

But I get this error message :

UnicodeEncodeError : 'charmap' codec can't encode character u'\ufeff' in position 0: charcater maps to undefined

I really don't understand because I was created this file in utf8.

Answer*

`UnicodeEncodeError` suggests that the code fails while *encoding* Unicode text to bytes i.e., your actual code tries to print to Windows console. See [Python, Unicode, and the Windows console](https://stackoverflow.com/a/32176732/4279).

---

The link above fixes `UnicodeEncodeError`. The next issue is to find out what character encoding is used by the text in your `"path"` file. If `notepad.exe` shows the text correctly then it means that it is either encoded using `locale.getprefferedencoding(False)` (something like `cp1252` on Windows) or the file has [BOM](https://en.wikipedia.org/wiki/Byte_order_mark).

If you are sure that the encoding is utf-8 then pass it to `open()` directly. Don't use `codecs.open()`:

 with open('path', encoding='utf-8') as file:
 html = file.read()

Sometimes, the input may contain text encoded using multiple (inconsistent) encodings e.g., smart quotes may be encoded using `cp1252` while the rest of html is utf-8 -- you could fix it using [`bs4.UnicodeDammit`](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#inconsistent-encodings). See also https://stackoverflow.com/q/14592762/4279

Draft saved

Draft discarded

Edit Summary*

Cancel

If Notepad says "Unicode" (as the OP said) it means UTF-16. The other encodings are usually called "ANSI" (cp1252 and friends) and "UTF-8" (which is UTF-8 with BOM).

roeland
– roeland

2015年09月24日 22:05:25 +00:00
Commented Sep 24, 2015 at 22:05
1

@roeland: yes. "it's encode on unicode (I saw it with the notepad)" from the question can be interpreted that way. The issue with that theory is that codecs.open("path", encoding='utf-8').read() returns u'\ufeff' i.e., utf-8-sig is more likely. 'utf-8' encoding fails for both BOM_UTF16_BE and BOM_UTF16_LE.

jfs
– jfs

2015年09月24日 22:19:49 +00:00
Commented Sep 24, 2015 at 22:19
Yeah, the question is a bit confusing as it involves two files, the original file in "Unicode", and the file he re-saved as "UTF-8".

roeland
– roeland

2015年09月24日 22:23:58 +00:00
Commented Sep 24, 2015 at 22:23
@roeland: anyway the issue is UnicodeEncodeError i.e., when OP tries to print Unicode text to Windows console.

jfs
– jfs

2015年09月24日 22:26:09 +00:00
Commented Sep 24, 2015 at 22:26
Aha, I see. That was subtle

roeland
– roeland

2015年09月24日 22:32:24 +00:00
Commented Sep 24, 2015 at 22:32

Add a comment |

How to Edit

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

How to Format

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="https://example.com">example</a>

formatting help »
answering help »

How to Tag

A tag is a keyword or label that categorizes your question with other, similar questions. Choose one or more (up to 5) tags that will help answerers to find and interpret your question.

complete the sentence: my question is about...
use tags that describe things or concepts that are essential, not incidental to your question
favor using existing popular tags
read the descriptions that appear below the tag

If your question is primarily about a topic for which you can't find a tag:

combine multiple words into single-words with hyphens (e.g. python-3.x), up to a maximum of 35 characters
creating new tags is a privilege; if you can't yet create a tag you need, then post this question without it, then ask the community to create it for you

popular tags »

lang-py

CollectivesTM on Stack Overflow

Utf-8 on windows python

Answer*