Edit - Stack Overflow

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Rev

Required fields*

Backpropagation of wrongly (double) encoded CSV

I have a CSV file that someone encoded wrongly.

The file is a database of movies with corresponding actors. I downloaded it in order to practise some coding for the so called Bacon number.

It looks like this:

movieId,title,actors
(...)
61,Eye for an Eye (1996),(a ton of other actors)|Dolores VelÌÁzquez|(more actors)
59,The Confessional (1995),(a ton of other actors)|Richard FrÌ©chette|FranÌ¤ois Papineau|Marie Gignac|Normand Daneau|Anne-Marie Cadieux|Suzanne ClÌ©ment|Lynda Beaulieu|Pascal Rollin|Billy Merasty|Paul HÌ©bert|Marthe Turgeon|Adreanne Lepage-Beaulieu|AndrÌ©e-Anne ThÌ©roux-Faille|Rodrigue Proteau|Philippe Paquin|Pierre HÌ©bert|Nathalie D'Anjou|Danielle Fichaud|Jules Philip|Jacques Laroche|Claude-Nicolas Demers|Jean-Philippe CÌ«tÌ©|Tristan Wiseman|Marc-Olivier Tremblay|Jacques Brouillet|Jean-Paul L'Allier|Denis Bernard|RenÌ©e Hudon|Serge Laflamme|Carl Mathieu
(...)

Now as you can see, instead of Umlauts and letters with accents (ÄÖÜ, É, À, Û etc.), the actors have a combination of other special characters instead.

Thanks to very helpful input on this question, we have established that this is indeed a case of Mojibake.

My goal is to programmatically fix the broken characters by decoding and encoding in the correct order.

Answer*

This is what comes closest to the solution (provided by JosefZ): 
> You face a double mojibake case (example in Python): 'FrÌ©chette|FranÌ¤ois'.encode( 'cp1252').decode( 'mac-romanian').encode( 'cp1252').decode( 'utf-8') returns 'Fréchette|François'.

Thanks to very helpful input on this question, we have established that this is indeed a case of [Mojibake][2].

I have made progress with the following configuration (python):

1. Read the current csv with encoding not specified
2. encode('cp1252').decode('mac-roman').encode('cp1252').decode('iso-8859-16')
3. Write the result to new csv with encoding 'iso-8859-16' specified

With this configuration, many characters are fixed now, but some are still missing. I don't know if this means I have to decode and encode again (for a total of three times), or if I simply haven't found the correct configuration for a set of two dec-encs yet.

Here is a list of characters that are still broken after the above re-encode:

```
after re-encoding | just reading | desired outcome | notes
==============================================================================
� | Ì | Á | the lower case á works;
�_ | Ì_ | ü | the upper case Ü works
�_ | Ì_ | ä | I can't confirm whether upper case Ä works
 | æ | I can't confirm whether this exists, but I confirmed one upper case that works; it's possible the file uses 'ae' instead
 | œ | I can't confirm whether any of this exists; it's possible the file uses 'oe' instead
�_ | Ì_ | í | I can't confirm whether upper case Í works
�_ | Ì_ | ó | the upper case Ó works
�_ | Ì_ | þ | the upper case Þ doesn't exist, I think
```

Conclusion:

As you can see, both the reading of the file 'as is', as well as reading it after re-encoding, renders all missing characters as **the same** broken character, so it seems impossible to restore the original information for those (the information got lost in the process of Mojibake). I'm afraid that those lines would have to be fixed manually.


 [1]: https://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon
 [2]: https://en.wikipedia.org/wiki/Mojibake

Draft saved

Draft discarded

Edit Summary*

Cancel

Add a comment |

How to Edit

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

How to Format

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="https://example.com">example</a>

formatting help »
answering help »

How to Tag

A tag is a keyword or label that categorizes your question with other, similar questions. Choose one or more (up to 5) tags that will help answerers to find and interpret your question.

complete the sentence: my question is about...
use tags that describe things or concepts that are essential, not incidental to your question
favor using existing popular tags
read the descriptions that appear below the tag

If your question is primarily about a topic for which you can't find a tag:

combine multiple words into single-words with hyphens (e.g. python-3.x), up to a maximum of 35 characters
creating new tags is a privilege; if you can't yet create a tag you need, then post this question without it, then ask the community to create it for you

popular tags »

lang-py

CollectivesTM on Stack Overflow

Backpropagation of wrongly (double) encoded CSV

Answer*