Questions tagged [character-encoding]
Questions that deal with various representations of characters & character sets, such as: ASCII, UTF-8, EBCDIC, among others. Often encountered when moving files between operating systems that encode new lines with carriage returns and/or newline characters.
423 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
3
votes
2
answers
168
views
Embedded special characters skewing sed output
The Issue
I've been parsing a file with sed trying to tweeze out the desired data. This has worked fine for most lines in the file but there appears to be some embedded special characters that are ...
1
vote
1
answer
89
views
Invalid characters in ssh sessions
I'm opening an SSH session from Fedora to Raspberry Pi OS. Accented and special characters are replaced with question marks. Preferably I would like to learn to solve this without changing the server'...
2
votes
1
answer
89
views
Tmux pane with long-running session using wrong character set?
Today I connected to a long-running process in tmux over ssh for work, to find that the pane the process was running in seems to have started using the wrong character encoding for its output, leading ...
6
votes
1
answer
358
views
Revert filenames after they were garbled by using different encoding
I have a file
СМП бвагTMвга†
The first three letters are proper Cyrllic and the remaining part is mojibake.
"Mojibake is the garbled or gibberish text that is the result of text being decoded ...
0
votes
1
answer
76
views
Output of echo uses different encoding than the one specified according to LANG and LC_CTYPE
It is my understanding that the LANG and LC_CTYPE environment variables define the encoding used by shell commands when writing to stdout. However, after executing
LANG=de_DE.iso88591 LC_CTYPE=de_DE....
0
votes
2
answers
209
views
To have or not Byte Order Mark (BOM) in UTF-8 text files?(Linux)
Is it advisable to have or not Byte Order Mark (BOM) in UTF-8 text files on Linux?
Is it correct to say byte order (even for multi-byte characters) is already strictly defined/fixed in UTF-8 standard?
...
0
votes
0
answers
92
views
Advanced CLI tool/code to determine text encoding (besides enca)
Looking for advanced CLI tool/code to determine text Codepage/Language (besides enca).
Goal: Automate as much as possible conversion of hundreds/thousands of 8-bit text files (including non-ASCII ...
0
votes
0
answers
33
views
file -i provide two different charsets for the same file on the same FS
I'm a bit confuse about a behavior of the file -i command. I searched a while and give up since I didn't have a sufficient knowledge regarding encoding as well as linux file command (to stay concise ...
-2
votes
1
answer
82
views
Convert subtitles so they are coded correctly (Polish and `"` even gets wrongly coded)
Wrong encoding:
1
00:01:27,879 --> 00:01:31,216
No i dupa.
Koniec z darmowym wi-fi.
2
00:01:33,009 --> 00:01:34,972
- Ki-jung!
- No?
3
00:01:35,219 --> 00:01:39,183
Kobieta z góry
...
2
votes
1
answer
170
views
removing hidden control characters in filenames
I have a huge number of files spread across a large directory structure that have hidden control characters in their names. ls lists them as, e.g.:
'614.7-4-F1-00-090-007-RozvadØ'$'302円237円'' RP1-...
1
vote
1
answer
160
views
How can I set the character to Latn-1 or MCS when using serial-getty?
I'd like to use my old VT420 terminal as system console. Adding RS232 ports and setting up serial-getty are not a problem, but: For years, almost all Linux distros have been using UTF-8 as the ...
11
votes
3
answers
2k
views
UTF-8 characters in POSIX shell script *comments* - anything against it?
I would like to include a couple of non-ASCII characters in my POSIX shell script comments. Note this is in no way a duplicate of e.g. "Which character encodings are supported by posix?" as ...
0
votes
1
answer
156
views
regex: how come the trademark symbol matches to a-z?
Sorry if this is a repeat or basic question but it is hard to search for a TM. I'm writing a script to remove weird characters from file names.
How come the trade mark symbol TM matches [^a-z] ???
$ ...
0
votes
1
answer
319
views
nmtui is not rendering correctly
When using nmtui, â"Œ and â"‚ are added in places where they definitely should not be (see attached screenshot):
How can I solve this?
4
votes
2
answers
1k
views
How can I convert full-width characters to half-width characters (and vice versa)?
Here is my simple problem, how can I convert half-width to full-width from the command line. I thought this would be built-in my iconv command line, but I did not find anything here:
$ iconv -l | ...