Fix wrong charset detection #57

BaumiCoder commented 2026年06月27日 16:45:58 +02:00

Version of ecformat

0.2.0

Operating system

Linux

Version of your operating system (e.g. Debian 11, Arch Linux or Windows 11)

Manjaro Linux

Installation method

Binary from Codeberg package ecformat_bin

Steps to reproduce the bug

1. Checkout github.com/rust-lang/rust at revision 267c4057bad01b0cf9733081d170c1358b4b3674
2. Run ecformat check (or ecformat fix) on for example the following files:
   1. src/doc/rustdoc/src/write-documentation/the-doc-attribute.md
   2. src/doc/rustdoc/src/unstable-features.md
   3. tests/ui/README.md

Expected result

A check passes without errors and a fix does not change anything.

Actual result

The check reports that the files have latin1 encoding and the fix changes some lines. These lines all have some special characters in there, such as a non-breaking space or 👻.

### Version of ecformat 0.2.0 ### Operating system Linux ### Version of your operating system (e.g. Debian 11, Arch Linux or Windows 11) Manjaro Linux ### Installation method Binary from Codeberg package ecformat_bin ### Steps to reproduce the bug 1. Checkout [github.com/rust-lang/rust at revision `267c4057bad01b0cf9733081d170c1358b4b3674`](https://github.com/rust-lang/rust/tree/267c4057bad01b0cf9733081d170c1358b4b3674) 2. Run `ecformat check` (or `ecformat fix`) on for example the following files: 1. src/doc/rustdoc/src/write-documentation/the-doc-attribute.md 2. src/doc/rustdoc/src/unstable-features.md 3. tests/ui/README.md ### Expected result A `check` passes without errors and a `fix` does not change anything. ### Actual result The `check` reports that the files have latin1 encoding and the `fix` changes some lines. These lines all have some special characters in there, such as a non-breaking space or 👻.

BaumiCoder commented 2026年06月29日 17:05:44 +02:00

Another example is in line 333001 of this file. However, it seems that this patch files contains some binary data.

gitlab.com/kalilinux/build-scripts/kali-arm/-@af5bfb63f2/patches/pinebook-pro/pbp-5.14/0005-staging-add-rtl8723cs-driver.patch

Another example is in line 333001 of this file. However, it seems that this patch files contains some binary data. https://gitlab.com/kalilinux/build-scripts/kali-arm/-/blob/af5bfb63f242348e577492b16aa9816e7dc8a766/patches/pinebook-pro/pbp-5.14/0005-staging-add-rtl8723cs-driver.patch

BaumiCoder commented 2026年07月01日 10:43:55 +02:00

Another example is a ® symbol in this file:

github.com/pgjdbc/pgjdbc@0b4077a529/docs/content/documentation/logging.md

Another example is a ® symbol in this file: https://github.com/pgjdbc/pgjdbc/blob/0b4077a529b2448cc55a6ca87b2be8667243c9ab/docs/content/documentation/logging.md

BaumiCoder commented 2026年07月01日 14:15:58 +02:00

Another example are ö and ? in this file:
github.com/spring-projects/spring-framework@c5d7908a78/spring-test/src/test/java/org/springframework/test/web/servlet/samples/standalone/resulthandlers/PrintingResultHandlerIntegrationTests.java

Another example are ö and ? in this file: https://github.com/spring-projects/spring-framework/blob/c5d7908a7876855c9385dcbb3ae24a3f67c42e01/spring-test/src/test/java/org/springframework/test/web/servlet/samples/standalone/resulthandlers/PrintingResultHandlerIntegrationTests.java

BaumiCoder commented 2026年07月01日 14:18:15 +02:00

A very special one is this file. It contains no special characters but mentions a charset in the text.

github.com/spring-projects/spring-framework@c5d7908a78/spring-webmvc/src/test/resources/org/springframework/web/servlet/view/testviews_fr.properties

A very special one is this file. It contains no special characters but mentions a charset in the text. https://github.com/spring-projects/spring-framework/blob/c5d7908a7876855c9385dcbb3ae24a3f67c42e01/spring-webmvc/src/test/resources/org/springframework/web/servlet/view/testviews_fr.properties