Issue: Inconsistent behavior between encoding detection modules

Summary

reuse lint skips valid UTF-8 files when python-magic detects them as binary. The same files pass when charset_normalizer handles detection. This causes CI/local inconsistency.

Related to #1244, but the inverse case: python-magic incorrectly returns binary, while charset_normalizer handles the files correctly.

Environment

• reuse version: 6.2.0
• Test file: A valid markdown file with SPDX headers containing an invisible 0x01 (SOH) control character

The Problem

The same file produces different results depending on which encoding module is installed:

This causes CI/local inconsistency. Linux CI (with libmagic) fails. macOS (without libmagic) passes.

Root Cause

reuse tries encoding modules in this order (from extract.py):

1. python-magic (requires libmagic system library)
2. file-magic
3. charset_normalizer
4. chardet

On a fresh macOS install without libmagic, reuse falls back to charset_normalizer. On Linux CI with libmagic, it uses python-magic. Install libmagic via Homebrew (brew install libmagic) to reproduce the CI failure locally.

The modules disagree on files with control characters:

# With python-magic (requires libmagic)
>>> import magic
>>> m = magic.Magic(mime_encoding=True)
>>> m.from_file('soh_present.md')
'binary'
# With charset_normalizer (pure Python)
>>> import charset_normalizer
>>> result = charset_normalizer.from_bytes(data).best()
>>> result.encoding
'utf_8'

Reproduction

This repository reproduces the issue:

├── soh_present.md # Test file with SOH byte
├── soh_absent.md # Same file without SOH (control)
├── Dockerfile.python-magic # Test with libmagic
├── Dockerfile.charset-normalizer # Test without libmagic
├── test.sh # Runs both tests
├── LICENSES/
│ ├── CC-BY-SA-4.0.txt # License for test markdown files
│ └── MIT-0.txt # License for test.sh
└── REUSE.toml

Quick Test

./test.sh

Expected Output

TEST 1: python-magic (with libmagic)
Module: python-magic
soh_present.md: None ← detected as binary, SKIPPED
Files with copyright information: 1 / 2
Unfortunately, your project is not compliant... :-(
TEST 2: charset_normalizer (no libmagic)
Module: charset_normalizer
soh_present.md: utf_8 ← detected as UTF-8, PARSED
Files with copyright information: 2 / 2
Congratulations! Your project is compliant... :-)

CI Logs

Public CI builds showing the before/after fix:

• Failed (with SOH): https://builds.sr.ht/~kerrick/job/1646144
• Passed (SOH removed): https://builds.sr.ht/~kerrick/job/1646171

Workaround

Override the encoding module with REUSE_ENCODING_MODULE=chardet:

REUSE_ENCODING_MODULE=chardet reuse lint

For pre-commit:

repos:- repo:https://codeberg.org/fsfe/reuse-toolrev:v6.2.0hooks:- id:reuseentry:env REUSE_ENCODING_MODULE=chardet reuse

This workaround is documented in the v6.1.0 changelog and man page for issue #1244. It also fixes this inverse case.

Suggested Fix

1. Improve lint output. Show why a file failed: skipped as binary, parsed but no SPDX tags, or not covered by REUSE.toml. All three currently appear as "missing copyright and licensing information."
2. Document the behavior. The man page lists REUSE_ENCODING_MODULE but not when to use it. Add that libmagic classifies files with control characters as binary, and REUSE_ENCODING_MODULE=chardet resolves this.

Alternative Fixes

1. Fall back on "binary". When python-magic returns binary for a known text extension (.md, .txt, .rst), try charset_normalizer before skipping.
2. Prefer pure-Python modules. Prioritize charset_normalizer over python-magic for consistent cross-platform results.

This issue includes creative contributions from Claude (Anthropic) via Antigravity (Google). See chat_log_01.md, chat_log_02.md, and chat_log_03.md for details. https://declare-ai.org/1.0.0/creative.html

Discovered while debugging CI failures that couldn't be reproduced locally.