-
Notifications
You must be signed in to change notification settings - Fork 889
Document iconv implementation quirks #5646
Description
tl;dr document some if the problems I ran into in php/php-src#22413
PHP's iconv() function mostly defers to the system's iconv_open() and iconv() routines. POSIX 2024 covers these:
- https://pubs.opengroup.org/onlinepubs/9799919799/functions/iconv_open.html
- https://pubs.opengroup.org/onlinepubs/9799919799/functions/iconv.html
As you can see, there are three magic suffixes that can be appended to the "to" charset:
- //IGNORE
- //TRANSLIT
- //NON_IDENTICAL_DISCARD
In practice, support for these varies widely, and when they are supported they may not conform to POSIX. For example, musl does not support //IGNORE, and (man 3 iconv) BSD/Solaris replace unknown translations with ? rather than raise an error that would require //TRANSLIT... so they do not have //TRANSLIT.
The details are sordid, and we should not try to capture them in the PHP documentation. Instead, we should simply state that support for any iconv() feature can vary widely depending on your implementation. Right now the PHP iconv docs mention //TRANSLIT and //IGNORE, but say only that //TRANSLIT is unreliable.
- We should also mention that
//IGNOREis implementation-dependent. PHP has a workaround for iconv implementations (glibc) that return an error upon ignoring an invalid sequence, which helps standardize PHP's behavior, but//IGNOREmay not be supported at all (musl), and there is no workaround for that. //NON_IDENTICAL_DISCARDneeds a mention.- The "to" and "from" charsets are also implementation-defined. Musl for example only supports a few, and will fall back to UTF-8 if you pass in something it doesn't understand.