Portability	non-portable
Stability	internal
Maintainer	libraries@haskell.org
Safe Haskell	Trustworthy

GHC.IO.Encoding

Description

Text codecs for I/O

Synopsis

data BufferCodec from to state = BufferCodec {
- encode :: Buffer from -> Buffer to -> IO (CodingProgress, Buffer from, Buffer to)
- recover :: Buffer from -> Buffer to -> IO (Buffer from, Buffer to)
- close :: IO ()
- getState :: IO state
- setState :: state -> IO ()
}
data TextEncoding = forall dstate estate . TextEncoding {
- textEncodingName :: String
- mkTextDecoder :: IO (TextDecoder dstate)
- mkTextEncoder :: IO (TextEncoder estate)
}
type TextEncoder state = BufferCodec CharBufElem Word8 state
type TextDecoder state = BufferCodec Word8 CharBufElem state
data CodingProgress
- = InputUnderflow
- | OutputUnderflow
- | InvalidSequence
latin1 :: TextEncoding
latin1_encode :: CharBuffer -> Buffer Word8 -> IO (CharBuffer, Buffer Word8)
latin1_decode :: Buffer Word8 -> CharBuffer -> IO (Buffer Word8, CharBuffer)
utf8 :: TextEncoding
utf8_bom :: TextEncoding
utf16 :: TextEncoding
utf16le :: TextEncoding
utf16be :: TextEncoding
utf32 :: TextEncoding
utf32le :: TextEncoding
utf32be :: TextEncoding
initLocaleEncoding :: TextEncoding
getLocaleEncoding :: IO TextEncoding
getFileSystemEncoding :: IO TextEncoding
getForeignEncoding :: IO TextEncoding
setLocaleEncoding :: TextEncoding -> IO ()
setFileSystemEncoding :: TextEncoding -> IO ()
setForeignEncoding :: TextEncoding -> IO ()
char8 :: TextEncoding
mkTextEncoding :: String -> IO TextEncoding

Documentation

data BufferCodec from to state Source

Constructors

BufferCodec

Fields

encode :: Buffer from -> Buffer to -> IO (CodingProgress, Buffer from, Buffer to)

The encode function translates elements of the buffer from to the buffer to. It should translate as many elements as possible given the sizes of the buffers, including translating zero elements if there is either not enough room in to, or from does not contain a complete multibyte sequence.

The fact that as many elements as possible are translated is used by the IO library in order to report translation errors at the point they actually occur, rather than when the buffer is translated.

To allow us to use iconv as a BufferCode efficiently, character buffers are defined to contain lone surrogates instead of those private use characters that are used for roundtripping. Thus, Chars poked and peeked from a character buffer must undergo surrogatifyRoundtripCharacter and desurrogatifyRoundtripCharacter respectively.

For more information on this, see Note [Roundtripping] in GHC.IO.Encoding.Failure.

recover :: Buffer from -> Buffer to -> IO (Buffer from, Buffer to)

The recover function is used to continue decoding in the presence of invalid or unrepresentable sequences. This includes both those detected by encode returning InvalidSequence and those that occur because the input byte sequence appears to be truncated.

Progress will usually be made by skipping the first element of the from buffer. This function should only be called if you are certain that you wish to do this skipping and if the to buffer has at least one element of free space. Because this function deals with decoding failure, it assumes that the from buffer has at least one element.

recover may raise an exception rather than skipping anything.

Currently, some implementations of recover may mutate the input buffer. In particular, this feature is used to implement transliteration.

close :: IO ()

Resources associated with the encoding may now be released. The encode function may not be called again after calling close.

getState :: IO state

Return the current state of the codec.

Many codecs are not stateful, and in these case the state can be represented as '()'. Other codecs maintain a state. For example, UTF-16 recognises a BOM (byte-order-mark) character at the beginning of the input, and remembers thereafter whether to use big-endian or little-endian mode. In this case, the state of the codec would include two pieces of information: whether we are at the beginning of the stream (the BOM only occurs at the beginning), and if not, whether to use the big or little-endian encoding.

setState :: state -> IO ()

data TextEncoding Source

A TextEncoding is a specification of a conversion scheme between sequences of bytes and sequences of Unicode characters.

For example, UTF-8 is an encoding of Unicode characters into a sequence of bytes. The TextEncoding for UTF-8 is utf8.

Constructors

forall dstate estate . TextEncoding

Fields

textEncodingName :: String: a string that can be passed to mkTextEncoding to create an equivalent TextEncoding .
mkTextDecoder :: IO (TextDecoder dstate): Creates a means of decoding bytes into characters: the result must not be shared between several byte sequences or simultaneously across threads
mkTextEncoder :: IO (TextEncoder estate): Creates a means of encode characters into bytes: the result must not be shared between several character sequences or simultaneously across threads

Instances

Show TextEncoding

type TextEncoder state = BufferCodec CharBufElem Word8 stateSource

type TextDecoder state = BufferCodec Word8 CharBufElem stateSource

data CodingProgress Source

Constructors

InputUnderflow

Stopped because the input contains insufficient available elements, or all of the input sequence has been sucessfully translated.

OutputUnderflow

Stopped because the output contains insufficient free elements

InvalidSequence

Stopped because there are sufficient free elements in the output to output at least one encoded ASCII character, but the input contains an invalid or unrepresentable sequence

Instances

Eq CodingProgress

Show CodingProgress

latin1 :: TextEncoding Source

The Latin1 (ISO8859-1) encoding. This encoding maps bytes directly to the first 256 Unicode code points, and is thus not a complete Unicode encoding. An attempt to write a character greater than '255円' to a Handle using the latin1 encoding will result in an error.

latin1_encode :: CharBuffer -> Buffer Word8 -> IO (CharBuffer, Buffer Word8)Source

latin1_decode :: Buffer Word8 -> CharBuffer -> IO (Buffer Word8, CharBuffer)Source

utf8 :: TextEncoding Source

The UTF-8 Unicode encoding

utf8_bom :: TextEncoding Source

The UTF-8 Unicode encoding, with a byte-order-mark (BOM; the byte sequence 0xEF 0xBB 0xBF). This encoding behaves like utf8 , except that on input, the BOM sequence is ignored at the beginning of the stream, and on output, the BOM sequence is prepended.

The byte-order-mark is strictly unnecessary in UTF-8, but is sometimes used to identify the encoding of a file.

utf16 :: TextEncoding Source

The UTF-16 Unicode encoding (a byte-order-mark should be used to indicate endianness).

utf16le :: TextEncoding Source

The UTF-16 Unicode encoding (litte-endian)

utf16be :: TextEncoding Source

The UTF-16 Unicode encoding (big-endian)

utf32 :: TextEncoding Source

The UTF-32 Unicode encoding (a byte-order-mark should be used to indicate endianness).

utf32le :: TextEncoding Source

The UTF-32 Unicode encoding (litte-endian)

utf32be :: TextEncoding Source

The UTF-32 Unicode encoding (big-endian)

initLocaleEncoding :: TextEncoding Source

getLocaleEncoding :: IO TextEncoding Source

The Unicode encoding of the current locale

getFileSystemEncoding :: IO TextEncoding Source

The Unicode encoding of the current locale, but allowing arbitrary undecodable bytes to be round-tripped through it.

This TextEncoding is used to decode and encode command line arguments and environment variables on non-Windows platforms.

On Windows, this encoding *should not* be used if possible because the use of code pages is deprecated: Strings should be retrieved via the wide W-family of UTF-16 APIs instead

getForeignEncoding :: IO TextEncoding Source

The Unicode encoding of the current locale, but where undecodable bytes are replaced with their closest visual match. Used for the CString marshalling functions in Foreign.C.String

setLocaleEncoding :: TextEncoding -> IO ()Source

setFileSystemEncoding :: TextEncoding -> IO ()Source

setForeignEncoding :: TextEncoding -> IO ()Source

char8 :: TextEncoding Source

An encoding in which Unicode code points are translated to bytes by taking the code point modulo 256. When decoding, bytes are translated directly into the equivalent code point.

This encoding never fails in either direction. However, encoding discards information, so encode followed by decode is not the identity.

mkTextEncoding :: String -> IO TextEncoding Source

Look up the named Unicode encoding. May fail with

isDoesNotExistError if the encoding is unknown

The set of known encodings is system-dependent, but includes at least:

```
UTF-8
```
UTF-16, UTF-16BE, UTF-16LE
UTF-32, UTF-32BE, UTF-32LE

On systems using GNU iconv (e.g. Linux), there is additional notation for specifying how illegal characters are handled:

a suffix of //IGNORE, e.g. UTF-8//IGNORE, will cause all illegal sequences on input to be ignored, and on output will drop all code points that have no representation in the target encoding.
a suffix of //TRANSLIT will choose a replacement character for illegal sequences or code points.

On Windows, you can access supported code pages with the prefix CP; for example, "CP1250".