Network Working Group A. Phillips, Ed. Internet-Draft Quest Software Expires: March 27, 2006 M. Davis, Ed. IBM September 23, 2005 Matching Tags for the Identification of Languages draft-ietf-ltru-matching-04 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on March 27, 2006. Copyright Notice Copyright (C) The Internet Society (2005). Abstract This document describes different mechanisms for comparing, matching, and evaluating language tags. Possible algorithms for language negotiation and content selection are described. Phillips & Davis Expires March 27, 2006 [Page 1]
Internet-Draft ltru-matching September 2005 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. The Language Range . . . . . . . . . . . . . . . . . . . . . . 4 2.1 Basic Language Range . . . . . . . . . . . . . . . . . . . 4 2.1.1 Matching . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.2 Lookup . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Extended Language Range . . . . . . . . . . . . . . . . . 7 2.2.1 Extended Range Matching . . . . . . . . . . . . . . . 8 2.2.2 Extended Range Lookup . . . . . . . . . . . . . . . . 9 2.2.3 Distance Metric Scheme . . . . . . . . . . . . . . . . 10 2.3 Meaning of Language Tags and Ranges . . . . . . . . . . . 12 2.4 Choosing Between Alternate Matching Schemes . . . . . . . 13 2.5 Considerations for Private Use Subtags . . . . . . . . . . 14 2.6 Length Considerations in Matching . . . . . . . . . . . . 14 3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 4. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 5. Security Considerations . . . . . . . . . . . . . . . . . . . 19 6. Character Set Considerations . . . . . . . . . . . . . . . . . 20 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 7.1 Normative References . . . . . . . . . . . . . . . . . . . 21 7.2 Informative References . . . . . . . . . . . . . . . . . . 22 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 23 A. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 24 Intellectual Property and Copyright Statements . . . . . . . . 25 Phillips & Davis Expires March 27, 2006 [Page 2]
Internet-Draft ltru-matching September 2005 1. Introduction Human beings on our planet have, past and present, used a number of languages. There are many reasons why one would want to identify the language used when presenting or requesting information. Information about a user's language preferences commonly needs to be identified so that appropriate processing can be applied. For example, the user's language preferences in a browser can be used to select web pages appropriately. A choice of language preference can also be used to select among tools (such as dictionaries) to assist in the processing or understanding of content in different languages. Given a set of language identifiers, such as those defined in [draft- registry], various mechanisms can be envisioned for performing language negotiation and tag matching. The suitability of a particular mechanism to a particular application depends on the needs of that application. This document defines several mechanisms for matching and filtering natural language content identified using Language Tags [draft- registry]. It also defines the syntax (called a "language range") associated with each of these mechanisms for specifying user language preferences. The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Phillips & Davis Expires March 27, 2006 [Page 3]
Internet-Draft ltru-matching September 2005 2. The Language Range Language Tags [draft-registry] are used to identify the language of some information item or content. Applications that use language tags are often faced with the problem of identifying sets of content that share certain language attributes. For example, HTTP 1.1 [RFC2616] describes language ranges in its discussion of the Accept- Language header (Section 14.4), which is used for selecting content from servers based on the language of that content. When selecting content according to its language, it is useful to have a mechanism for identifying sets of language tags that share specific attributes. This allows users to select or filter content based on specific requirements. Such an identifier is called a "Language Range". 2.1 Basic Language Range A basic language range identifies the set of content whose language tags begin with the same sequence of subtags. Basic language ranges are described in [RFC3066] and HTTP 1.1 [RFC2616] (where they are referred to as simply a "language range"). A basic language range is identified by its 'language-range' tag, by adapting the ABNF[RFC2234bis] from HTTP/1.1 [RFC2616] : language-range = language-tag / "*" language-tag = 1*8[alphanum] *["-" 1*8alphanum] alphanum = ALPHA / DIGIT That is, a language-range has the same syntax as a language-tag or is the single character "*". Basic Language Ranges imply that there is a semantic relationship between language tags that share the same prefix. While this is often the case, it is not always true. In particular, the set of language tags that match a specific language-range might not all be mutually intelligible. Matching a language-range (prefix) to various language tags does not mean that it is always true that if a user understands a language identified by a certain tag, then this user will also understand all of the languages for which this tag is a prefix. The use of prefixes (and thus basic language ranges) simply allows the use of a prefix if this is the case. Language tags and thus language ranges are to be treated as case insensitive: there exist conventions for the capitalization of some of the subtags, but these MUST NOT be taken to carry meaning. Matching of language tags to language ranges MUST be done in a case insensitive manner. Phillips & Davis Expires March 27, 2006 [Page 4]
Internet-Draft ltru-matching September 2005 When working with tags and ranges users SHOULD also note the following: 1. Private-use and Extension subtags are normally orthogonal to language tag fallback. Implementations SHOULD ignore unrecognized private-use and extension subtags when performing language tag fallback. Since these subtags are always at the end of the sequence of subtags, they don't normally interfere with the use of prefixes for matching in the schemes described below. 2. Implementations that choose not to interpret one or more private- use or extension subtags SHOULD NOT remove or modify these extensions in content that they are processing. When a language tag instance is to be used in a specific, known protocol, and is not being passed through to other protocols, language tags MAY be filtered to remove subtags and extensions that are not supported by that protocol. Such filtering SHOULD be avoided, if possible, since it removes information that might be relevant if services on the other end of the protocol would make use of that information. 3. Some applications of language tags might want or need to consider extensions and private-use subtags when matching tags. If extensions and private-use subtags are included in a matching or filtering process that utilizes the one of the schemes described in this document, then the implementation SHOULD canonicalize the language tags and/or ranges before performing the matching. Note that language tag processors that claim to be "well-formed" processors as defined in [draft-registry] generally fall into this category. There are two matching schemes that are commonly associated with basic language ranges: matching and lookup. Note that neither matching nor lookup using basic language ranges attempt to process the semantics of the tags or ranges in any way. The language tag and language range are compared in a case insensitive manner using basic string processing. The choice of subtags in both the language tag and language range may affect the results produced as a result. 2.1.1 Matching Language tag matching is used to select all content that matches a given prefix. In matching, the language range represents the least specific tag which is an acceptable match and every piece of content that matches is returned. Phillips & Davis Expires March 27, 2006 [Page 5]
Internet-Draft ltru-matching September 2005 For example, if an application is applying a style to all content in a document in a particular language, it might use language tag matching to select the content to which the style is applied. A language-range matches a language-tag if it exactly equals the tag, or if it exactly equals a prefix of the tag such that the first character following the prefix is "-". (That is, the language-range "de-de" matches the language tag "de-DE-1996", but not the language tag "de-Deva".) The special range "*" matches any tag. A protocol which uses language ranges MAY specify additional rules about the semantics of "*"; for instance, HTTP/1.1 specifies that the range "*" matches only languages not matched by any other range within an "Accept-Language" header. 2.1.2 Lookup Content lookup is used to select the single information item that best matches the language range for a given request. In lookup, the language range represents the most specific tag which is an acceptable match and only the closest matching item is returned. For example, if an application inserts some dynamic content into a document, returning an empty string if there is no exact match is not an option. Instead, the application "falls back" until it finds a suitable piece of content to insert. When performing lookup, the language range is progressively truncated from the end until a matching piece of content is located. For example, starting with the range "zh-Hant-CN-x-wadegile", the lookup would progressively search for content as shown below: Range to match: zh-Hant-CN-x-wadegile 1. zh-Hant-CN-x-wadegile 2. zh-Hant-CN 3. zh-Hant 4. zh 5. (default content or the empty tag) Figure 2: Default Fallback Pattern Example This scheme allows some flexibility in finding content. It also typically provides better results when data is not available at a specific level of tag granularity or is sparsely populated (than if the default language for the system or content were used). Phillips & Davis Expires March 27, 2006 [Page 6]
Internet-Draft ltru-matching September 2005 2.2 Extended Language Range Prefix matching using a Basic Language Range, as described above, is not always the most appropriate way to access the information contained in language tags when selecting or filtering content. Some applications might wish to define a more granular matching scheme and such a matching scheme requires the ability to specify the various attributes of a language tag in the language range. An extended language range can be represented by the following ABNF: extended-language-range = range ; a range / privateuse ; private use tag / grandfathered ; grandfathered registrations range = (language ["-" script] ["-" region] *("-" variant) *("-" extension) ["-" privateuse]) language = (2*3ALPHA [ extlang ]) ; shortest ISO 639 code / 4ALPHA ; reserved for future use / 5*8ALPHA ; registered language subtag / "*" ; ... or wildcard extlang = *2("-" 3ALPHA) ("-" ( 3ALPHA / "*")) ; reserved for future use ; wildcard can only appear ; at the end script = 4ALPHA ; ISO 15924 code / "*" ; or wildcard region = 2ALPHA ; ISO 3166 code / 3DIGIT ; UN M.49 code / "*" ; ... or wildcard variant = 5*8alphanum ; registered variants / (DIGIT 3alphanum) ; / "*" ; ... or wildcard extension = singleton *("-" (2*8alphanum)) [ "-*" ] ; extension subtags ; wildcard can only appear ; at the end singleton = %x41-57 / %x59-5A / %x61-77 / %x79-7A / DIGIT ; "a"-"w" / "y"-"z" / "A"-"W" / "Y"-"Z" / "0"-"9" Phillips & Davis Expires March 27, 2006 [Page 7]
Internet-Draft ltru-matching September 2005 ; Single letters: x/X is reserved for private use privateuse = ("x"/"X") 1*("-" (1*8alphanum)) grandfathered = 1*3ALPHA 1*2("-" (2*8alphanum)) ; grandfathered registration ; Note: i is the only singleton ; that starts a grandfathered tag alphanum = (ALPHA / DIGIT) ; letters and numbers In an extended language range, the identifier takes the form of a series of subtags which must consist of well-formed subtags or the special subtag "*". For example, the language range "en-*-US" specifies a primary language of 'en', followed by any script subtag, followed by the region subtag 'US'. A field not present in the middle of an extended language range MAY be treated as if the field contained a "*". For example, the range "en-US" MAY be considered to be equivalent to the range "en-*-US". This also means that multiple wildcards can be collapsed (so that "en-*-*-US" is equivalent to "en-*-US"). There are several matching algorithms or schemes which can be applied when matching extended language ranges to language tags. 2.2.1 Extended Range Matching In extended range matching, the subtags in a language tag are compared to the corresponding subtags in the extended language range. A subtag is considered to match if it exactly matches the corresponding subtag in the range or the range contains a subtag with the value "*" (which matches all subtags, including the empty subtag). Extended Range Matching is an extension of basic matching (Section 2.1.1): the language range represents the least specific tag which is an acceptable match. By default all extensions and their subtags are ignored for extended language range matching. Private use subtags MAY be specified in the language range and MUST NOT be ignored when matching. Subtags not specified, including those at the end of the language range, are assigned the value "*". This makes each range into a prefix much like that used in basic language range matching. For example, the extended language range "zh-*-CN" matches all of the following tags because the unspecified variant field is expanded to Phillips & Davis Expires March 27, 2006 [Page 8]
Internet-Draft ltru-matching September 2005 "*": zh-Hant-CN zh-CN zh-Hans-CN zh-CN-x-wadegile zh-Latn-CN-boont zh-cmn-Hans-CN-x-wadegile 2.2.2 Extended Range Lookup In extended range lookup, the subtags in a language tag are compared to the corresponding subtags in the extended language range. The subtag is considered to match if it exactly matches the corresponding subtag in the range or the range contains a subtag with the value "*" (which matches all subtags, including the empty subtag). Extended language range lookup is an extension of basic lookup (Section 2.1.2): the language range represents the most specific tag which will form an acceptable match. Subtags not specified are assigned the value "*" prior to performing tag matching. Unlike in extended range matching, however, fields at the end of the range MUST NOT be expanded in this manner. For example, "en-US" MUST NOT be considered to be the same as the range "en-US-*". This allows ranges to be specific. The "*" wildcard MUST be used at the end of the range to indicate that all tags with the range as a prefix are allowable matches. That is, the range "zh-*" matches the tags "zh-Hant" and "zh-Hant-CN", while the range "zh" matches neither of those tags. The wildcard "*" at the end of a range SHOULD be considered to match any private use subtag sequences (making extended language range lookup function exactly like extended range matching Section 2.2.1). By default all extensions and their subtags SHOULD be ignored for extended language range lookup. Private use subtags MAY be specified in the language range and MUST NOT be ignored when performing lookup. The wildcard "*" at the end of a range SHOULD be considered to match any private use subtag sequences in addition to variants. For example, the range "*-US" matches all of the following tags: Phillips & Davis Expires March 27, 2006 [Page 9]
Internet-Draft ltru-matching September 2005 en-US en-Latn-US en-US-r-extends (extensions are ignored) fr-US For example, the range "en-*-US" matches _none_ of the following tags: fr-US en (missing region US) en-Latn (missing region US) en-Latn-US-scouse (variant field is present) For example, the range "en-*" matches all of the following tags: en-Latn en-Latn-US en-Latn-US-scouse en-US en-scouse Note that the ability to be specific in extended range lookup can make this matching scheme a more appropriate replacement for basic matching than the extended range matching scheme. 2.2.3 Distance Metric Scheme Both Basic and Extended Language Ranges produce simple boolean matches. Some applications may benefit by providing an array of results with different levels of matching, for example, sorting results based on the overall "quality" of the match. This type of matching is sometimes called a "distance metric". A distance metric assigns a pair of language tags a numeric value representing the 'distance' between the two. A distance of zero means that they are identical, a small distance indicates that they are very similar, and a large distance indicated that they are very different. Using a distance metric, implementations can, for Phillips & Davis Expires March 27, 2006 [Page 10]
Internet-Draft ltru-matching September 2005 example, allow users to select a threshold distance for a match to be successful or a filter to be applied. The first step in the process is to normalize the extended language range and the language tags to be matched to it by canonicalizing them, mapping grandfathered and obsolete tags into modern equivalents. The language range and the language tags are then transformed into quintuples of elements of the form (language, script, country, variant, extension). Any extended language subtags are considered part of the language element; private use subtag sequences are considered part of the language element if in the initial position in the tag and part of the variant element if not. Language subtags 'und', 'mul', and the script subtag 'Zyyy' are converted to "*". Missing components in the language-tag are set to "*"; thus a "*" pattern becomes the quintuple ("*", "*", "*", "*", "*"). Missing components in the extended language-range are handled similarly to extended range lookup: missing internal subtags are expanded to "*". Missing end subtags are expanded as the empty string. Thus a pattern "en-US" becomes the quintuple ("en","*","US","",""). Here are some examples of language-tags and their quintuples: en-US ("en","*","US","*","*") sr-Latn ("sr,"Latn","*","*","*") zh-cmn-Hant ("zh-cmn","Hant","*","*","*") x-foo ("x-foo","*","*","*","*") en-x-foo ("en","*","*","x-foo","*") i-default ("i-default","*","*","*","*") sl-Latn-IT-roazj ("sl","Latn","IT","rozaj","*") zh-r-wadegile ("zh","*","*","*","r-wadegile") // hypothetical Each language-range/language-tag pair being matched or filtered is assigned a distance value, whereby small values indicate better matches and large values indicate worse ones. The distance between the pair is the sum of the distances for each of the corresponding elements of the quintuple. If the elements are identical or one is '*', then the distance value between them is zero. Otherwise, it is given by the following table: Phillips & Davis Expires March 27, 2006 [Page 11]
Internet-Draft ltru-matching September 2005 256 language mismatch 128 script mismatch 32 region mismatch 4 variant mismatch 1 extension mismatch A value of 0 is a perfect match; 421 is no match at all. Different threshold values might be appropriate for different applications and implementations will probably allow users to choose the most appropriate selection value, ranking the selections based on score. Examples of various tag's distances from the range "en-US": "fr" 256 (language mismatch, region match) "en-GB" 384 (language, region mismatch) "en-Latn-US" 0 (all fields match) "en-Brai" 32 (region mismatch) "en-US-x-foo" 4 (variant mismatch: range is the empty string) "en-US-r-wadegile" 1 (extension mismatch: range is the empty string) Implementations may want to use more sophisticated weights that depend on the values of the corresponding elements. For example, depending on the domain, an implemenation might give a small distance to the difference between the language subtag 'no' and the closely related language subtags 'nb' or 'nn'; or between the script subtags 'Kata' and 'Hira'; or between the region subtags 'US' and 'UM'. 2.3 Meaning of Language Tags and Ranges A language tag defines a language as spoken (or written, signed or otherwise signaled) by human beings for communication of information to other human beings. If a language tag B contains language tag A as a prefix, then B is typically "narrower" or "more specific" than A. For example, "zh- Hant-TW" is more specific than "zh-Hant". This relationship is not guaranteed in all cases: specifically, languages that begin with the same sequence of subtags are NOT guaranteed to be mutually intelligible, although they might be. For example, the tag "az" shares a prefix with both "az-Latn" (Azerbaijani written using the Latin script) and "az-Cyrl" (Azerbaijani written using the Cyrillic script). A person fluent in one script might not be able to read the other, even though the text might be otherwise identical. Content tagged as "az" most probably is written in just one script and thus might not be intelligible to a reader familiar with the other script. Phillips & Davis Expires March 27, 2006 [Page 12]
Internet-Draft ltru-matching September 2005 Variant subtags in particular seem to represent specific divisions in mutual understanding, since they often encode dialects or other idiosyncratic variations within a language. The relationship between the language tag and the information it relates to is defined by the standard describing the context in which it appears. Accordingly, this section can only give possible examples of its usage. o For a single information object, the associated language tags might be interpreted as the set of languages that are necessary for a complete comprehension of the complete object. Example: Plain text documents. o For an aggregation of information objects, the associated language tags could be taken as the set of languages used inside components of that aggregation. Examples: Document stores and libraries. o For information objects whose purpose is to provide alternatives, the associated language tags could be regarded as a hint that the content is provided in several languages, and that one has to inspect each of the alternatives in order to find its language or languages. In this case, the presence of multiple tags might not mean that one needs to be multi-lingual to get complete understanding of the document. Example: MIME multipart/ alternative. o In markup languages, such as HTML and XML, language information can be added to each part of the document identified by the markup structure (including the whole document itself). For example, one could write <span lang="FR">C'est la vie.</span> inside a Norwegian document; the Norwegian-speaking user could then access a French-Norwegian dictionary to find out what the marked section meant. If the user were listening to that document through a speech synthesis interface, this formation could be used to signal the synthesizer to appropriately apply French text-to-speech pronunciation rules to that span of text, instead of misapplying the Norwegian rules. 2.4 Choosing Between Alternate Matching Schemes Implementations can choose to implement different styles of matching for different kinds of processing. For example, an implementation could treat an absent script subtag as a "wildcard" field; thus "az-AZ" would match "az-AZ", "az-Cyrl-AZ", "az-Latn-AZ", etc. but not "az" (this is an example of extended range lookup). If one item is to be chosen, the implementation could pick among those matches based Phillips & Davis Expires March 27, 2006 [Page 13]
Internet-Draft ltru-matching September 2005 on other information, such as the most likely script used in the language/region in question or the script used by other content selected. Because the primary language subtag cannot be absent in a language tag, the 'und' subtag is sometimes be used as a 'wildcard' in basic matching. For example, in a query where you want to select all language tags that contain 'Latn' as the script code and 'AZ' as the region code, you could use the range "und-Latn-AZ". This requires an implementation to examine the actual values of the subtags, though, and users SHOULD NOT assume that the value 'und' will be treated as a wildcard. The matching schemes described elsewhere in this document are designed such that implementations do not have to examine the values of the subtags supplied and, except for scored matching, they do not need access to the Language Subtag Registry nor do they require the use of valid subtags in language tags or ranges. This has great benefit for speed and simplicity of implementation. Implementations might also wish to use semantic information external to the langauge tags when performing fallback. For example, the primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal Norwegian) might both be usefully matched to the more general subtag 'no' (Norwegian). Or an application might infer that content labeled "zh-CN" is morely likely to match the range "zh-Hans" than equivalent content labeled "zh-TW". 2.5 Considerations for Private Use Subtags Private-use subtags require private agreement between the parties that intend to use or exchange language tags that use them and great caution SHOULD be used in employing them in content or protocols intended for general use. Private-use subtags are simply useless for information exchange without prior arrangement. The value and semantic meaning of private-use tags and of the subtags used within such a language tag are not defined. Matching private use tags using language ranges or extended language ranges can result in unpredictable content being returned. 2.6 Length Considerations in Matching RFC 3066 [RFC3066] did not provide an upper limit on the size of language tags or ranges. RFC 3066 did define the semantics of particular subtags in such a way that most language tags or ranges consisted of language and region subtags with a combined total length of up to six characters. Larger tags and ranges (in terms of both Phillips & Davis Expires March 27, 2006 [Page 14]
Internet-Draft ltru-matching September 2005 subtags and characters) did exist, however. [draft-registry] also does not impose a fixed upper limit on the number of subtags in a language tag or range (and thus an upper bound on the size of either). The syntax in that document suggests that, depending on the specific language or range of languages, more subtags (and thus characters) are sometimes necessary as a result. Length considerations and their impact on the selection and processing of tags are described in Section 2.1.1 of that document. A matching implementation MAY choose to limit the length of the language tags or ranges used in matching. Any such limitation SHOULD be clearly documented, and such documentation SHOULD include the disposition of any longer tags or ranges (for example, whether an error value is generated or the language tag or range is truncated). If truncation is permitted it MUST NOT permit a subtag to be divided, since this changes the semantics of the subtag being matched and can result in false positives or negatives. Implementations that restrict storage SHOULD consider the impact of tag or range truncation on the resulting matches. For example, removing the "*" from the end of an extended language range (see Section 2.2) can greatly modify the set of returned matches. A protocol that allows tags or ranges to be truncated at an arbitrary limit, without giving any indication of what that limit is, has the potential for causing harm by changing the meaning of values in substantial ways. In practice, most tags do not require additional subtags or substantially more characters. Additional subtags sometimes add useful distinguishing information, but extraneous subtags interfere with the meaning, understanding, and especially matching of language tags. Since language tags or ranges MAY be truncated by an application or protocol that limits storage, when choosing language tags or ranges users and applications SHOULD avoid adding subtags that add no distinguishing value. In particular, users and implementations SHOULD follow the 'Prefix' and 'Suppress-Script' fields in the registry (defined in Section 3.6 of [draft-registry]): these fields provide guidance on when specific additional subtags SHOULD (and SHOULD NOT) be used. Implementations MUST support a limit of at least 33 characters. This limit includes at least one subtag of each non-extension, non-private use type. When choosing a buffer limit, a length of at least 42 characters is strongly RECOMMENDED. The practical limit on tags or ranges derived solely from registered values is 42 characters. Implementations MUST be able to handle tags Phillips & Davis Expires March 27, 2006 [Page 15]
Internet-Draft ltru-matching September 2005 and ranges of this length. Support for tags and ranges of at least 62 characters in length is RECOMMENDED. Implementations MAY support longer values, including matching extensive sets of private use or extension subtags. Applications or protocols which have to truncate a tag MUST do so by progressively removing subtags along with their preceding "-" from the right side of the language tag until the tag is short enough for the given buffer. If the resulting tag ends with a single-character subtag, that subtag and its preceding "-" MUST also be removed. For example: Tag to truncate: zh-Hant-CN-variant1-a-extend1-x-wadegile-private1 1. zh-Hant-CN-variant1-a-extend1-x-wadegile 2. zh-Hant-CN-variant1-a-extend1 3. zh-Hant-CN-variant1 4. zh-Hant-CN 5. zh-Hant 6. zh Figure 6: Example of Tag Truncation Phillips & Davis Expires March 27, 2006 [Page 16]
Internet-Draft ltru-matching September 2005 3. IANA Considerations This document presents no new or existing considerations for IANA. Phillips & Davis Expires March 27, 2006 [Page 17]
Internet-Draft ltru-matching September 2005 4. Changes This is the first version of this document. The following changes were put into this document since draft-03: Modified the ABNF to match changes in [draft-registry] (K.Karlsson) Matched the references and reference formats to [draft-registry] (K.Karlsson) Various edits, additions, and emendations to deal with changes in the Last Call of draft-registry as well as cleaning up the text. Phillips & Davis Expires March 27, 2006 [Page 18]
Internet-Draft ltru-matching September 2005 5. Security Considerations Language ranges used in content negotiation might be used to infer the nationality of the sender, and thus identify potential targets for surveillance. In addition, unique or highly unusual language ranges or combinations of language ranges might be used to track specific individual's activities. This is a special case of the general problem that anything you send is visible to the receiving party. It is useful to be aware that such concerns can exist in some cases. The evaluation of the exact magnitude of the threat, and any possible countermeasures, is left to each application protocol. Phillips & Davis Expires March 27, 2006 [Page 19]
Internet-Draft ltru-matching September 2005 6. Character Set Considerations The syntax of language tags and language ranges permit only the characters A-Z, a-z, 0-9, and HYPHEN-MINUS (%x2D). These characters are present in most character sets, so presentation of language tags should not present any character set issues. Phillips & Davis Expires March 27, 2006 [Page 20]
Internet-Draft ltru-matching September 2005 7. References 7.1 Normative References [ID.ietf-ltru-initial] Ewell, D., Ed., "Language Tags Initial Registry (work in progress)", August 2005, <http://www.ietf.org/ internet-drafts/draft-ietf-ltru-initial-04.txt>. [RFC1327] Hardcastle-Kille, S., "Mapping between X.400(1988) / ISO 10021 and RFC 822", RFC 1327, May 1992. [RFC1521] Borenstein, N. and N. Freed, "MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC 1521, September 1993. [RFC2028] Hovey, R. and S. Bradner, "The Organizations Involved in the IETF Standards Process", BCP 11, RFC 2028, October 1996. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2231] Freed, N. and K. Moore, "MIME Parameter Value and Encoded Word Extensions: Character Sets, Languages, and Continuations", RFC 2231, November 1997. [RFC2234bis] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", draft-crocker-abnf-rfc2234bis-00 (work in progress), March 2005. [RFC2396] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifiers (URI): Generic Syntax", RFC 2396, August 1998. [RFC2434] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. [RFC2860] Carpenter, B., Baker, F., and M. Roberts, "Memorandum of Understanding Concerning the Technical Work of the Internet Assigned Numbers Authority", RFC 2860, June 2000. Phillips & Davis Expires March 27, 2006 [Page 21]
Internet-Draft ltru-matching September 2005 [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003. [draft-registry] Phillips, A., Ed. and M. Davis, Ed., "Tags for the Identification of Languages (work in progress)", August 2005, <http://www.ietf.org/internet-drafts/ draft-ietf-ltru-registry-12.txt>. 7.2 Informative References [ISO15924] "ISO 15924:2004. Information and documentation -- Codes for the representation of names of scripts", January 2004. [ISO3166-1] "ISO 3166-1:1997. Codes for the representation of names of countries and their subdivisions -- Part 1: Country codes", 1997. [ISO639-1] "ISO 639-1:2002. Codes for the representation of names of languages -- Part 1: Alpha-2 code", 2002. [ISO639-2] "ISO 639-2:1998. Codes for the representation of names of languages -- Part 2: Alpha-3 code, first edition", 1998. [RFC1766] Alvestrand, H., "Tags for the Identification of Languages", RFC 1766, March 1995. [RFC3066] Alvestrand, H., "Tags for the Identification of Languages", BCP 47, RFC 3066, January 2001. [RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, July 2002. [UN_M.49] Statistics Division, United Nations, "Standard Country or Area Codes for Statistical Use", UN Standard Country or Area Codes for Statistical Use, Revision 4 (United Nations publication, Sales No. 98.XVII.9, June 1999. Phillips & Davis Expires March 27, 2006 [Page 22]
Internet-Draft ltru-matching September 2005 Authors' Addresses Addison Phillips (editor) Quest Software Email: addison dot phillips at quest dot com Mark Davis (editor) IBM Email: mark dot davis at ibm dot com Phillips & Davis Expires March 27, 2006 [Page 23]
Internet-Draft ltru-matching September 2005 Appendix A. Acknowledgements Any list of contributors is bound to be incomplete; please regard the following as only a selection from the group of people who have contributed to make this document what it is today. The contributors to [draft-registry], [RFC3066] and [RFC1766], each of which is a precursor to this document, made enormous contributions directly or indirectly to this document and are generally responsible for the success of language tags. The following people (in alphabetical order by family name) contributed to this document: Jeremy Carroll, John Cowan, Frank Ellermann, Doug Ewell, Kent Karlsson, Ira McDonald, M. Patton, Randy Presuhn and many, many others. Very special thanks must go to Harald Tveit Alvestrand, who originated RFCs 1766 and 3066, and without whom this document would not have been possible. For this particular document, John Cowan originated the scheme described in Section 2.2.3. Mark Davis originated the scheme described in the Section 2.1.2. Phillips & Davis Expires March 27, 2006 [Page 24]
Internet-Draft ltru-matching September 2005 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Phillips & Davis Expires March 27, 2006 [Page 25]