Language basics

Text is rendered or processed differently according to the language it is in. For example, screen readers need to be prompted when a language changes, and spell checkers should be language-sensitive. When rendering text a knowledge of language is need in order to apply correct fonts, hyphenation, line-breaking, upper/lower case changes, and other features.

For example, ideographic characters such as 雪, 刃, 直, 令, 垔 have slight but important differences when used with Japanese vs Chinese fonts, and it's important not to apply a Chinese font to the Japanese text, and vice versa when it is presented to a user.

Language information for a given resource can be used with two main objectives in mind: for text-processing, or as a statement of the intended use of the resource. We will explain the difference below.

Defining language values

See related review comments.

BCP 47 is the language tag system used by Internet and Web standards (and many other places). It defines a method of using subtags from an IANA registry to form a string which describes the language of content. The subtags in the registry are primarily based on (and maintain strict compatibility with) ISO and UN standards for identifying languages, scripts, regions, and countries. BCP47 also forms the basis for Unicode locales.

For an overview of the key features of BCP 47, see Language tags in HTML and XML.

The link to and name of BCP 47 was created specifically so that there is an unchanging reference to the definition of Tags for the Identification of Languages. RFCs 1766, 3066, 4646 were previous (superseded) versions. The current version of BCP 47 is made up of two RFCs: 5646 and 4647.

A well-formed BCP 47 language tag follows the syntax defined for a language tag: implementations check that each language tag consists of hyphen-separated subtags; each subtag has a specific length and specific content (letters, digits or specific combinations) depending on the placement in the tag. A valid BCP 47 language tag is well-formed but additionally ensures that only subtags that are listed in the IANA Subtag Registry are used. Note that the IANA Subtag Registry is frequently updated with new subtags.

Most specifications are second-order consumers of language metadata – they are using data already provided in the document format (HTML lang, XML xml:lang, or the document format's language fields/attributes).

Generally most specifications are concerned with selecting resources (such as spell checkers, tokenizers, fonts, etc.) or with matching (selecting which string to show, for example) and don't directly care about the content of the language tag. Invalid-but-well-formed tags just don't match anything and usually fallback schemes provide some behavior that is appropriate.

There might be cases where a specification really wants implementation-level checking for validity. In those cases, the result of a tag failing to be valid has to be specified (should the application die, warn the user, etc.). It's also a problem that the registry is sizeable and changes over time, so each implementation is registry-version dependent. The changes over time are often minor, but real users will encounter interoperability issues if random (out of date) implementations of the specification reject language tags that have become valid at a later date.

In addition, BCP 47 has an extension mechanism which defines add-on subtag sequences. For example, one extension [[RFC6067]] (Unicode Locales, which uses the singleton -u), is commonly used for controlling the internationalization features of JavaScript (and has other uses). Validating these additional subtags is likely out of scope for most specifications.

Normative language regarding language tags might be different between content and implementation requirements. Specification authors need to carefully consider what conformance requirements and tests are needed for their specification and what implementations are required to do. One solution is to normatively require that "valid" language tags be used by content authors but only require implementations to check for "well-formed" language tags.

In the past, some of the standards used to provide subtags found in language tags were not freely or publicly available, so some specifications provided lists in order to help ensure interoperability. This is no longer necessary. As part of BCP 47, IANA maintains the language subtag registry, which is a publicly available, machine-readable list of valid subtags for use in constructing language tags. This registry is based on underlying standards, including the various parts of ISO 639 (639-1, 639-2, 639-3, etc.), ISO 15924 script codes, and ISO 3166 and UN M.49 region codes. The registry is actively maintained, stabilized, and comprehensive in ways that other lists found on the Internet might not be. Each of the subtag types is kept in sync with parent standards with the help and participation of those standards maintainers, so extracting or making your own list of codes or referring to ones found elsewhere can lead to maintenance problems or confusion.

Making your own list of fully formed language tags will unnecessarily restrict the list of languages that can be used. In addition, locale data is always being expanded, so a list that describes support today will become outdated in the future. Restricting which tags or subtags are available to users conflicts with our goal of providing universal access.

Declaring language

See related review comments.

Identifying the language of strings

The information in this section is being developed in Requirements for Language and Direction Metadata in Data Formats [[STRING-META]]. That document is still being written, so these guidelines are likely to change at any time.

The exchange of data on the Web, to the degree possible, should use locale-neutral standardized formats. However, some data on the Web necessarily consists of natural language information intended for display to humans. This natural language information depends on and benefits from the presence of language and direction metadata for proper display. Along with support for Unicode, mechanisms for including and specifying the base direction and the natural language of spans of text are one of the key internationalization considerations when developing new formats and technologies for the Web.

The most basic best practice, which the Internationalization Working Group looks for in every specification, is:

See related review comments.

Individual data values can differ in language or direction from other values found in the same data file or document. Providing metadata values directly associated with each localizable text field allows for the metadata to be overridden appropriately and helps applications automate processing when assembling, extracting, forwarding, or otherwise processing each data field for use.

Many documents contain data in a single language. Providing a means of indicating the intended language audience, perhaps in a header, can reduce overall document size and complexity. However, the ability to override specific string values remains important, as it is always possible that some strings might not be available in the document language or when the base direction is not consistent with the default direction of other localizable text values in the document as a whole.

Document formats on the Web consist of text. In most cases, data values in a given document format are meant to be representative and meaningful, not just arbitrary strings. The fact that a data value consists of, for example, an English keyword does not make the data value a natural language string meant for display as text (that is, the value is not localizable text). Such data values are part of the syntactic content of the document: not only do they not require language and direction metadata, but they should not be associated with such metadata.

Some string values depend on or are defined by existing protocols or formats. Often these strings are not associated with or do not provide language or direction metadata. For example, many HTTP headers define their contents as if they were not localizable text, even when, in some cases, they contain natural language text. Consuming specifications sometimes need to take a dependency on strings of this nature or define a format that describes one of these strings. In these cases there will be no language or direction metadata for consumers to associate with the string in the specification's data structure or document format, and any metadata that the specification's data structure or document format provides (when functioning as a producer) will not be serialized through the underlying format.

The Unicode "language tag" characters are deprecated for use as language tags and there are many reasons why they are a poor solution to the language metadata problem in document formats and wire protocols. Specification authors are cautioned not to repurpose these characters or try to build new mechanisms for transmitting language information based on them.

Producers sometimes need to supply localized values for a given content item or data record. Sometimes this is done by language negotiation between the producer and consumer. Localization then takes place in the producer using the negotiated language to select the content returned.

Other times localization of a content item is done by having the producer return multiple language representations for the item and letting the consumer choose the value to display. This latter process is called language indexing. For more information about language indexing, see Localization Considerations in [[STRING-META]].

Detecting & matching language

See related review comments.

In addition to defining language tags (in RFC 5646) BCP 47 also contains an RFC on the topic of matching language tags to a [=language range=]. Just as it is most appropriate to refer to the stable identifier BCP 47 for the definition of language tags, it is best to refer to BCP 47 when referencing matching schemes found therein.

Unicode's [[CLDR]] project defines additional algorithms, rules and processes for matching language tags when used as [=locale=] identifiers.

Basic requirements

The basic requirements are as follows.

A special case of the above applies to [=natural language=] string values in data structures and document formats:

Requiring a speaker of Arabic, Divehi, Hebrew, Persian, Urdu, etc. to add markup or control characters to every paragraph or small data item they write is far too much to be manageable. Typically, the format should establish a default direction and require the user to intervene only when exceptions have to be dealt with.

Background information

In this section we try to set out some key concepts associated with text direction, so that it will be easier to understand the recommendations that follow.

Base direction values

See related review comments.

The auto value allows automatic detection of the base direction for a piece of text. For example, the auto value of dir in HTML looks for the first strong directional character in the text, but ignores certain items of markup also, to guess the base direction of the text. Note that automatic detection algorithms are far from perfect. First-strong detection is unable to correctly identify text that is really right-to-left, but that begins with a strong LTR character. Algorithms that attempt to judge the base direction based on contents of the text are also problematic. The best scenario is one where the base direction is known and declared.

Handling direction in markup

This section is about defining approaches to bidi handling that work with resources that organize content using markup. Some of the recommendations are different from those for handling strings on the Web (see [[[#bidi_strings]]]).

See related review comments.

Handling base direction for strings

The information in this section is pulled from Strings on the Web: Language and Direction Metadata. That document is still being written, so these guidelines are likely to change at any time.

See related review comments.

Setting base direction for inline or substring text

'Inline text' here has a readily understandable meaning in markup. It also applies to strings (eg. in JSON, CSV, or other plain text formats), meaning runs of characters which don't include all the characters in the string.

The first-strong algorithm looks for the first character in the paragraph with a strong directional property according to the Unicode definitions. It then sets the [=base direction=] of the paragraph according to the direction of that character.

Note that the first-strong algorithm may incorrectly guess the direction of the paragraph when the first character is not typical of the rest of the paragraph, such as when an [=RTL=] paragraph or line starts with a [=LTR=] brand name or technical term.

For additional information about algorithms for detecting direction, see Estimation algorithms in the document where this was discussed with reference to HTML.

The Unicode bidirectional control characters U+200F RIGHT-TO-LEFT MARK and U+200E LEFT-TO-RIGHT MARK are not sufficient on their own to manage bidirectional text. They cannot produce a different base direction for embedded text. For that you need to be able to indicate the start and end of the range of the embedded text.  This is best done by markup, if available, or failing that using the other Unicode bidirectional controls mentioned just above.

Detecting & matching direction (TBD)

See related review comments.

Choosing a definition of 'character'

See related review comments.

The term character is used differently in a variety of contexts and often leads to confusion when used outside of these contexts. In the context of the digital representations of text, a character can be defined as a small logical unit of text. Text is then defined as sequences of characters. While such an informal definition is sufficient to create or capture a common understanding in many cases, it is also sufficiently open to create misunderstandings as soon as details start to matter. In order to write effective specifications, protocol implementations, and software for end users, it is very important to understand that these misunderstandings can occur.

This section examines some of these contexts, meanings and confusions.

Specific terms could include [=code point=], [=grapheme cluster=], typographic character unit, [=code unit=], and [=glyph=].

The developers of specifications, and the developers of software based on those specifications, are likely to be more familiar with usages of the term 'character' they have experienced and less familiar with the wide variety of usages in an international context. Furthermore, within a computing context, characters are often confused with related concepts, resulting in incomplete or inappropriate specifications and software.

Computer storage and communication rely on units of physical storage and information interchange, such as bits and bytes (8-bit units, also called octets). A frequent error in specifications and implementations is the equating of characters with units of physical storage. The mapping between characters and such units of storage is actually quite complex.

In some scripts, characters have a close relationship to phonemes (a phoneme is a minimally distinct sound in the context of a particular spoken language), while in others they are closely related to meanings. Even when characters (loosely) correspond to phonemes, this relationship may not be simple, and there is rarely a one-to-one correspondence between character and phoneme.

Visual rendering introduces the notion of a glyph. Glyphs are defined by ISO/IEC 9541-1 as a recognizable abstract graphic symbol which is independent of a specific design. There is not a one-to-one correspondence between characters and glyphs:

• A single character can be represented by multiple glyphs (each glyph is then part of the representation of that character). These glyphs may be physically separated from one another.
• A single glyph may represent a sequence of characters (this is the case with ligatures, among others).
• A character may be rendered with very different glyphs depending on the context.
• A single glyph may represent different characters (e.g. capital Latin A, capital Greek A and capital Cyrillic A).

A set of glyphs makes up a font. Glyphs can be construed as the basic units of organization of the visual rendering of text, just as characters are the basic unit of organization of encoded text.

See Examples of Characters, Keystrokes and Glyphs for examples of the complexities of character to glyph mapping.

In keyboard input, it is not always the case that keystrokes and input characters correspond one-to-one. A limited number of keys can fit on a keyboard. Some keyboards will generate multiple characters from a single keypress. In other cases ('dead keys') a key will generate no characters, but affect the results of subsequent keypresses. Many writing systems have far too many characters to fit on a keyboard and must rely on more complex input methods, which transform keystroke sequences into character sequences. Other languages may make it necessary to input some characters with special modifier keys.

See Examples of Characters, Keystrokes and Glyphs for examples of non-trivial input.

Defining a Reference Processing Model

Including and excluding character ranges

See related review comments.

A "surrogate code point" refers here to the use of character values in the range U+D800 through U+DFFF inclusive. These code points are reserved to allow the UTF-16 character encoding to address supplementary characters. Surrogates are always used in pairs and only appear when the UTF-16 encoding is being used. A single surrogate code point is referred to as an "unpaired surrogate" and should never be used.

Using the Private Use Area

Choosing character encodings

See related review comments.

The above guideline needs further consideration: UTF-16 and UTF-32 are not recommended these days. UTF-8 is the recommended encoding.

The above guideline needs further consideration: the list of character encodings recommended for Web specifications is listed in the Encoding specification.

Identifying character encodings

Designing character escapes

See related review comments.

It is generally recommended that character escapes be provided so that difficult to enter or edit sequences can be introduced using a plain text editor. Escape sequences are particularly useful for invisible or ambiguous Unicode characters, including zero-width spaces, soft-hyphens, various bidi controls, mongolian vowel separators, etc.

For advice on use of escapes in markup, but which is mostly generalisable to other formats, see Using character escapes in markup and CSS.

Storing text

Defining 'string'

A string is a sequence of characters. Because [[UNICODE]] is fundamental to understanding and working with text, including text that uses legacy character encodings, the basic definition of a string depends on Unicode and its concept of a encoded character. Specifically:

A string is a well-formed sequence of zero or more Unicode Scalar Values.

Because there are multiple ways of working with strings, different terminology has evolved to support the needs of different specifications. Be sure to understand your specification's needs and use the most appropriate and precise terminology. On the Web, there are three types of strings:

• {{USVString}}. Strings based on Unicode code points, also known as Unicode Scalar Values
• {{DOMString}}. Strings based on UTF-16 code units
• {{ByteString}}. Strings based on bytes in some character encoding form (preferably UTF-8)

One difference between these different string types is how surrogate code points are handled. Note the difference between a code point (which represents a Unicode Scalar Value, i.e. a character) and a code unit (a unit of encoding in a character encoding form).

The UTF-16 character encoding form uses 16-bit code units. Characters whose scalar values require more than 16-bits are encoded using a pair of surrogate code units: a "low surrogate" (in the range U+D800-U+DBFF) followed by a "high surrogate" (in the range U+DC00-U+DFFF). Unicode reserves the code points in these ranges as non-characters so that there is no confusion between the code units in UTF-16 and normal text.

In a {{USVString}}, isolated surrogate code points are invalid and implementations are required to replace any found in a string with the Unicode replacment character (�U+FFFD REPLACEMENT CHARACTER). For strings whose most common algorithms operate on scalar values (such as percent-encoding), or for operations which can’t handle surrogates in input (such as APIs that pass strings through to native platform APIs), {{USVString}} should be used. Any of these references are equivalent to this:

• {{USVString}} [[WEBIDL]]
• scalar value string [[INFRA]]
• xsd:string [[XMLSCHEMA11-2]]

In a {{DOMString}}, unpaired surrogate code units can appear in a string. Most string operations don’t need to interpret the code units inside of strings. Specifying {{DOMString}} means that implementations are not required to validate the contents of the string, making this the ideal string type for most data structures, formats, or APIs. The [[DOM]] and JavaScript strings use {{DOMString}} as their string type and the [[INFRA]] standard defines the term 'string' to mean a {{DOMString}}:

A string is a sequence of unsigned 16-bit integers, also known as code units.

[[INFRA]]'s use of the term code unit refers specifically to the UTF-16 character encoding's code units, rather than the more general definition of a code unit that can refer to different size values, such as bytes, in any character encoding form.

A {{ByteString}} depends on the character encoding form used to encode characters into bytes. Legacy character encodings do not have a concept of "surrogates", so there is generally no way to encode a surrogate code point. Valid UTF-8 does not permit surrogate code points: these are replaced by �U+FFFD REPLACEMENT CHARACTER when encoding or decoding text in UTF-8. When converting UTF-16 to UTF-8, any surrogate pairs are transformed into the proper UTF-8 byte sequence encoding the specific scalar value.

The type {{ByteString}} defines strings as sequences of bytes (octets). Interpretation of byte strings thus requires the specification of a character encoding form. UTF-8 is the preferred encoding for wire and document formats [[ENCODING]], but there is generally no reason to specify strings in terms of the underlying byte values.

See for additional best practices.

Whitespace characters

See related review comments.

Whitespace characters are characters that represent horizontal or vertical space in typography. Whitespace characters can have different visual effects: some whitespace characters have no visible effect, while others represent larger, smaller, or variable amounts of space on the page.

Some specifications, such as ECMAScript, have provided their own definition of whitespace which differ from the above to meet their own specific requirements.

The following table is the definition of whitespace characters in various specifications.

Links to the latest definitions of the information in the table can be found by expanding the "explanations & examples".

	white_space property	pattern_white_space property	ASCII whitespace (HTML)	CSS whitespace	ECMAScript	XML
U+0009 (horizontal tab)	✓	✓	✓	✓	✓	✓
U+000A (line feed)	✓	✓	✓	✓		✓
U+000B (vertical tab)	✓	✓			✓
U+000C (form feed)	✓	✓	✓		✓
U+000D (carriage return)	✓	✓	✓			✓
U+0020 SPACE	✓	✓	✓	✓	✓	✓
U+0085 (next line)	✓	✓
U+00A0 NO-BREAK SPACE	✓				✓
U+1680 OGHAM SPACE MARK	✓				✓
U+2000 EN QUAD	✓				✓
U+2001 EM QUAD	✓				✓
U+2002 EN SPACE	✓				✓
U+2003 EM SPACE	✓				✓
U+2004 THREE-PER-EM SPACE	✓				✓
U+2005 FOUR-PER-EM SPACE	✓				✓
U+2006 SIX-PER-EM SPACE	✓				✓
U+2007 FIGURE SPACE	✓				✓
U+2008 PUNCTUATION SPACE	✓				✓
U+2009 THIN SPACE	✓				✓
U+200A HAIR SPACE	✓				✓
U+200E LEFT-TO-RIGHT MARK		✓
U+200F RIGHT-TO-LEFT MARK		✓
U+2028 LINE SEPARATOR	✓	✓
U+2029 PARAGRAPH SEPARATOR	✓	✓
U+202F NARROW NO-BREAK SPACE	✓				✓
U+205F MEDIUM MATHEMATICAL SPACE	✓				✓
U+3000 IDEOGRAPHIC SPACE	✓				✓
U+FEFF ZERO WIDTH NO-BREAK SPACE					✓

Some specifications use the same definition as one of the columns above and are not listed in the table. For example, WebDriver uses the white_space property and WebGPU Shading Language uses the pattern_white_space property.

Referring to Unicode characters

See related review comments.

The U+XXXX format is well understood when referring to Unicode code points in a specification. These are space separated when appearing in a sequence. No additional decoration is needed. Note that a code point may contain four, five, or six hexadecimal digits. When fewer than four digits are needed, the code point number is zero filled.

Unicode assigns unique, immutable names to each assigned Unicode code point. Using these names in your specification when referring to specific characters (along with the code point in U+XXXX notation) will help make your specification unambiguous.

For most characters, the template looks like this:

The bdi element is used to ensure that example characters that are right-to-left do not interfere with the layout of the page. Do not include line breaks or a space between the closing bdi and the following code element; spacing and presentation is controlled by styling.

The lang attribute should be filled in appropriately to get the correct font selection for a given context. Examples in East Asian languages (such as Chinese, Japanese, or Korean) or in the Arabic script can sometimes require greater care in choosing a language tag. Rarely, for certain languages, it might be necessary to adjust the style of the bdi element with a font-family and/or font-size in your own stylesheet.

For invisible characters (such as control characters), combining characters, or for whitespace, use an image instead of the character; or you may also omit the character and its surrounding bdi element.

Short sequences of characters should list the character names, separated by +.

There are cases where including the character name and additional markup is overly pedantic and detracts from usability, but be cautious about being so informal as to impair meaning. In particular, long sequences will sometimes just list the code points, although the character names should be retained where possible for clarity. An example can be found in this document in the discussion of the composed "family" emoji: 👨‍👩‍👧‍👧U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F467

Choosing text units for segmentation, indexing, etc.

See related review comments.

There are many situations where a software process needs to access a substring or to point within a string and does so by the use of indices, i.e. numeric "positions" within a string. Where such indices are exchanged between components of the Web, there is a need for an agreed-upon definition of string indexing in order to ensure consistent behavior. The two main questions that arise are: "What is the unit of counting?" and "Do we start counting at 0 or 1?".

A counter-example is the use of UTF-16 in DOM Level 1. The use of UTF-16 code points is discouraged because it leaves open the possibility of an index occuring between two surrogate characters, which would cause significant problems (see [[[#char_truncation]]]).

Matching string identity for identifiers and syntactic content

See related review comments.

Working with Unicode Normalization

See related review comments.

Case folding

See related review comments.

Truncating or limiting the length of strings

See related review comments.

Some specifications, formats, or protocols or their implementations need to specify limits for the size of a given data structure or text field. This could be due to many reasons, such as limits on processing, memory, data structure size, and so forth. When selecting or specifying limits on the length of a given string, specifications or implementations need to ensure that they do not cause corruption in the text.

There are many reasons why a length limit might be needed in a specification or format. Generally length limits correspond to underlying limits in the implementation, such as the use of fixed-size fields in a database or data store, the desire to fit into practical boundaries such as packet size, or some other implementation detail related to storage allocation or efficiency.

When truncating strings, it's necessary to decide what units to use when counting the size of the string. In many cases this is beyond the control of the specification, since the truncation is occuring for some preordained reason. However, when the choice is available, some general guidelines can be applied.

If the limitation is related to the number of display positions, the grapheme count usually corresponds most closely to the expected limit. Note that proportional width fonts, combining marks, complex scripts, and many other factors complicate counting "screen positions". In Web pages, for example, the CSS text-overflow property provides visual truncation without disturbing the content of the text. Attempts to estimate the size of a given piece of text based on the number of Unicode code points or even the number of grapheme clusters is mostly futile.

Otherwise most limits are expressed in terms of code points in Unicode or code units (such as bytes) in a specific character encoding. Code points provides the best user experience, since all Unicode code points are treated identically: if text is truncated after 40 code points, all languages and scripts get the same number of code points to work with. By contrast, when the size limit is expressed in code units such as bytes in UTF-8, users who write in a language that mostly uses ASCII letters get many more characters (code points) for a given size limit than user's whose language is mostly made up of characters that take 2-, 3-, or 4-bytes per code point.

Note that this best practice applies equally to specifications based on UTF-16, which uses 16-bit code units, not just to multibyte encodings such as UTF-8.

Specifications or APIs that interact with the [[DOM]] need to contend with the fact that character data, including operations such as length, substringData, insertData, deleteData, and so forth, is specified using UTF-16 code units, not Unicode code points. This can lead to inappropriate mid-character (code point) truncation. Specifications that reference DOM should specify that string operations not occur inside code points, and, where appropriate avoid starting or ending inside grapheme boundaries. Specifications should also include a health warning for implementers and users.

If a specification permits or requires truncation of a field, the character encoding is important in knowing what the limit means. If the limit is in bytes and legacy character encodings are permitted, note that conversion of Unicode data to a non-Unicode encoding can also result in data loss (since most legacy character encodings encode only a subset of Unicode).

Concatenation of strings

See related review comments.

Creation of [=natural language=] text values by concatenating multiple strings together is an internationalization anti-pattern. Languages vary greatly in word order, count, grammatical gender or case, punctuation, and many other requirements. As a result, avoid requiring or suggesting that implementations generate human-readable messages from sub-strings.

Specifications for APIs, protocols, or document formats sometimes require an implementation to create or provide a field containing a display name or description. When such a string is assembled from separate parts, it can result in problems with presentation or understanding due to the way that the Unicode Bidirectional Algorithm [[UAX9]] processes the assembled string. In such cases, the specification should guide implementers about how to create values that will display properly.

Working with file and path names

Some specifications need to define how file names or file paths are constructed by various implementations. One challenge is building definitions that work consistently when used on the different file systems used by different operating systems. This section contains general guidance when defining restrictions on file names or file paths. It is based on requirements developed in [[EPUB-33]], as well as implementation experience.

Specifying sort and search functionality

See related review comments.

Applications often need to organize sets of information or content. Frequently this involves sorting the content. Many non-textual data types, such as numbers or dates, can be easily sorted using the internal data representation. When it comes to textual information, however, the nature of character encodings and user expectations regarding "alphabetical" order brings some additional complexity.

One key choice is whether the sorting of textual data will be strictly internal or whether the results will be shown to users.

Defining elements and attributes in markup

See related review comments.

Handling plain text in markup

See related review comments.

Internationalization information may include language and base direction metadata, inline changes of language, bidirectional text behavioural changes, translate flags, etc.

Defining identifiers

See related review comments.

A common feature of document formats is the definition of various identifiers. This includes reserved keywords as well as user-defined values. To foster interoperability, implementations need to be able to match identifier values reliably and consistently. For a detailed look at this problem, see Character Model: String Matching [[CHARMOD-NORM]].

Sometimes specifications need to define a set of identifiers that content authors interact with or which are meaningful to various types of end-users. Restricting the set of allowable characters to ASCII impedes usability, particularly for speakers of languages that do not use the Latin script or that use characters outside of the ASCII range.

One key issue when defining an identifier namespace or set of identifiers in a new specification is the handling of combining marks and certain other characters (such as joiners or bidi controls) when parsing the document format: special focus needs to be paid to how the identifier can be "tokenized" (separated from the surrounding text). One means of doing this is to restrict the range of characters allowed to start an identifier to ensure that normal text processing doesn't interfere with matching the identifier later.

Unicode Identifier and Pattern Syntax [[UAX31]] provides one model, used notably in programming languages such as Java or JavaScript. HTML and CSS also provide character range definitions for custom identifiers, such as this EBNF [[XML]] production:

PCENChar ::=
    "-" | "." | [0-9] | "_" | [a-zA-Z] | #xB7 | [#xC0-#xD6] | [#xD8-#xF6] | [#xF8-#x37D] | 
    [#x37F-#x1FFF] | [#x200C-#x200D] | [#x203F-#x2040] | [#x2070-#x218F] | [#x2C00-#x2FEF] |
    [#x3001-#xD7FF] | [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]
	

Specifications should exercise care when defining identifiers with regards to the handling of whitespace. Note that there are Unicode horizontal whitespace characters other than the ASCII characters U+0020 SPACE and U+0009 TAB.

When defining identifiers, including field names and values, choose names that are as culturally-neutral as possible. For example, prefer postalCode to the (USA-specific) ZIPCode or prefer givenName/familyName to the more-culturally linked firstName/lastName.

Text decoration

See related review comments.

Skipping ink for text decoration such as underlines may not be appropriate for some scripts, such as Arabic, which prefers to move the underline further away from the baseline instead.

Vertical text

See related review comments.

RTL/bidi text

See related review comments.

Setting box positioning coordinates when text direction varies

It is typical, when localizing a user interface or web page, to create mirror-images for the RTL and LTR versions. For example, it is likely that a box that appears near the left side of a window containing English content would appear near the right side of the window if the content is Arabic or Hebrew. It should preferably automatic for this to change, based on the base direction of the current context, unless there is a strong reason for using absolute geometry. One way to achieve this is to use keywords such as start and end, rather than left and right, to indicate position.

Logical properties (TBD)

See related review comments.

Cursive text

See related review comments.

Ruby text annotations

See related review comments.

Font management (TBD)

See related review comments.

Miscellaneous

See related review comments.

Various non-Latin writing systems don't simply wrap text on inter-word spaces. They have additional rules that must be respected. For example

• Chinese, Japanese and Korean wrap after characters, but don't put certain characters at the start/end of a line.
• Thai and other SE Asian scripts wrap at word boundaries, but words are not delimited by spaces – spaces are instead used to separate phrases.
• Tibetan wraps after the tsek character that follows a syllable – words are not separated by spaces, and lines can break within a word.
• Indic and other complex scripts break at orthographic syllable boundaries, which are often two or more grapheme clusters.

See the CSS Text Level 3 specification for additional background. (This tutorial provides additional examples, if needed.)

It is best to avoid presentational markup b, i or u, since it isn't interoperable across writing systems and furthermore may cause unnecessary problems for localisation. In addition, some scripts have native approaches to things such as emphasis, that do not involve, and can be very different from, bolding, italicisation, etc.

In the HTML case, there was a legacy issue, but unless there is one for your specification, the recommendation is that styling be used instead to determine the presentation of the text, and that any markup or tagging should allow for general semantic approaches.

For an explanation of the issues surrounding b and i tags, see Using <b> and <i> elements.

Working with locale-affected values

Software systems that support languages and cultural preferences are said to be internationalized. An internationalized system uses APIs to provide language or culturally specific processing, based on user preferences. These user preferences are usually referred to as a locale. For more information on general internationalization terminology, see Language Tags and Locale Identifiers [[LTLI]]

Data values that are machine-readable and not specific to any particular language or culture are more durable and less open to misinterpretation than values that use one of the many different cultural representations. Things like dates, currencies, and numbers might look similar but have different meanings in different locales. For example, a date represented as the string 4/7 can be read as the 7th of April or the 4th of July depending on the user's preference. Similarly, €2,000 is either two thousand Euros or an over-precise representation of two Euros. By using a locale-neutral format, systems avoid the need to establish specific interchange rules that vary according to the language or location of the user. When the data is already in a locale-specific format, making the locale and language explicit by providing locale parameters (usually in the form of a language tag) allows users to determine how to work with the data or perhaps enable automated translation services.

Most common data serialization formats are locale-neutral. For example, [[XMLSchema-2]] types such as xsd:integer and xsd:date are intended for locale-neutral data interchange. Using locale-neutral representations allows the data values to be processed accurately without complex parsing or misinterpretation and also allows the data to be presented in the format most comfortable for the consumer of the data in any locale. For example, rather than storing "€2000,00" as a string, it is strongly preferred to exchange a data structure such as:

…
"price" {
    "value": 2000.00,
    "currency": "EUR"
}
…

Working with time

See related review comments.

These occur occasionally when the number of seconds in a minute is allowed to range from 0 to 60 (ie. there are sixty-ONE seconds in that minute).

For example, the weekend is not always Saturday/Sunday; the first day of week is not always Sunday [or Monday or...], etc.

Working with personal names

See related review comments.

Developers who create applications that use personal names (in web forms, databases, ontologies, and so forth) are often unaware of how different names can be in other countries. They build their forms or databases in a way that assumes too much on the part of foreign users. This section provides guidelines for working with personal names from around the world.

Working with numbers

Designing forms

See related review comments.

User input (TBD)

See related review comments.

Creating examples (TBD)

See related review comments.

Localization

Localization [[LTLI]] enables users to employ software in the language and locale of their choice. Specifications for protocols and document formats need to consider how to provide the language and formatting that the end-user expects.

Natural language data values need language and base direction in order to ensure proper presentation, even if localized messages are not provided. This includes any error messages or other internal messages that are human readable in an API or protocol. See also [[STRING-META]].

Specifications do not need to require that messages be returned in all possible or all available locales. It is sufficient to make it possible to localize the end-user's customer experience. Implementations can choose which languages or locales to support.

Providing for content negotiation based on language

See related review comments.