| Character Encoding |
Article Index for Character |
Shopping Encoding |
Articles about Character Encoding |
Website Links For Character Encoding |
Information AboutCharacter Encoding |
| CATEGORIES ABOUT CHARACTER ENCODING | |
| character encodingcharacter encoding | |
| computer data | |
| encodings | |
| coding theory | |
| presentation layer protocols | |
| digital typography | |
|
In earlier days of computing, the introduction of character sets such as ASCII (1963) and EBCDIC (1964) began the process of standardisation. The limitations of such sets soon became apparent, and a number of ad-hoc methods developed to extend them. The need to support multiple Writing System s, including the CJK family of East Asian scripts, required support for a far larger number of characters and demanded a systematic approach to character encoding rather than the previous ad hoc approaches. SIMPLE CHARACTER SETS Conventionally character set and character encoding were considered synonymous, as the same standard would specify both what characters were available and how they were to be encoded into a stream of code units (usually with a single character per code unit). For historical reasons, MIME and systems based on it use the term charset to refer to the complete system for encoding a sequence of characters into a sequence of octets. MODERN ENCODING MODEL Unicode and its parallel standard, ISO 10646 Universal Character Set , which together constitute the most modern character encoding, broke away from this idea, and instead separated the ideas of what characters are available, their numbering, how those numbers are encoded as a series of "code units" (limited-size numbers), and finally how those units are encoded as a stream of octets (bytes). The idea behind this decomposition is to establish a universal set of characters that can be encoded in a variety of ways. To correctly describe this model needs more precise terms than "character set" and "character encoding". The terms used in the modern model follow: A character repertoire is the full set of abstract characters that a system supports. The repertoire may be closed, that is no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series), or it may be open, allowing additions (as is the case with Unicode and to a limited extent the Windows Code Pages ). The characters in a given repertoire reflect decisions that have been made about how to divide writing systems into linear information units. The Latin, Greek, and Cyrillic alphabets, for example, are naturally broken down into letters, digits, diacritics, punctuation, and a few special characters like the space, which can all be arranged in simple linear sequences (although processing them requires adding rules about how certain sequences, such as a letter followed by its diacritics, are interpreted—this is not within the scope of a character repertoire). For convenience, such repertoires may contain precomposed combinations of letters and diacritics. Other writing systems, such as Arabic and Hebrew, are represented with more complex character repertoires due to the need to accommodate things like bidirectional text and glyphs that are joined together in different ways for different situations. A coded character set specifies how to represent a repertoire of characters using a number of non-negative integer codes called ''code points''. For example, in a given repertoire, a character representing the capital letter "A" in the Latin alphabet might be assigned to the integer 65, the character for "B" to 66, and so on. A complete set of characters and corresponding integers is a coded character set. Multiple coded character sets may share the same repertoire, for example ISO-8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different codes. In a coded character set, each code point only represents one character. A character encoding form (CEF) specifies the conversion of a coded character set's integer codes into a set of limited-size integer ''code values'' that facilitate storage in a system that represents numbers in binary form using a fixed number of bits (e.g., virtually any computer system). For example, a system that stores numeric information in 16-bit units would only be able to directly represent integers from 0 to 65,536 in each unit, but larger integers could be represented if more than one 16-bit unit could be used. This is what a CEF accommodates: it defines a way of mapping ''single'' code ''point'' from a range of, say, 0 to 1.4 million, to a series of ''one or more'' code ''values'' from a range of, say, 0 to 65,536. The simplest CEF system is simply to choose large enough units that the values from the coded character set can be encoded directly (one code point to one code value). This works well for coded character sets that fit in 8 bits (as most legacy non-CJK encodings do) and reasonably well for coded character sets that fit in 16 bits (such as early versions of Unicode). However, as the size of the coded character set increases (e.g. modern Unicode requires at least 21 bits/character), this becomes less and less efficient, and it is difficult to adapt existing systems to use larger code values. Therefore, most systems working with later versions Unicode use either UTF-8 , which maps Unicode code points to variable-length sequences of octets, or UTF-16 , which maps Unicode code points to variable-length sequences of 16-bit words. Finally, a character encoding scheme (CES) specifies how the fixed-size integer codes should be mapped into an octet sequence suitable for saving on an octet-based file system or transmitting over an octet-based network. With Unicode in most cases a simple character encoding scheme is used, simply specifying if the bytes for each integer should be in big- Endian or little-endian order (even this isn't needed with UTF-8). However, there are also compound character encoding schemes, which use escape sequences to switch between several simple schemes (such as ISO 2022 ), and compressing schemes, which try to minimise the number of bytes used per code unit (such as SCSU , BOCU , and Punycode ). HISTORY OF CHARACTER ENCODINGS
POPULAR CHARACTER ENCODINGS
CHARACTER CONVERSION TOOLS linux:
SEE ALSO
EXTERNAL LINKS
|
|
|