Information AboutUtf-8 |
| CATEGORIES ABOUT UTF-8 | |
| unicode | |
| utf-08 | |
| character sets | |
| encodings | |
| character encoding | |
| unicode transformation formats | |
|
UTF-8 encodes each character in one to four Octet s (8-bit Byte s): #One byte is needed to encode the 128 US-ASCII characters (Unicode range to U+007F). #Two bytes are needed for Latin letters with Diacritic s and for characters from Greek , Cyrillic , Armenian , Hebrew , Arabic , Syriac and Thaana alphabets (Unicode range U+0080 to U+07FF). #Three bytes are needed for the rest of the Basic Multilingual Plane (which contains virtually all characters in common use). #Four bytes are needed for characters in the Other Planes Of Unicode , which are rarely used in practice. Four bytes may seem like a lot for one character ( Code Point ). However, code points outside the Basic Multilingual Plane are generally very rare. Furthermore, UTF-16 (the main alternative to UTF-8) also needs four bytes for these code points. Whether UTF-8 or UTF-16 is more efficient depends on the range of code points being used. However, the differences between different encoding schemes can become negligible with the use of traditional compression systems like DEFLATE . For short items of text where traditional algorithms do not perform well and size is important, the Standard Compression Scheme For Unicode could be considered instead. The Internet Engineering Task Force (IETF) requires all Internet Protocol s to identify the Encoding used for character data with UTF-8 as at least one supported encoding.RFC 2277 section 3.1 The Internet Mail Consortium (IMC) recommends[http://www.imc.org/mail-i18n.html IMC]. that all email programs be able to display and create mail using UTF-8. HISTORY By early 1992 a search was on for a good byte-stream encoding of multi-byte character sets. The draft ISO 10646 standard contained a non-required annex called UTF that provided a byte-stream encoding of its 32-bit characters. This encoding was not satisfactory on performance grounds, but did introduce the notion that bytes in the ASCII range of 0–127 represent themselves in UTF, thereby providing backward compatibility. In July 1992 the X/Open committee XoJIG was looking for a better encoding. Dave Prosser of Unix System Laboratories submitted a proposal for one that had faster implementation characteristics and introduced the improvement that 7-bit ASCII characters would ''only'' represent themselves; all multibyte sequences would include only 8-bit characters, i.e. those where the high bit was set. In August 1992 this proposal was circulated by an IBM X/Open representative to interested parties. Ken Thompson of the Plan 9 Operating System group at Bell Laboratories then made a crucial modification to the encoding, to allow it to be self-synchronizing, meaning that it was not necessary to read from the beginning of the string in order to find character boundaries. Thompson's design was outlined on September 2 , 1992 , on a Placemat in a New Jersey Diner with Rob Pike . The following days, Pike and Thompson implemented it and updated Plan 9 to use it throughout, and then communicated their success back to X/Open. UTF-8 was first officially presented at the USENIX conference in San Diego January 25 – 29 1993 . Microsoft's specification for CAB (MS Cabinet) from 1996 allows for UTF-8 encoded strings everywhere specifically (though this was before UTF-8 was actually formally standardised), but the encoder never actually implemented it. DESCRIPTION There are several current, slightly different definitions of UTF-8 in various standards documents:
They supersede the definitions given in the following obsolete works:
They are all the same in their general mechanics with the main differences being on issues such as allowed range of code point values and safe handling of invalid input. The bits of a Unicode character are divided into several groups which are then divided among the lower bit positions inside the UTF-8 bytes. A character whose code point is below U+0080 is encoded with a single byte that contains its code point: these correspond exactly to the 128 characters of 7- Bit ASCII . In other cases, up to four bytes are required. The Most Significant Bit of these bytes is 1, to prevent confusion with 7-bit ASCII characters and therefore keep standard byte-oriented string processing safe. For example, the character aleph (א), which is Unicode U+05D0, is encoded into UTF-8 in this way:
Width by first byte: So the first 128 characters (US-ASCII) need one byte. The next 1920 characters need two bytes to encode. This includes Latin Alphabet Characters With Diacritics , Greek , Cyrillic , Coptic , Armenian , Hebrew , and Arabic characters. The rest of the BMP characters use three bytes, and additional characters are encoded in four bytes. By continuing the pattern given above it is possible to deal with much larger numbers. The original specification allowed for sequences of up to six bytes covering numbers up to 31 bits (the original limit of the Universal Character Set ). However, UTF-8 was restricted by RFC 3629 to use only the area covered by the formal Unicode definition, U+0000 to U+10FFFF, in November 2003. With these restrictions, the following byte values never appear in a legal UTF-8 sequence: While the two categories labeled "Restricted by RFC" above were technically allowed by earlier UTF-8 specifications, no characters were ever assigned to the code points they represent, so they should never have appeared in UTF-8-encoded text. The design of the algorithm has some similarities with Huffman Coding . UTF-8 DERIVATIONS Windows Although not part of the standard, many Windows programs (including Windows Notepad ) use the byte sequence EF BB BF at the beginning of a file to indicate that the file is encoded using UTF-8. This is the Byte Order Mark U+FEFF encoded in UTF-8, which appears as the ISO-8859-1 characters "" in most Text Editor s and Web Browser s not prepared to handle UTF-8. Java In normal usage, the Java Programming Language supports standard UTF-8 when reading and writing strings through and . However, Java also supports a non-standard variant of UTF-8 called for object Serialization , for the Java Native Interface , and for embedding constants in Class Files . There are two differences between modified and standard UTF-8. The first difference is that the Null Character (U+0000) is encoded with two bytes instead of one, specifically as 0xc0 0x80. This ensures that there are no embedded nulls in the encoded string, to address the concern that if the encoded string is processed in a language such as C , a null byte would signal the end of the string, and cause premature truncation. The second difference is in the way characters outside the Basic Multilingual Plane are encoded. In standard UTF-8 these characters are encoded using the four-byte format above. In modified UTF-8 these characters are first represented as surrogate pairs (as in UTF-16), and then the surrogate pairs are encoded individually in sequence as in CESU-8 , taking up 6 bytes in total. Each Java character represents a 16 bit value. This aspect of the language predates the supplementary planes of Unicode; however, it is important for performance as well as backwards compatibility, and is unlikely to change. Because modified UTF-8 is not UTF-8, one needs to be very careful to '''avoid mislabelling''' data in modified UTF-8 as UTF-8 when interchanging information over the Internet . Mac OS X The Mac OS X Operating System uses canonically decomposed Unicode, encoded using UTF-8 for file names in the Filesystem . This is sometimes referred to as UTF-8-MAC. In canonically decomposed Unicode, the use of precomposed characters is forbidden and Combining Diacritics must be used to replace them. This makes sorting far simpler but can be confusing for software built around the assumption that precomposed characters are the norm and combining diacritics only used to form unusual combinations. This is an example of the NFD variant of Unicode Normalization —most other platforms, including Windows and Linux , use the NFC form of Unicode normalization, which is also used by W3C standards, so NFD data must typically be converted to NFC for use on other platforms or the Web. This is discussed in Apple Q&A 1173 . RATIONALE BEHIND UTF-8'S DESIGN As a consequence of the design of UTF-8, the following properties of multi-byte sequences hold:
UTF-8 was designed to satisfy these properties in order to guarantee that no byte sequence of one character is contained within a longer byte sequence of another character. This ensures that byte-wise sub-string matching can be applied to search for words or phrases within a text; some older variable-length 8-bit encodings (such as Shift-JIS ) did not have this property and thus made string-matching algorithms rather complicated. Although this property adds Redundancy to UTF-8–encoded text, the advantages outweigh this concern; besides, Data Compression is not one of Unicode's aims and must be considered independently. This also means that if one or more complete bytes are lost due to error or corruption, one can resynchronize at the beginning of the next character and thus limit the damage. Also due to the design of the byte sequences, if a sequence of bytes supposed to represent text validates as UTF-8 then it is fairly safe to assume it is UTF-8. The chance of a random sequence of bytes being valid UTF-8 and not pure ASCII is 3.1% for a 2 byte sequence, 2.0% for a 3 byte sequence and even lower for longer sequences. While natural languages encoded in traditional encodings are far from random byte sequences, they are also unlikely to produce byte sequences that would pass a UTF-8 validity test and then be misinterpreted (obviously pure ASCII text would pass a UTF-8 validity test, but provided the legacy encodings under consideration are also ASCII-based, this is not a problem). For example, for ISO-8859-1 text to be misrecognized as UTF-8, the only non-ASCII characters in it would have to be in sequences starting with either an accented letter or the multiplication symbol and ending with a symbol. The bit patterns can be used to identify UTF-8 characters. If the byte's first hex code begins with 0–7, it is an ASCII character. If it begins with C or D, it is an 11 bit character (expressed in two bytes). If it begins with E, it is 16 bit (expressed in 3 bytes), and if it begins with F, it is 21 bits (expressed in 4 bytes). 8 through B cannot be first hex codes, but all ''following'' bytes must begin with a hex code between 8 through B. Thus, at a glance, it can be seen that "0xA9" is not a valid UTF-8 character, but that "0x54" or "0xE3 0xB4 0xB1" ''are'' valid UTF-8 characters. OVERLONG FORMS, INVALID INPUT, AND SECURITY CONSIDERATIONS The exact response required of a UTF-8 decoder on invalid input is not uniformly defined by the standards. In general, there are several ways a UTF-8 decoder might behave in the event of an invalid byte sequence: # Insert a replacement character (usually '?' or '￭�' (U+FFFD)). # Ignore the bytes. # Interpret each byte according to a legacy encoding (often ISO-8859-1 or CP1252 ). # Not notice and decode as if the bytes were some similar bit of UTF-8. # Stop decoding and report an error (possibly giving the caller the option to continue). It is possible for a decoder to behave in different ways for different types of invalid input. Overlong forms are one of the most troublesome types of UTF-8 data. The current RFC says they must not be decoded but older specifications for UTF-8 only gave a warning and many simpler decoders will happily decode them. Overlong forms have been used to bypass security validations in high profile products including Microsoft's IIS web server. Therefore, great care must be taken to avoid security issues if validation is performed before conversion from UTF-8, and it is generally much simpler to handle overlong forms before any input validation is done. Another common problem is decoders that do not check that the trailing bytes are really trailing bytes. This will cause more characters to be lost than necessary if some bytes are lost or corrupted. To maintain security in the case of invalid input, there are a few options. The first is to decode the UTF-8 ''before'' doing any input validation checks. The second is to use a decoder that, in the event of invalid input either returns an error or text that the application knows to be harmless. A third possibility is to not decode the UTF-8 at all, this is quite practical if the system only treats some ASCII characters (like slash and NUL) specially, and treats all other bytes as identifiers or other data but requires care to avoid passing invalid UTF-8 to other code (such as third party libraries or an operating system) that cannot safely handle it. ADVANTAGES AND DISADVANTAGES String length In general, it is not possible to determine from the number of code points in a Unicode string how much space it needs to be displayed, or where on a screen the cursor should be placed in a text buffer after displaying a string; combining characters, double width characters, proportional fonts, non-printing characters and right-to-left characters all contribute to this. So while the number of octets in an UTF-8 string is related in a more complex way to the number of code points than for UTF-32, it is very rare to encounter a situation where this makes a difference in practice. General Advantages
Disadvantages
Compared to single-byte legacy encodings Advantages
Disadvantages
Compared to multi-byte legacy encodings Advantages
Disadvantages
Compared to UTF-7 Advantages
Disadvantages
Compared to UTF-16 Advantages
Disadvantages
NOTES SEE ALSO
EXTERNAL LINKS
|
|
|