Unicode In 5 Minutes

From Second Life Wiki
Jump to navigation Jump to search

Writing

Unicode is a standard for digital processing of written characters and text. By enabling the exchange of text data internationally, it is a foundation for global software.

Unicode-1-Writing.png

Characters

Unicode encodes characters, the smallest components of written language that have semantic value. It assigns each one a unique code point, or number. Code points are expressed as U+n where n is the number in hexadecimal.

Unicode-2-Characters.png

Variations

Unicode does not encode font or stylistic differences. For example, most forms of the lower case 'a' are the same code point, U+0061. However, for compatibility with other systems, variants are sometimes given independent code points.

Unicode-3-Variations.png

Allocation

The full Unicode codespace supports over a million code points, of which about 100k are currently assigned. The vast majority of characters used in modern languages are allocated within the first 65,536 code points, called the Basic Multilingual Plane. Notably, the first 128 code points are the same as ASCII.

Unicode-4-Allocation.png

Encoding

For Unicode text to be transmitted or stored, it must be encoded into a sequence of bytes. UTF-8 and UTF-16 are common encodings that can handle the entire range of characters. Both are variable length encodings where some characters take more bytes than others.

ASCII, LATIN1 and SJIS are common encodings that are outside of Unicode. They cannot encode all characters. ASCII has the interesting property that all ASCII sequences are valid UTF-8 sequences and encode the same characters.

Unicode-5-Encoding.png

Normalization

For a large variety of reasons, some texts can be represented in Unicode with more than one character sequence. Depending on the intended processing, text may need to be normalized into a particular style of sequence. For example, people expect the word "file" to sort and compare the same no matter if the first two letters are represented as two characters, or as a ligature. Unicode defines normal forms to convert text into a regularized form for text processing.

Unicode-6-Normalization.png

Fun

The Unicode Standard is a fascinating document with many interesting write-ups about various writing systems, and pictures of fantastic characters. These are a few of my favorites.

Unicode-7-Fun.png

- Zero Linden

Unicode-Khomut-red.png