What is Unicode and how does it works?

#

What is Unicode: A Universal Character Set

#

Unicode is a standard that assigns a unique number to every character, no matter the platform, program, or language. It’s like a global dictionary for characters.

Why is it important?

Before Unicode, different systems used different character sets, leading to compatibility issues. For instance, a document created on one system might appear garbled when opened on another. Unicode solved this problem by providing a unified standard.

Key points about Unicode:

• Comprehensive: It covers almost all writing systems in the world, including Latin, Cyrillic, Greek, Arabic, Chinese, Japanese, Korean, and many more.
• Unique identifier: Each character has a unique code point, a number that represents it.
• Encoding formats: Unicode characters are often stored in different encoding formats like UTF-8, UTF-16, and UTF-32.

In essence, Unicode is the foundation for consistent text representation across different systems and applications.

UTF-8 can represent how many characters?

#

UTF-8 doesn’t have a fixed number of characters. The key point to understand is that UTF-8 is an encoding scheme, not a character set. It’s a way to represent characters from the Unicode standard in a binary format.

How UTF-8 Works

#

• Variable-length encoding: UTF-8 uses 1 to 4 bytes to represent a character. This flexibility allows it to encode a vast range of characters, from basic ASCII to complex symbols.
• Compatibility: It’s backward compatible with ASCII, meaning ASCII characters are represented by a single byte.

Why There’s No Fixed Number

#

The number of characters in a UTF-8 encoded text depends entirely on the specific characters used. For example:

• A text containing only ASCII characters will use one byte per character.
• A text with a mix of Latin characters and emojis might use a combination of one, two, or three-byte sequences.

In Summary

#

Instead of focusing on the total number of characters in UTF-8, it’s more practical to consider:

• The range of characters: UTF-8 can represent over 1 million characters.
• The byte size of individual characters: This varies from one to four bytes.
• The specific text: The actual character count depends on the content.

Can you explain UTF-8 Encoding with and example?

#

The Sentence:

#

Let’s take the sentence: “नमस्ते, Hello! How are you? 😊”

Breaking it Down:

#

• नमस्ते: This is a Hindi word. In UTF-8, it will likely be represented using multiple bytes for each character, as Hindi characters often require more than one byte.
• Hello!: These English characters will be represented using a single byte each, as they fall within the ASCII range.
• How are you?: Similar to “Hello!”, these English words will use one byte per character.
• 😊: This emoji is a special character and will likely be encoded using multiple bytes.

Encoding Process:

#

1. Each character is converted into its corresponding Unicode code point.
2. The Unicode code point is then converted into a UTF-8 byte sequence based on specific encoding rules.

Example (Simplified):

#

• न: Might be encoded as a three-byte sequence like 11100110 10101001 10110010
• म: Might be encoded as a three-byte sequence like 11100110 10101010 10110101
• स: Might be encoded as a three-byte sequence like 11100110 10101100 10110111
• ्त: Might be encoded as a three-byte sequence like 11100110 10101101 10111000
• े: Might be encoded as a three-byte sequence like 11100110 10101110 10111001
• , will be encoded as a single byte 00101100
• H: will be encoded as a single byte 01001000
• e: will be encoded as a single byte 01100101
• l: will be encoded as a single byte 01101100
• l: will be encoded as a single byte 01101100
• o: will be encoded as a single byte 01101111
• !: will be encoded as a single byte 00100001
• : will be encoded as a single byte 00111010
• H: will be encoded as a single byte 01001000
• o: will be encoded as a single byte 01101111
• w: will be encoded as a single byte 01110111
• … (similar encoding for the rest of the sentence)
• 😊: Might be encoded as a four-byte sequence (specific bytes will depend on the emoji’s Unicode code point)

Note: This is a simplified explanation and the actual encoding might be different. The purpose is to illustrate the concept of variable-length encoding in UTF-8.

Why UTF-8 is called 8 and not 4 or 10?

#

UTF-8 stands for Unicode Transformation Format - 8 bit.

The “8” in UTF-8 refers to the fact that it uses 8-bit bytes as the fundamental unit for encoding characters. This means each individual byte in a UTF-8 sequence consists of 8 bits.

The Misconception

#

The confusion might arise because:

• Variable-length encoding: UTF-8 uses a variable number of bytes (1, 2, 3, or 4) to represent characters. This might give the impression it’s dealing with 4-byte units.
• Other encodings: There are other Unicode encodings like UTF-16 and UTF-32 which use 16-bit and 32-bit units respectively.

Key Point

#

While UTF-8 can use up to 4 bytes to represent a single character, the underlying unit it operates on is still an 8-bit byte. This is why it’s called UTF-8.

To summarize:

• UTF-8 uses 8-bit bytes as its building blocks.
• It’s a variable-length encoding, meaning characters can be represented using 1 to 4 bytes.
• The “8” in UTF-8 refers to the byte size, not the character size.

What are different unicoding systems

#

While UTF-8 has become the dominant character encoding, there are several others that have been used historically or are still in use for specific purposes. Let’s explore some of them:

Legacy Encoding Systems

#

• ASCII (American Standard Code for Information Interchange): This was one of the earliest encoding systems, representing 128 characters (including uppercase and lowercase letters, numbers, punctuation, and control characters) using 7 bits.
• ISO-8859: A family of character encodings based on the Latin alphabet, each covering different character sets for specific regions (e.g., Western Europe, Latin America, Cyrillic).
• EBCDIC (Extended Binary Coded Decimal Interchange Code): Primarily used in IBM mainframe computers, EBCDIC differs significantly from ASCII in character representation.

Unicode Encodings (Apart from UTF-8)

#

• UTF-16: Uses 16-bit code units to represent characters. It’s more efficient for languages with a large character set, but can be less efficient for English text compared to UTF-8.
• UTF-32: Uses 32-bit code units for each character, providing a fixed-width representation. While simple to implement, it can be inefficient in terms of storage space.

Other Notable Encodings

#

• GB2312, GBK, GB18030: These are Chinese character encoding standards used in mainland China.
• Shift-JIS, EUC-JP: Japanese character encodings.
• Big5: Traditional Chinese character encoding.

Choosing the Right Encoding

#

The best encoding for a specific task depends on several factors:

• Character set: The range of characters you need to represent.
• Efficiency: The desired balance between storage space and processing speed.
• Compatibility: The systems and applications that will handle the encoded data.

UTF-8 has gained popularity due to its flexibility, compatibility, and efficiency in handling a wide range of characters. However, understanding other encoding systems is essential for working with legacy data or specific character sets.

How computer/system/human can know from the sequence of words that next character is 1 byte or 2 bytes or 3 bytes or 4 bytes long?

#

How UTF-8 Determines Character Length

#

UTF-8 employs a clever system to indicate the length of a character based on the pattern of bits in the first byte.

The Byte Pattern:

#

• Single-byte characters: If the first byte is in the range 0x00 to 0x7F (00000000 to 01111111), it represents a single-byte character (ASCII).
• Multi-byte characters: If the first byte starts with 0b110 (110xxxxx), it’s the beginning of a two-byte character.
• If the first byte starts with 0b1110 (1110xxxx), it’s the beginning of a three-byte character.
• If the first byte starts with 0b11110 (11110xxx), it’s the beginning of a four-byte character.

Continuation Bytes:

#

Following bytes in a multi-byte character always start with 0b10 (10xxxxxx). This pattern helps identify the continuation bytes of a character.

Example:

#

Let’s say we have the following byte sequence: 11000010 10111101 11100110 10011100 10101010 10111101

• The first byte 11000010 starts with 110, indicating a two-byte character.
• The second byte 10111101 starts with 10, confirming it’s a continuation byte for the first character.
• The third byte 11100110 starts with 1110, indicating a three-byte character.
• The fourth, fifth, and sixth bytes start with 10, confirming they are continuation bytes for the third character.

Summary:

#

By examining the first byte of a character sequence, a computer can determine the number of bytes required to represent the character. The subsequent bytes, with their specific pattern, confirm the character’s length. This mechanism ensures accurate decoding of UTF-8 text.

UTF-8, UTF 16, UTF 32 all can represent same number of characters. Then why to chose UTF 32?

#

The key difference lies in their efficiency and specific use cases.

When to Consider UTF-32:

#

While UTF-32 might seem like the straightforward choice due to its fixed-width nature, it’s generally not the preferred encoding for most applications. However, there are specific scenarios where it can be advantageous:

1. Simplicity and Performance: If you prioritize simplicity in character handling and have ample storage space, UTF-32 can be attractive. Basic text manipulation operations might be slightly faster due to the fixed-width nature.
2. Internal Representations: Some software applications might use UTF-32 internally for processing text, even if the external data is stored in a different encoding. This can simplify character handling within the application.
3. Specific Use Cases: In rare cases where every character needs to be accessed independently without considering byte boundaries or character lengths, UTF-32 might be suitable.

Why UTF-8 is Often Preferred:

#

• Efficiency: UTF-8 is generally more space-efficient, especially for text containing primarily ASCII characters.
• Wide Compatibility: It’s supported by almost all systems and applications.
• Flexibility: Handles a wide range of character sets efficiently.

When to Consider UTF-16:

#

• Legacy Systems: Some older systems might still use UTF-16, so compatibility might be a factor.
• Balancing Efficiency and Complexity: It offers a balance between UTF-8 and UTF-32 in terms of space efficiency and complexity.

In conclusion, while UTF-32 can be a viable option in specific circumstances, UTF-8 is generally the preferred encoding due to its efficiency, compatibility, and flexibility.

Author
Dr Hari Thapliyaal
dasarpai.com
linkedin.com/in/harithapliyal