The Evolution of Character Encoding: ASCII to Unicode and UTF-8

Starting from the question "is it two bytes per character?"

Many beginners think "a Chinese character = two bytes", which is only roughly true under certain legacy encodings. The truth: the byte count is decided by the encoding, not the character itself—the same '中' takes 2 bytes in GBK, 3 bytes in UTF-8 and 4 bytes in UTF-32. To explain that we need to trace the evolution of encoding.

First leap: ASCII and the 7-bit finite universe

In the 1950s-60s, ASCII defined 128 characters with 7 bits: uppercase/lowercase letters, digits, punctuation and control characters. It was simple and uniform, but only covered English—128 slots cannot hold the world's scripts.

'A' = 65 = 0x41 = 1000001     (1 byte, 7 significant bits)

Second leap: extended encodings and the cradle of mojibake

Every country wanted to express its own writing, so each defined "what the bytes above 0x80 mean": Western Europe ISO-8859-1, Chinese GBK/GB2312, Japanese Shift-JIS… The trouble is they are mutually incompatible—the same byte run 0xE4 B8 AD (UTF-8 for '中') reads as a different character under GBK. Mojibake's essence = the write encoding and the read encoding disagree.

Third leap: Unicode and UTF-8 each doing their job

• Unicode (character set): gives every character a globally unique code point. '中' = U+4E2D;
• UTF-8 (encoding): encodes those code points into bytes.

UTF-8 variable-length rules

First byte	Bytes	Significant bits	Coverage
0xxxxxxx	1	7	U+0000–007F (ASCII)
110xxxxx	2	11	U+0080–07FF
1110xxxx	3	16	U+0800–FFFF (incl. CJK)
11110xxx	4	21	U+10000+ (emoji etc.)

Continuation bytes are always 10xxxxxx. A decoder reads the first byte to know the character's length, and pure ASCII text is valid UTF-8 by construction.

Putting '中' down on paper

code point  U+4E2D = 0x4E2D
binary      0100 1110 0010 1101  (16 bits → needs 3 bytes)
expand with 1110xxxx 10xxxxxx 10xxxxxx:
            1110 0100  10 111000  10 101101
            ───────── ──────── ────────
            E4        B8        AD
→ UTF-8 bytes: 0xE4 0xB8 0xAD

That's where "one Chinese character is 3 bytes in UTF-8" comes from.

UTF-16 and HTML entities: two other routes

• UTF-16: Basic Multilingual Plane characters take 2 bytes; supplementary-plane characters take 4 via surrogate pairs. Windows and many systems use it internally, but UTF-8 is more common over the network and in files;
• HTML entity 中: writes a character by its hex code point directly in HTML, so it displays correctly regardless of file encoding—useful for referencing symbols in constrained environments.

A checklist to end mojibake for good

• [ ] All text files uniformly UTF-8 (no BOM);
• [ ] HTTP responses carry Content-Type: text/html; charset=utf-8;
• [ ] Database tables use utf8mb4 (not the legacy utf8, which can't hold emoji);
• [ ] Agree on UTF-8 for front/back-end transfer; JSON is UTF-8 by default;
• [ ] On mojibake, first establish "what encoding wrote these bytes, what is reading them" and convert accordingly—don't guess.

Self-check

Use a text-to-Unicode / Unicode-escape tool to convert '中', an emoji and an ASCII letter into U+xxxx code points and UTF-8 bytes, verifying the variable-length rules above. When you can expand a code point into UTF-8 bytes by hand and explain why pure ASCII is valid UTF-8, you've fully understood encoding.