The Evolution of Character Encoding: ASCII to Unicode and UTF-8

Q: Are Unicode and UTF-8 the same thing?

No. Unicode is a character set: every character gets a unique number, the code point, e.g. '中' is U+4E2D. UTF-8 is an encoding: a rule for turning a sequence of code points into bytes. The same code point can be stored by different schemes—UTF-8 (variable 1–4 bytes), UTF-16 (2 or 4 bytes), UTF-32 (fixed 4 bytes). So Unicode answers 'what number is this character'; UTF-8 answers 'which bytes represent that number'—a spec vs an implementation.

Q: Why is UTF-8 variable-length? How does it know a character's byte count?

UTF-8 self-describes length with leading-byte prefix bits: ASCII (U+0000–U+007F) is 1 byte whose first byte starts with 0; multi-byte characters start with 110/1110/11110 for 2/3/4 bytes, and all continuation bytes start with 10. The decoder only needs the first byte to know the character's total length. This makes UTF-8 backward-compatible with ASCII (pure ASCII bytes are valid UTF-8), covers every code point, and makes errors detectable.

Q: Should databases use utf8 or utf8mb4?

Use utf8mb4 for web projects. In MySQL/MariaDB the legacy utf8 is an alias for utf8mb3, which is at most 3 bytes and cannot store emoji or supplementary-plane characters needing 4 bytes (U+10000+)—those would error or get truncated. utf8mb4 is true full UTF-8 and holds every Unicode character. So use utf8mb4 for tables, connection strings and collations (commonly utf8mb4_unicode_ci/utf8mb4_0900_ai_ci) so emoji and rare glyphs don't break on insert.

2026-09-05·Updated 2026-10-02·3 min read

Starting from the question "is it two bytes per character?"

Many beginners think "a Chinese character = two bytes", which is only roughly true under certain legacy encodings. The truth: the byte count is decided by the encoding, not the character itself—the same '中' takes 2 bytes in GBK, 3 bytes in UTF-8 and 4 bytes in UTF-32. To explain that we need to trace the evolution of encoding.

First leap: ASCII and the 7-bit finite universe

In the 1950s-60s, ASCII defined 128 characters with 7 bits: uppercase/lowercase letters, digits, punctuation and control characters. It was simple and uniform, but only covered English—128 slots cannot hold the world's scripts.

'A' = 65 = 0x41 = 1000001     (1 byte, 7 significant bits)

Second leap: extended encodings and the cradle of mojibake

Every country wanted to express its own writing, so each defined "what the bytes above 0x80 mean": Western Europe ISO-8859-1, Chinese GBK/GB2312, Japanese Shift-JIS… The trouble is they are mutually incompatible—the same byte run 0xE4 B8 AD (UTF-8 for '中') reads as a different character under GBK. Mojibake's essence = the write encoding and the read encoding disagree.

Third leap: Unicode and UTF-8 each doing their job

Unicode (character set): gives every character a globally unique code point. '中' = U+4E2D;
UTF-8 (encoding): encodes those code points into bytes.

UTF-8 variable-length rules

First byte	Bytes	Significant bits	Coverage
`0xxxxxxx`	1	7	U+0000–007F (ASCII)
`110xxxxx`	2	11	U+0080–07FF
`1110xxxx`	3	16	U+0800–FFFF (incl. CJK)
`11110xxx`	4	21	U+10000+ (emoji etc.)

Continuation bytes are always 10xxxxxx. A decoder reads the first byte to know the character's length, and pure ASCII text is valid UTF-8 by construction.

Putting '中' down on paper

code point  U+4E2D = 0x4E2D
binary      0100 1110 0010 1101  (16 bits → needs 3 bytes)
expand with 1110xxxx 10xxxxxx 10xxxxxx:
            1110 0100  10 111000  10 101101
            ───────── ──────── ────────
            E4        B8        AD
→ UTF-8 bytes: 0xE4 0xB8 0xAD

That's where "one Chinese character is 3 bytes in UTF-8" comes from.

UTF-16 and HTML entities: two other routes

UTF-16: Basic Multilingual Plane characters take 2 bytes; supplementary-plane characters take 4 via surrogate pairs. Windows and many systems use it internally, but UTF-8 is more common over the network and in files;
HTML entity 中: writes a character by its hex code point directly in HTML, so it displays correctly regardless of file encoding—useful for referencing symbols in constrained environments.

A checklist to end mojibake for good

[ ] All text files uniformly UTF-8 (no BOM);
[ ] HTTP responses carry Content-Type: text/html; charset=utf-8;
[ ] Database tables use utf8mb4 (not the legacy utf8, which can't hold emoji);
[ ] Agree on UTF-8 for front/back-end transfer; JSON is UTF-8 by default;
[ ] On mojibake, first establish "what encoding wrote these bytes, what is reading them" and convert accordingly—don't guess.

Self-check

Use a text-to-Unicode / Unicode-escape tool to convert '中', an emoji and an ASCII letter into U+xxxx code points and UTF-8 bytes, verifying the variable-length rules above. When you can expand a code point into UTF-8 bytes by hand and explain why pure ASCII is valid UTF-8, you've fully understood encoding.

Reproducible Results

How many bytes does one CJK character really take in UTF-8? Python answers in 30 seconds:

"你".encode("utf-8")   # → b'\xe4\xbd\xa0'
"你".encode("utf-8").hex()   # → 'e4bda0'
len("你")             # → 1 (characters)
len("你".encode())    # → 3 (bytes)

你 is U+4F60, inside the U+0800–U+FFFF range, so UTF-8 encodes it as 3 bytes (E4 BD A0). That is the root of the "varchar length by characters or bytes" debate: 10 such characters cost 10 by characters but 30 as UTF-8 bytes. Enter 你 in the Unicode escape tool to see the same code point escaped.

Frequently Asked Questions

Why does Chinese text get garbled? Where does mojibake come from?

Mojibake is produced by an encoding mismatches: bytes written with one encoding are read with another. For example, the two Chinese characters written in GBK will render as gibberish when decoded as UTF-8, and vice versa, because the same bytes map to different characters in each scheme. The cure: declare and keep the encoding consistent—charset=utf-8 in files/HTTP headers, utf8mb4 in database tables, and both ends agreeing to read/write UTF-8. As long as byte-level encoding matches on both sides, garbling disappears.

Are Unicode and UTF-8 the same thing?

No. Unicode is a character set: every character gets a unique number, the code point, e.g. '中' is U+4E2D. UTF-8 is an encoding: a rule for turning a sequence of code points into bytes. The same code point can be stored by different schemes—UTF-8 (variable 1–4 bytes), UTF-16 (2 or 4 bytes), UTF-32 (fixed 4 bytes). So Unicode answers 'what number is this character'; UTF-8 answers 'which bytes represent that number'—a spec vs an implementation.

Why is UTF-8 variable-length? How does it know a character's byte count?

UTF-8 self-describes length with leading-byte prefix bits: ASCII (U+0000–U+007F) is 1 byte whose first byte starts with 0; multi-byte characters start with 110/1110/11110 for 2/3/4 bytes, and all continuation bytes start with 10. The decoder only needs the first byte to know the character's total length. This makes UTF-8 backward-compatible with ASCII (pure ASCII bytes are valid UTF-8), covers every code point, and makes errors detectable.

Should databases use utf8 or utf8mb4?

Use utf8mb4 for web projects. In MySQL/MariaDB the legacy utf8 is an alias for utf8mb3, which is at most 3 bytes and cannot store emoji or supplementary-plane characters needing 4 bytes (U+10000+)—those would error or get truncated. utf8mb4 is true full UTF-8 and holds every Unicode character. So use utf8mb4 for tables, connection strings and collations (commonly utf8mb4_unicode_ci/utf8mb4_0900_ai_ci) so emoji and rare glyphs don't break on insert.

← Back to Blog