The Evolution of Character Encoding: ASCII to Unicode and UTF-8

The answer first: byte counts come from the encoding, not the character

"A Chinese character takes two bytes" is only roughly true under legacy encodings such as GBK. The same '中' takes 2 bytes in GBK, 3 in UTF-8 and 4 in UTF-32. A character has no inherent byte size; its encoding does.

ASCII: 7 bits and 128 characters

ASCII defined 128 characters with 7 bits: uppercase and lowercase letters, digits, punctuation and control codes. It served English and could not hold other scripts — every compatibility problem that follows starts here.

'A' = 65 = 0x41 = 1000001     (1 byte, 7 significant bits)

Extended encodings: the breeding ground for mojibake

Each region defined what bytes above 0x80 meant: Western Europe chose ISO-8859-1, Chinese GBK/GB2312, Japanese Shift-JIS. These schemes are mutually incompatible: the byte run 0xE4 B8 AD (UTF-8 for '中') reads as a different character under GBK. Mojibake means the writing encoding and the reading encoding disagree; the character itself was never "broken".

Diagnose in this order:

  1. Get the raw bytes (xxd, hexdump, or Python's .encode()) and establish which sequence they actually are;
  2. Decode explicitly with the matching encoding instead of letting an editor or browser guess;
  3. Record the writer's encoding in the interface documentation so the next person does not have to guess.

Unicode and UTF-8: a character set versus an encoding

Unicode assigns every character a globally unique code point — '中' is U+4E2D — while UTF-8 specifies how those code points become bytes. The two are a specification and an implementation, not synonyms.

UTF-8 variable-length rules

First byte Bytes Significant bits Coverage
0xxxxxxx 1 7 U+0000–007F (ASCII)
110xxxxx 2 11 U+0080–07FF
1110xxxx 3 16 U+0800–FFFF (incl. CJK)
11110xxx 4 21 U+10000+ (emoji etc.)

Continuation bytes are always 10xxxxxx. The first byte alone tells a decoder how many bytes follow, and pure ASCII is valid UTF-8 by construction.

Hand-computing '中': U+4E2D → E4 B8 AD

code point  U+4E2D = 0x4E2D
binary      0100 1110 0010 1101  (16 bits → needs 3 bytes)
expand with 1110xxxx 10xxxxxx 10xxxxxx:
            1110 0100  10 111000  10 101101
            ───────── ──────── ────────
            E4        B8        AD
→ UTF-8 bytes: 0xE4 0xB8 0xAD

Sixteen bits fit exactly into the 16 significant bits of a three-byte sequence, and filling the template yields E4 B8 AD. That is the origin of "one Chinese character is 3 bytes in UTF-8".

UTF-16 and HTML entities: two other routes

  • UTF-16: two bytes for Basic Multilingual Plane characters, four via surrogate pairs for supplementary planes. Windows and many systems use it internally; UTF-8 dominates on the network.
  • HTML entity 中: writes the hex code point directly, so it displays regardless of file encoding; useful in constrained environments, at the cost of readability.

A checklist against mojibake

  • [ ] All text files UTF-8, without a BOM;
  • [ ] HTTP responses carry Content-Type: text/html; charset=utf-8;
  • [ ] Tables, connection strings and collations all use utf8mb4;
  • [ ] Both ends agree on UTF-8 for transport, which JSON already defaults to;
  • [ ] On mojibake, first establish who wrote these bytes and who is reading them, then convert — never guess.

30-second verification

"你".encode("utf-8")   # → b'\xe4\xbd\xa0'
"你".encode("utf-8").hex()   # → 'e4bda0'
len("你")             # → 1 (characters)
len("你".encode())    # → 3 (bytes)

你 is U+4F60, inside U+0800–U+FFFF, so UTF-8 spends 3 bytes on it (E4 BD A0). This is also the root of the "varchar by characters or bytes" argument: ten such characters cost 10 by count and 30 as UTF-8 bytes. Enter 你 in the text to Unicode tool to see the same code point escaped.

Frequently Asked Questions

Why does Chinese text get garbled? Where does mojibake come from?

Mojibake is produced by an encoding mismatches: bytes written with one encoding are read with another. For example, the two Chinese characters written in GBK will render as gibberish when decoded as UTF-8, and vice versa, because the same bytes map to different characters in each scheme. The cure: declare and keep the encoding consistent—charset=utf-8 in files/HTTP headers, utf8mb4 in database tables, and both ends agreeing to read/write UTF-8. As long as byte-level encoding matches on both sides, garbling disappears.

Are Unicode and UTF-8 the same thing?

No. Unicode is a character set: every character gets a unique number, the code point, e.g. '中' is U+4E2D. UTF-8 is an encoding: a rule for turning a sequence of code points into bytes. The same code point can be stored by different schemes—UTF-8 (variable 1–4 bytes), UTF-16 (2 or 4 bytes), UTF-32 (fixed 4 bytes). So Unicode answers 'what number is this character'; UTF-8 answers 'which bytes represent that number'—a spec vs an implementation.

Why is UTF-8 variable-length? How does it know a character's byte count?

UTF-8 self-describes length with leading-byte prefix bits: ASCII (U+0000–U+007F) is 1 byte whose first byte starts with 0; multi-byte characters start with 110/1110/11110 for 2/3/4 bytes, and all continuation bytes start with 10. The decoder only needs the first byte to know the character's total length. This makes UTF-8 backward-compatible with ASCII (pure ASCII bytes are valid UTF-8), covers every code point, and makes errors detectable.

Should databases use utf8 or utf8mb4?

Use utf8mb4 for web projects. In MySQL/MariaDB the legacy utf8 is an alias for utf8mb3, which is at most 3 bytes and cannot store emoji or supplementary-plane characters needing 4 bytes (U+10000+)—those would error or get truncated. utf8mb4 is true full UTF-8 and holds every Unicode character. So use utf8mb4 for tables, connection strings and collations (commonly utf8mb4_unicode_ci/utf8mb4_0900_ai_ci) so emoji and rare glyphs don't break on insert.

← Back to Blog