Handy online tools for developers

/Ctrl+K
← Back to Blog

URL Encoding/Decoding Guide: Percent-Encoding, Query Strings & Common Pitfalls

What is URL Encoding?

URL encoding (also known as percent-encoding) is the process of converting characters that are not allowed in a URL into a % followed by two hexadecimal digits. Based on RFC 3986, it ensures URLs can be safely transmitted across different systems and network protocols.

URLs only allow a subset of the ASCII character set:

  • Unreserved characters: A-Z a-z 0-9 - _ . ~
  • Reserved characters: : / ? # [ ] @ ! $ & ' ( ) * + , ; =
  • All other characters (including Chinese, spaces, special symbols) must be encoded

Reserved and Unreserved Characters

Unreserved Characters

These characters have no special meaning in URLs and can be used directly without encoding:

Character Description
A-Z a-z 0-9 Letters and digits
- Hyphen
_ Underscore
. Period
~ Tilde

Reserved Characters

Reserved characters serve specific syntactic purposes in URLs, split into two groups:

General delimiters: : / ? # [ ] @

Sub-delimiters: ! $ & ' ( ) * + , ; =

Whether a reserved character needs encoding depends on its position in the URL. For example, / acts as a hierarchy separator in a path, but if it appears as part of a query parameter value, it must be encoded as %2F.

Query String Encoding

The query string is the part of the URL after ?, consisting of &-separated key-value pairs. Encoding rules:

Encoding Rules

  • Spaces become + (application/x-www-form-urlencoded) or %20 (standard percent-encoding)
  • &, =, + as parameter values must be encoded as %26, %3D, %2B
  • Non-ASCII characters are first converted to UTF-8 byte sequences, then each byte is encoded

Construction Example

To pass the parameter name=John & Jane:

?name=John+%26+Jane

Per-Language Construction

JavaScript:

const params = new URLSearchParams({ name: 'John & Jane' });
const url = `/api?${params.toString()}`;

Python:

from urllib.parse import urlencode
query = urlencode({'name': 'John & Jane'})

Go:

v := url.Values{}
v.Set("name", "John & Jane")
query := v.Encode()

Never manually concatenate query strings. Always use standard library encoding functions.

International URLs

Non-ASCII Character Encoding

The URL standard requires ASCII characters only. For Chinese, Japanese, Korean, emoji, etc.:

  1. Convert the character to a UTF-8 byte sequence
  2. Percent-encode each byte

For example, café encodes to caf%C3%A9.

Punycode and Internationalized Domain Names (IDN)

The domain name part uses Punycode (RFC 3492). For example, münchen.de becomes xn--mnchen-3ya.de. Modern browsers display the original characters in the address bar, but the actual request uses Punycode.

Unicode in Paths

Path segments use percent-encoding. Modern browsers and HTTP clients handle this automatically, but server logs show the encoded form.

Common Encoding Bugs

1. Double Encoding

Encoding an already-encoded string turns % into %25:

Original:   café
First pass: caf%C3%A9
Second pass: caf%25C3%25A9  ❌

Cause: The framework auto-encodes, then you manually call encodeURIComponent again.

2. Encoding/Decoding Mismatch

Frontend uses encodeURI, backend uses a decoder with a non-UTF-8 charset, causing garbled text.

3. Space Handling Differences

  • encodeURIComponent(' ') -> %20
  • application/x-www-form-urlencoded -> +

For URL paths, spaces should be %20; for form-submitted query strings, spaces are +. Mixing them causes server-side parsing errors.

4. Unencoded Reserved Characters

Directly concatenating user input into a URL path. If input contains ?, #, /, it breaks the URL structure and can even cause open redirect vulnerabilities.

5. Hash Fragments Not Sent to Server

The part after # (fragment) is never sent to the server. Putting sensitive parameters in the hash is a security misconception.

encodeURI vs encodeURIComponent

Function Purpose Characters Not Encoded
encodeURI Encode a full URL A-Za-z0-9;-._~:/?#[]@!$&'()*+,;=
encodeURIComponent Encode a URL component (param value) A-Za-z0-9;-._!~'()*

Best Practices

  1. Always use standard libraries: URLSearchParams, url.parse, urllib.parse
  2. Distinguish encoding contexts: Use encodeURI for paths, encodeURIComponent for parameter values
  3. Standardize on UTF-8: Agree on UTF-8 between frontend and backend
  4. Don't encode twice: Check if the framework already auto-encodes
  5. Validate user input: Prevent reserved character injection
  6. Use HTTPS: Encoded URLs still need transport-layer encryption
← Back to Blog