URL Encoding/Decoding Guide: Percent-Encoding, Query Strings & Common Pitfalls

What is URL Encoding?

URL encoding (percent-encoding) converts characters that cannot appear literally in a URL into % followed by two hexadecimal digits. Rooted in RFC 3986, it lets URLs travel safely across systems and protocols.

A URL may only use a subset of ASCII:

• Unreserved: A-Z a-z 0-9 - _ . ~
• Reserved: : / ? # [ ] @ ! $ & ' ( ) * + , ; =
• Everything else (including Chinese, spaces, and symbols) must be encoded

Start with a map, because every rule below answers one question: can this character appear literally in this part of the URL?

https://api.example.com:443/v1/search?q=caf%C3%A9&page=2#results
└─┬──┘ └──────┬──────┘└┬┘└────┬────┘ └────┬────┘ └───┬───┘
scheme     host      port    path       query      fragment

Reserved and Unreserved Characters

Unreserved

Unreserved characters may appear literally anywhere. Encoding them is not wrong (it decodes to the same value) but it needlessly inflates the URL.

Reserved

General delimiters (separating the major parts): : / ? # [ ] @

Sub-delimiters (separating within a part): ! $ & ' ( ) * + , ; =

The key insight: a reserved character is not "must always encode" — it is "do not encode where it is structural, do encode where it is data." The same / is a hierarchy separator in a path and must become %2F inside a query value.

What each part allows

This is the most useful table in the article — the same character follows different rules in different parts:

Note how # is special: its first occurrence anywhere splits off the fragment, and everything after it belongs to the fragment. A # inside a query value, left unencoded, silently hands the rest of the query to the frontend.

Query String Encoding

The query string is everything after ?, made of &-separated key-value pairs.

Three encoding styles

URLSearchParams, Python's urlencode and Go's url.Values.Encode() use the second style (space → +); hand-built paths use the first. Both can coexist in one URL (path %20, query +), but never mix them inside the same segment.

Three conventions for arrays and nesting

There is no standard, only convention. Agree explicitly when crossing languages, or the backend receives a string where it expected an array.

Per-language construction

JavaScript:

const params = new URLSearchParams({ name: 'John & Jane' });
const url = `/api?${params.toString()}`;
// /api?name=John+%26+Jane

Python:

from urllib.parse import urlencode, quote
query = urlencode({'name': 'John & Jane'})   # query: space → +
path  = quote('John & Jane', safe='')        # path segment: space → %20

Go:

import "net/url"
v := url.Values{}
v.Set("name", "John & Jane")
query := v.Encode()                    // query string
seg := url.PathEscape("John & Jane")   // path segment

Never manually concatenate query strings. Always use standard library encoding functions.

International URLs

Non-ASCII encoding

The standard requires ASCII only. For CJK, emoji and the rest, two steps:

1. Character → UTF-8 byte sequence
2. Percent-encode each byte

Emoji take four bytes — one emoji becomes twelve characters, so budget for length (see the limits table below).

Punycode and IDN

Domains use Punycode (RFC 3492): 中文.com becomes xn--fiq228c.com. Browsers show the original characters, but the request uses Punycode.

⚠️ Homograph attacks: Cyrillic а is visually identical to Latin a, so аpple.com (real Cyrillic) can impersonate apple.com. Defend by displaying Punycode for user-supplied domains, or validating against a character allow-list.

Unicode in paths

Path segments use percent-encoding. Modern browsers and HTTP clients handle it automatically, but server logs show the encoded form — do not let logs mislead you while debugging.

Base64 vs URL encoding

Two frequently confused mechanisms:

Standard Base64's +, / and = are all ambiguous inside a URL (+ reads as a space), so RFC 4648 defines Base64URL: - for +, _ for /, and no = padding. JWT segments use Base64URL — which is exactly why a JWT can sit in a URL without another layer of encoding.

Verify conversions with the URL encoder and the Base64 converter; build URL-safe identifiers with the slug generator; and break a full URL into its parts with the URL parser.

Common Encoding Bugs

1. Double encoding

Original:    café
First pass:  caf%C3%A9
Second pass: caf%25C3%25A9  ❌

Cause: the framework already encoded it, then you called encodeURIComponent again. Signature: %25 in the output.

2. Encoding/decoding mismatch

Frontend uses encodeURI, backend decodes with a non-UTF-8 charset → mojibake. Fix by agreeing on UTF-8 end to end and specifying it explicitly when decoding.

3. Space handling differences

Paths use %20; form query strings use +. Mixing them makes the server read + as a literal plus, or %20 as three characters.

4. Unencoded reserved characters

Concatenating user input straight into a URL path. Input containing ?, # or / breaks the structure, and splicing it into a redirect target can produce an open redirect (?next=https://evil.com).

5. Hash fragments are not sent to the server

Nothing after # reaches the server. Putting tokens or callback parameters in the fragment is not security — it merely keeps them out of server logs; frontend JavaScript reads them freely and any XSS can steal them.

6. Wrong encoding order

Encode-then-join and join-then-encode produce very different results:

// ❌ Join first, then encode: the delimiters get encoded too, so the server sees one param
encodeURIComponent('a=1&b=2')          // a%3D1%26b%3D2

// ✅ Encode each value, then join with delimiters
`?a=${encodeURIComponent('1')}&b=${encodeURIComponent('2')}`

7. Case and normalisation

Hex digits in percent-encoding are case-insensitive (%2F ≡ %2f), but the rest of the path is case-sensitive (/API ≠ /api). Before signing, caching or de-duplicating, normalise the URL: decode once → lowercase scheme and host → sort query parameters alphabetically → drop default ports. Without this, one resource yields multiple cache entries or failed signature checks.

8. Length limits

No specification defines a maximum URL length, but reality does:

GET requests carrying many filters hit this quickly — switch to POST with a body.

encodeURI vs encodeURIComponent

// Encode a whole URL (fixes illegal characters, preserves structure)
encodeURI('https://example.com/path?name=café')
// https://example.com/path?name=caf%C3%A9

// Encode a value (delimiters included)
encodeURIComponent('name=café&p=1')
// name%3Dcaf%C3%A9%26p%3D1

⚠️ encodeURIComponent lets through ! ~ * ' ( ) — sub-delimiters under RFC 3986. It rarely matters, but strict signing schemes (AWS SigV4, OAuth 1.0) need one extra pass:

const rfc3986 = (s) => encodeURIComponent(s)
  .replace(/[!'()*]/g, (c) => '%' + c.charCodeAt(0).toString(16).toUpperCase());

When the server decodes

Debugging tip: when a route 404s but the URL looks correct, check for %2F, %5C (backslash) and %2E%2E (..) first. Those are exactly the encodings used for path traversal and route hijacking, so many WAFs and gateways block them outright. See the HTTP status code reference for the codes involved.

Checklist

• [ ] Use encodeURIComponent for values, encodeURI only for a whole URL
• [ ] %20 for spaces in paths; let the standard library handle query strings
• [ ] Never mix + and %20 within one segment
• [ ] Encode user input before putting it in a path, and reject .. and %2F
• [ ] Normalise URLs before signing or caching (decode once, lowercase, sort params)
• [ ] UTF-8 end to end, with the charset specified when decoding
• [ ] Keep query strings under 2,000 characters; switch to POST beyond that
• [ ] Never concatenate redirect targets directly (open redirect)
• [ ] Do not place sensitive parameters in the fragment (readable by frontend JS)
• [ ] Confirm there is no double encoding (search for %25)

Best Practices

1. Always use standard libraries: URLSearchParams, URL, url.Values, urllib.parse
2. Distinguish contexts: encodeURI / PathEscape for paths, encodeURIComponent / QueryEscape for values
3. Standardise on UTF-8: Specify it explicitly on both sides
4. Don't encode twice: Check whether the framework already did
5. Validate user input: Prevent reserved-character injection and path traversal
6. Use HTTPS: Encoded URLs still need transport-layer encryption