URL Encoding/Decoding Guide: Percent-Encoding, Query Strings & Common Pitfalls

The answer first: the rule depends on which segment the character sits in

URL encoding is not "replace every special character with %xx". The same character has different legality in the path, the query and the fragment, so whether it is encoded — and into what — depends on position. One principle covers it: a reserved character stays literal where it is structural and must be encoded where it is data.

https://api.example.com:443/v1/search?q=caf%C3%A9&page=2#results
└─┬──┘ └──────┬──────┘└┬┘└────┬────┘ └────┬────┘ └───┬───┘
scheme     host      port    path       query      fragment

Character classes and per-segment rules

A URL allows only a subset of ASCII. Unreserved characters A-Z a-z 0-9 - _ . ~ may appear literally anywhere; reserved characters : / ? # [ ] @ ! $ & ' ( ) * + , ; = appear literally only when they carry structure and must be encoded as data. Everything else — Chinese, spaces, emoji — is always encoded.

A per-segment reference table

Character Path Query Fragment Notes
Alphanumeric - _ . ~ ✅ literal ✅ literal ✅ literal Unreserved
/ ⚠️ delimiter ✅ allowed ✅ allowed %2F when data
? ⚠️ starts query ✅ allowed ✅ allowed %3F when data
# ❌ starts fragment ❌ starts fragment ❌ starts fragment Always %23 as data
& = ✅ allowed ⚠️ delimiter ✅ allowed %26 %3D in values
+ ⚠️ literal plus ⚠️ space in forms ✅ literal See below
% ❌ escape prefix ❌ escape prefix ❌ escape prefix %25 when literal
Space %20 %20 or + %20 Paths require %20
Non-ASCII UTF-8 bytes, each encoded same same See internationalisation

# is special because its first occurrence takes everything after it: one unencoded # inside a query value turns the remaining parameters into a fragment the server never receives.

Base64URL is not URL encoding

URL encoding rewrites characters as % plus hex; Base64 turns binary into A-Za-z0-9+/=; Base64URL then swaps +// for -/_ and drops padding so the result can sit directly in a URL. JWT segments use Base64URL, which is why no extra percent-encoding layer is needed; decode by hand with the Base64 converter.

Building query strings

The query is the part after ?, made of &-separated pairs. There is no single standard here, only two conventions living side by side.

Spaces: where %20 ends and + begins

Context Space as Origin
Percent-encoding %20 RFC 3986
application/x-www-form-urlencoded + HTML form submission

URLSearchParams, Python's urlencode and Go's url.Values.Encode() follow the second convention, turning spaces into +; hand-built paths must use %20. Path and query may each use their own convention, but never mix them inside one segment.

Arrays and nesting: convention only

Style Example Commonly supported by
Repeated keys tag=a&tag=b Nearly everything (PHP needs tag[])
Brackets tag[]=a&tag[]=b PHP, Rails, Express (qs)
Dotted path user.name=Tom Java Spring, some Go frameworks

Cross-language integrations must agree explicitly, or the backend receives a string where it expected an array.

Use the standard library

JavaScript:

const params = new URLSearchParams({ name: 'John & Jane' });
const url = `/api?${params.toString()}`;
// /api?name=John+%26+Jane

Python:

from urllib.parse import urlencode, quote
query = urlencode({'name': 'John & Jane'})   # query: space → +
path  = quote('John & Jane', safe='')        # path segment: space → %20

Go:

import "net/url"
v := url.Values{}
v.Set("name", "John & Jane")
query := v.Encode()                    // query string
seg := url.PathEscape("John & Jane")   // path segment

International URLs

Non-ASCII: UTF-8 first, then byte by byte

Two steps: convert the character to its UTF-8 byte sequence, then percent-encode each byte.

Character UTF-8 bytes Encoded
中 E4 B8 AD %E4%B8%AD
é C3 A9 %C3%A9
€ E2 82 AC %E2%82%AC
😀 F0 9F 98 80 %F0%9F%98%80

An emoji occupies four bytes, so one emoji becomes twelve characters; budget URL length in bytes, not characters.

IDN and homographs

Domains use Punycode (RFC 3492): 中文.com becomes xn--fiq228c.com. Browsers display the original characters while requests use the Punycode form.

⚠️ Homograph attacks: Cyrillic а is visually identical to Latin a, so аpple.com can impersonate apple.com. Defend by displaying Punycode for user-supplied domains or validating against a character allow-list. Break a real link into its parts with the URL parser.

Encoding bugs, ordered by frequency and impact

1. Double encoding

Symptom: %25 appears — the % itself got encoded again. Cause: the framework already encoded the value and the code called encodeURIComponent on top. Check: decode twice and see whether it round-trips, or simply search for %25. Fix: separate the layers — the data layer keeps raw values, the construction layer encodes once, the transport layer touches nothing.

Original:    café
First pass:  caf%C3%A9
Second pass: caf%25C3%25A9  ❌

2. Join first, encode second

Symptom: ?a=1&b=2 reaches the server as one parameter, or the value contains a%3D1%26b%3D2. Cause: the whole pair string was encoded at once, so & and = stopped delimiting. Check: print the final URL and count the & separators; the correct count is one, the broken count is zero. Fix: encode each value separately, then join with the delimiters.

// ❌ Join first, then encode: the delimiters get encoded too, so the server sees one param
encodeURIComponent('a=1&b=2')          // a%3D1%26b%3D2

// ✅ Encode each value, then join with delimiters
`?a=${encodeURIComponent('1')}&b=${encodeURIComponent('2')}`

3. Unencoded reserved characters rewrite the structure

Symptom: the path is truncated, parameters vanish, or the user lands on another site. Cause: raw user input is concatenated into a URL, so ?, # or / changes the structure; splicing it into a next= parameter creates an open redirect. Check: feed input containing ?, # and / and observe the path segment count and parameter count the server receives. Fix: encode every value, and restrict redirect targets to allow-listed relative paths.

4. %2F turns routes into 404s

Symptom: an encoded slash returns 404 or matches the wrong route. Cause: most servers and gateways decode the path before route matching, so %2F becomes / and the segment count changes. Check: compare the decoded path in the access log against the raw path sent by curl --path-as-is. Fix: move such values into the query string, switch to Base64URL, or stop the server decoding — Apache's AllowEncodedSlashes NoDecode, or Nginx keeping $request_uri.

5. Charset mismatch produces mojibake

Symptom: Chinese turns into replacement characters while ASCII stays intact. Cause: one side writes UTF-8 and the other decodes as GBK, or the reverse. Check: compare the original bytes with the decoded bytes, and inspect the response charset. Fix: UTF-8 end to end with explicit declarations; specify the charset when decoding instead of relying on detection.

6. Missing normalisation breaks caching and signatures

Symptom: one resource yields several cache keys, or signature verification passes intermittently. Cause: hex case, default ports, parameter order and empty parameters differ across implementations. Check: normalise two visually identical URLs and compare them. Fix: decode once, lowercase scheme and host, sort query parameters alphabetically, drop default ports — then sign or build the cache key.

encodeURI vs encodeURIComponent

Function Purpose Characters left alone
encodeURI Encode a whole URL A-Za-z0-9 ; , / ? : @ & = + $ - _ . ! ~ * ' ( ) #
encodeURIComponent Encode a parameter value A-Za-z0-9 - _ . ! ~ * ' ( )
new URL() Parse and normalise Handles each segment automatically
// Encode a whole URL (fixes illegal characters, preserves structure)
encodeURI('https://example.com/path?name=café')
// https://example.com/path?name=caf%C3%A9

// Encode a value (delimiters included)
encodeURIComponent('name=café&p=1')
// name%3Dcaf%C3%A9%26p%3D1

⚠️ encodeURIComponent lets ! ~ * ' ( ) through — RFC 3986 sub-delimiters. It rarely matters, but strict signing schemes (AWS SigV4, OAuth 1.0) need one extra pass:

const rfc3986 = (s) => encodeURIComponent(s)
  .replace(/[!'()*]/g, (c) => '%' + c.charCodeAt(0).toString(16).toUpperCase());

When the server decodes

Stage Decodes? Notes
Nginx / Apache route matching ⚠️ Usually first %2F becomes /, changing the segment count → 404
Framework routing ✅ Path segments %2F is a common exception
Query parsing ✅ Automatic Both + and %20 become spaces
Request body Depends on Content-Type Forms decode; JSON does not

Another boundary: nothing after # enters the request. Putting a token in the fragment only keeps it out of server logs; frontend JavaScript reads it freely and any XSS can steal it, so it is not a security measure.

When a route 404s but the URL looks right, check %2F, %5C (backslash) and %2E%2E (..) first — they are the encodings used for path traversal, and many WAFs block them outright. See the HTTP status code reference for related codes.

Length limits

No specification sets a maximum URL length, but reality does:

Component Limit
IE (historical) 2,083 characters
Most CDNs / gateways 8,192 characters
Nginx default large_client_header_buffers 8 KB (beyond it returns 414)
Practical guidance Keep query strings under 2,000 characters

GET requests with many filters hit this fastest; move to POST with a body.

Checklist

  • [ ] encodeURIComponent for values, encodeURI only for a whole URL
  • [ ] %20 for spaces in paths; let the standard library handle queries
  • [ ] Never mix + and %20 inside one segment
  • [ ] Encode user input before it enters a path, and reject .. and %2F
  • [ ] Normalise URLs before signing or caching
  • [ ] UTF-8 end to end, with no double encoding (search for %25)
  • [ ] Switch to POST once a query passes 2,000 characters
  • [ ] Keep sensitive parameters out of the fragment

30-second verification

from urllib.parse import quote, quote_plus
s = '你好 world'
print(quote(s))        # → %E4%BD%A0%E5%A5%BD%20world (space kept as %20)
print(quote_plus(s))   # → %E4%BD%A0%E5%A5%BD+world (space becomes +)

The same input yields two valid but non-equivalent results: quote for path segments, quote_plus for query strings, with unquote / unquote_plus on the way back. The Chinese text becomes six percent-encoded bytes (two characters × three UTF-8 bytes each), visible right here. Encode online with the URL encoder and build URL-safe identifiers with the slug generator.

Frequently Asked Questions

Should a space be encoded as %20 or +?

It depends on which part of the URL it appears in. In the path it must be %20. In the query string both parse correctly in mainstream frameworks, but they mean different things: %20 is RFC 3986 percent-encoding, while + comes from application/x-www-form-urlencoded, the legacy HTML form format. Practical rule: if you build the query with a standard library (URLSearchParams, urlencode) it emits +; when hand-building a path, use %20. Never mix them within the same segment, and never put a bare + in a path — the server will read it as a literal plus sign.

How do I choose between encodeURIComponent and encodeURI?

Choose by whether you are encoding a whole URL or a single component. encodeURI assumes a complete URL, so it leaves ? # / & = : intact — use it to make an otherwise-illegal URL valid (for example, fixing non-ASCII characters in a path). encodeURIComponent assumes a single value and encodes all of those — use it whenever you are inserting a parameter value. Most encoding bugs come from reaching for encodeURI where encodeURIComponent was required, so an & inside a value gets parsed as a separator.

Why does the server return 404 after I encode a slash as %2F?

Because many web servers (Apache by default, some Nginx configurations, and most gateways) decode the path before route matching, so the decoded / is treated as a hierarchy separator and the number of path segments changes. RFC 3986 intends %2F to mean *this slash is data, not a delimiter*, but servers do not always honour that. Three fixes: (1) pass such values as query parameters instead of path segments; (2) use an encoding that cannot collide with delimiters, such as Base64URL; (3) enable the relevant switch explicitly (Apache AllowEncodedSlashes NoDecode, or keep Nginx on $request_uri without decoding).

How do I detect and avoid double encoding?

The symptom is unmistakable: %25 appears in the output, meaning the % itself got encoded. It happens when a framework already encoded the value and you encode it again by hand. Prevent it with a clear layering rule: the data layer stores raw values, the construction layer encodes exactly once, and the transport layer leaves it alone. To test a suspicious string, decode it twice with decodeURIComponent and see whether it round-trips — the [URL encoder](/url-encoder.html) shows encoded and decoded output side by side, so you can spot it at a glance.

Can URL encoding be used as a security measure?

No. Encoding solves *can this character travel safely*, not *is this content malicious*. An encoded <script> is still a <script> — the server decodes it and executes it as-is. All encoding actually prevents is user input breaking URL structure (a stray ?, # or & truncating parameters or producing an open redirect). Real protection still comes from input validation, output escaping (HTML entities for HTML contexts), parameterised queries, and a CSP. Treating encoding as XSS defence is a common misconception.

← Back to Blog