Text Processing: Invisible Characters, CSV Parsing and Regex Boundaries as Silent Failures

The answer first: all three failures run cleanly

A comparison returns false, a regex never matches, CSV columns drift — the program keeps running, the exit code is 0, and the problem only surfaces when the result is used. All three live in the text layer, where types and compilers offer no warning.

Invisible characters: why matching silently fails

Text copied from web pages, Word or PDFs often carries characters that occupy no space and display nothing:

Character Code point Common origin
Zero-width space U+200B hidden separators on the web
Zero-width joiner U+200D emoji sequences, Persian
Non-breaking space U+00A0  , Word typography
BOM U+FEFF UTF-8 file header
Curly quotes U+2018/2019/201C/201D Word auto-replace

The consequences are concrete: trim() leaves content behind (U+200B is not whitespace to JavaScript's trim), ^abc$ fails on a string that is really \u200Babc, filenames look identical but compare unequal, and a unique constraint collides for no visible reason.

Diagnosis means printing code points. The text statistics tool lists each character's code point, making differences invisible to the eye obvious. Fix it at the entrance, not at every comparison site:

const normalize = s => s
  .replace(/[\u200B-\u200D\uFEFF]/g, '')   // zero-width chars and BOM
  .replace(/\u00A0/g, ' ')                // NBSP → ordinary space
  .replace(/[\u2018\u2019]/g, "'")        // curly quotes → ASCII
  .normalize('NFKC')                       // fullwidth/compatibility folding
  .replace(/\r\n/g, '\n')                  // unify line endings
  .trim()

NFKC turns fullwidth letters and digits into ASCII (ABC → ABC) and folds compatibility characters (① → 1), which matters greatly for pasted text. It also rewrites subscripts and superscripts, so apply it once at ingestion rather than repeatedly to display text.

CSV: splitting on newlines corrupts records

RFC 4180 permits newlines inside a field as long as the field is double-quoted:

a,b
"line1
line2",c

That is three lines but only two records. Splitting on \n yields three fragments, the second being line2",c, and every downstream step now rests on the wrong record count. Worse, nothing is raised: split always succeeds.

A real parser handles quote doubling (""), embedded newlines, delimiter escaping and the BOM. Python has csv, JavaScript has papaparse or ExcelJS, and the site's Excel to text and Excel to JSON exporters follow the same rules.

To verify, count the file's lines and the parser's records separately; with quoted newlines the two must differ. Manual splitting is safe only when you know there are no quoted fields, and note that splitting on a delimiter and splitting on line breaks are orthogonal operations — conflating them causes many bugs.

Regex replacement: count first, then constrain

Two properties get misread: a dot matches any character (a.c hits abc, a1c and a-c), and global replacement has collateral effects.

"Replace every dot with a comma" also rewrites version numbers 1.2.3, domains example.com and IP addresses 192.168.1.1 — not just the decimal points you meant.

  1. Count matches first: the length of str.match(re)?.length. A pattern expected to hit five sites but returning 500 means it is wrong.
  2. Constrain with a character class: /\d+\.\d+/g touches only number-dot-number and leaves versions and IPs alone.
  3. Anchor with \b: \bword\b does not match wordsmith.
const text = '1.5 and 1.2.3 and v1.0 and 2.25'
console.log((text.match(/\./g) || []).length)          // 4 — every dot
console.log((text.match(/\d+\.\d+/g) || []).length)    // 2 — decimals only
text.replace(/\d+\.\d+/g, m => m.replace('.', ','))
// 1,5 and 1.2.3 and v1.0 and 2,25  ← versions and v1.0 untouched

Counting before replacing takes seconds and catches nearly every malformed pattern, yet it is almost always skipped.

Deduplication: decide the business key first

b,2
a,1
b,2

Removing duplicate lines leaves two rows. But if the repetition means the same user ID appeared twice, the real issue is the data itself — aggregate it or reject it, rather than deleting rows quietly. Before deduplicating, state whether the key is the whole line or a business key, and which occurrence to keep. The two semantics are entirely different.

Checklist and verification

  • [ ] Print code points when comparison fails, never eyeball
  • [ ] Normalise with NFKC and strip zero-width characters at ingestion
  • [ ] Parse CSV with a real parser, never split on newlines
  • [ ] Count regex matches before replacing
  • [ ] Constrain with classes such as \d+\. instead of a bare dot
  • [ ] Decide whether deduplication is by line or by business key

Copy the same passage from Notepad, Word and a web page into the text statistics tool and the invisible-character counts differ — the Word copy usually carries U+00A0 and curly quotes, the web copy may carry U+200B. That explains why "data pasted from Word never matches". Run a dot replacement and compare the before/after with the text diff tool to see version numbers break when boundaries are not constrained.

Frequently Asked Questions

Why do two visually identical strings compare unequal?

Almost always because of invisible characters. Text copied from web pages, Word documents and PDFs often carries zero-width spaces (U+200B), zero-width joiners (U+200D), non-breaking spaces (U+00A0), byte order marks, and curly quote variants. These occupy no space and display nothing, yet they survive string comparison, regex matching and trim. Typical symptoms: content remaining after trim, ^abc$ failing to match, filenames that differ while looking identical. Do not debug this by eye — print the code points (a text statistics tool will show them per character). Prevent it by normalising at the boundary: strip zero-width characters, convert U+00A0 to a plain space, and apply NFKC.

Can I process CSV with split on newlines?

Not unless you know the file has no quoted fields. RFC 4180 explicitly permits newlines inside fields as long as they are wrapped in double quotes. So a line a,b followed by "line1 line2",c splits into two halves when you split on newlines, corrupting everything downstream — and silently. Use a real CSV parser (Python's csv module, papaparse, ExcelJS), which handle quote escaping, embedded newlines and delimiter escaping. Manual splitting is only safe for simple formats known to contain no quotes or newlines, or when you split on a delimiter such as comma, semicolon or tab instead of on line breaks.

What is wrong with using a dot to match a separator in a regex?

A dot in a regex matches any character, not a literal dot. a.c matches abc, a1c and a-c — anything three characters long starting with a and ending with c. Using it to convert separators therefore damages far more than intended. A subtler problem is collateral damage from global replacement: replacing every dot with a comma also hits decimal points, version numbers like 1.2.3, domain names and IP addresses. The correct approach: (1) count matches first via the length of str.match(re) to confirm the blast radius; (2) constrain boundaries with character classes, for example /\d+\.\d+/g matches only decimals and leaves version numbers alone; (3) anchor with \b where appropriate. A text statistics tool can count matches quickly, and this step should always come before replacing.

← Back to Blog