The answer first: all three failures run cleanly
A comparison returns false, a regex never matches, CSV columns drift — the program keeps running, the exit code is 0, and the problem only surfaces when the result is used. All three live in the text layer, where types and compilers offer no warning.
Invisible characters: why matching silently fails
Text copied from web pages, Word or PDFs often carries characters that occupy no space and display nothing:
| Character | Code point | Common origin |
|---|---|---|
| Zero-width space | U+200B | hidden separators on the web |
| Zero-width joiner | U+200D | emoji sequences, Persian |
| Non-breaking space | U+00A0 | , Word typography |
| BOM | U+FEFF | UTF-8 file header |
| Curly quotes | U+2018/2019/201C/201D | Word auto-replace |
The consequences are concrete: trim() leaves content behind (U+200B is not whitespace to JavaScript's trim), ^abc$ fails on a string that is really \u200Babc, filenames look identical but compare unequal, and a unique constraint collides for no visible reason.
Diagnosis means printing code points. The text statistics tool lists each character's code point, making differences invisible to the eye obvious. Fix it at the entrance, not at every comparison site:
const normalize = s => s
.replace(/[\u200B-\u200D\uFEFF]/g, '') // zero-width chars and BOM
.replace(/\u00A0/g, ' ') // NBSP → ordinary space
.replace(/[\u2018\u2019]/g, "'") // curly quotes → ASCII
.normalize('NFKC') // fullwidth/compatibility folding
.replace(/\r\n/g, '\n') // unify line endings
.trim()
NFKC turns fullwidth letters and digits into ASCII (ABC → ABC) and folds compatibility characters (① → 1), which matters greatly for pasted text. It also rewrites subscripts and superscripts, so apply it once at ingestion rather than repeatedly to display text.
CSV: splitting on newlines corrupts records
RFC 4180 permits newlines inside a field as long as the field is double-quoted:
a,b
"line1
line2",c
That is three lines but only two records. Splitting on \n yields three fragments, the second being line2",c, and every downstream step now rests on the wrong record count. Worse, nothing is raised: split always succeeds.
A real parser handles quote doubling (""), embedded newlines, delimiter escaping and the BOM. Python has csv, JavaScript has papaparse or ExcelJS, and the site's Excel to text and Excel to JSON exporters follow the same rules.
To verify, count the file's lines and the parser's records separately; with quoted newlines the two must differ. Manual splitting is safe only when you know there are no quoted fields, and note that splitting on a delimiter and splitting on line breaks are orthogonal operations — conflating them causes many bugs.
Regex replacement: count first, then constrain
Two properties get misread: a dot matches any character (a.c hits abc, a1c and a-c), and global replacement has collateral effects.
"Replace every dot with a comma" also rewrites version numbers 1.2.3, domains example.com and IP addresses 192.168.1.1 — not just the decimal points you meant.
- Count matches first: the length of
str.match(re)?.length. A pattern expected to hit five sites but returning 500 means it is wrong. - Constrain with a character class:
/\d+\.\d+/gtouches only number-dot-number and leaves versions and IPs alone. - Anchor with
\b:\bword\bdoes not matchwordsmith.
const text = '1.5 and 1.2.3 and v1.0 and 2.25'
console.log((text.match(/\./g) || []).length) // 4 — every dot
console.log((text.match(/\d+\.\d+/g) || []).length) // 2 — decimals only
text.replace(/\d+\.\d+/g, m => m.replace('.', ','))
// 1,5 and 1.2.3 and v1.0 and 2,25 ← versions and v1.0 untouched
Counting before replacing takes seconds and catches nearly every malformed pattern, yet it is almost always skipped.
Deduplication: decide the business key first
b,2
a,1
b,2
Removing duplicate lines leaves two rows. But if the repetition means the same user ID appeared twice, the real issue is the data itself — aggregate it or reject it, rather than deleting rows quietly. Before deduplicating, state whether the key is the whole line or a business key, and which occurrence to keep. The two semantics are entirely different.
Checklist and verification
- [ ] Print code points when comparison fails, never eyeball
- [ ] Normalise with NFKC and strip zero-width characters at ingestion
- [ ] Parse CSV with a real parser, never split on newlines
- [ ] Count regex matches before replacing
- [ ] Constrain with classes such as
\d+\.instead of a bare dot - [ ] Decide whether deduplication is by line or by business key
Copy the same passage from Notepad, Word and a web page into the text statistics tool and the invisible-character counts differ — the Word copy usually carries U+00A0 and curly quotes, the web copy may carry U+200B. That explains why "data pasted from Word never matches". Run a dot replacement and compare the before/after with the text diff tool to see version numbers break when boundaries are not constrained.