Text Processing Cheat Sheet
Deduplicating, stripping blank lines, diffing, masking and counting words — these small text chores come up for nearly every developer and operator. This page tabulates nine common operations and disambiguates the five counting metrics people most often confuse.
Nine Common Text Operations
| Operation | What It Does | Typical Use |
|---|---|---|
| Text Stats | Count characters, words, lines, bytes and the longest line | Submission word limits, API payload length checks |
| Deduplicate Lines | Drop duplicate lines, usually keeping first occurrence order | Cleaning exported lists, merging keyword sheets |
| Remove Empty Lines | Strip blank lines and whitespace-only lines | Tidying text copied from PDFs or web pages |
| Text Diff | Compare two texts line by line or word by word | Reviewing config changes, manuscript edits |
| Obfuscate | Replace selected characters with asterisks or a mask | Hiding phone numbers, emails and keys before screenshots |
| Pluralise | Convert English nouns between singular and plural | Generating i18n copy, quantifiers in code |
| Normalise Email | Normalise case, split alias and domain, fix dots | Deduplicating signups, cleaning mailing lists |
| NATO Alphabet | Spell letters as Alpha, Bravo, Charlie and so on | Reading order numbers and codes over the phone |
| Lorem Ipsum | Generate filler paragraphs or words for layout | Design mockups, front-end component wiring |
Five Counting Metrics Compared
The same passage can yield numbers three times apart depending on the metric. Confirm which one is required before counting.
| Metric | Definition | Example |
|---|---|---|
| Characters | Unicode code points; one CJK character counts as 1 | "你好 hi" -> 4 |
| Bytes | Bytes after UTF-8 encoding; a CJK character is usually 3 | "你好 hi" -> 8 |
| Words | Split on whitespace; Chinese needs per-character or word segmentation | "你好 hi" -> 1 or 3 depending on the rule |
| Lines | Newline count plus one; mind CRLF versus LF | Two lines -> 2 |
| Display Width | Full-width characters count as 2, half-width as 1 | "你好 hi" -> 6 |
Frequently Asked Questions
How should Chinese text be counted?
There is no single right answer — it depends on the purpose. Publications and freelance payment usually count characters excluding whitespace, where one Han character is one unit. Database column limits are byte-based, and a Han character takes 3 bytes in UTF-8. Layout cares about display width, where a full-width character occupies two cells. Confirm which metric the other party wants; the results can differ by a factor of three.
Does removing duplicate lines preserve the original order?
Most tools preserve first-occurrence order by default: the first time a line is seen it stays in place, and later duplicates are dropped. If you want alphabetical order, sort first and then deduplicate. Conversely, if the order itself carries meaning — a log timeline, for example — do not sort first.
Is the NATO spelling alphabet still useful?
Yes, and it is remarkably practical for phone support and remote collaboration. It maps the 26 letters to words that are hard to confuse acoustically (A is Alpha, B is Bravo), specifically solving problems like b versus d or m versus n. Spelling out an order number, verification code or serial letter by letter is far more efficient than repeatedly asking "b or d?".
What is Lorem ipsum and why does everyone use it?
It is scrambled filler text that looks like Latin but carries no meaning, derived from Cicero and used by the typesetting trade since the sixteenth century. Its advantage is that readers are not distracted by content and instead focus on typography, leading and whitespace. For design mockups and front-end component wiring it also makes it easier to judge how Latin-script text will look.