Building a Text Normalizer

Text normalization is the umbrella pipeline that turns raw, inconsistent text into a standard form a model can work with — tokenization, lowercasing, punctuation removal, and stopword removal are all fairly mechanical steps, contraction expansion is covered in depth (including its one genuine ambiguity, it's) in the lesson on expanding contractions in text mining, and word-frequency representations like bag-of-words and TF-IDF are covered in the lesson on stemming and text preprocessing. This lesson covers two things a normalization pipeline gets wrong surprisingly often: treating stemming and lemmatization as interchangeable, and applying every normalization step in the same fixed order regardless of what comes later.

Stemming vs. Lemmatization: Not Interchangeable Outputs

Both techniques reduce a word to a common base form, but they don't produce the same kind of result. Stemming chops off suffixes using fixed rules, without checking whether what's left is an actual word. Lemmatization looks the word up against a real vocabulary (using its part of speech) and returns the correct dictionary form. The difference shows up clearly on words a simple suffix-stripping rule can't handle correctly:

"studies"  -> Stemming (Porter):  "studi"   (not a real English word)
           -> Lemmatization:      "study"   (the correct dictionary form)
"better"   -> Stemming (Porter):  "better"  (unchanged — no suffix rule applies)
           -> Lemmatization:      "good"    (correctly resolves the irregular comparative,
                                              using its part of speech: adjective)

Stemming's "studi" isn't a bug so much as an inherent limitation: a rule that strips -ies and adds -y works for most words but produces a non-word here, and stemming has no vocabulary to check the result against. Lemmatization avoids this because it looks up the actual word rather than mechanically transforming it — at the cost of needing a vocabulary and part-of-speech information the way stemming doesn't. Whether that trade-off is worth it depends on the task: stemming is faster and good enough when all that matters is grouping related word forms together for counting purposes (as in the bag-of-words representation), while lemmatization matters more when the normalized output itself needs to be readable or linguistically correct, not just internally consistent.

Pipeline Order Is Not Interchangeable Either

A normalization pipeline is usually described as a fixed sequence of steps applied in order, but one very common ordering mistake breaks a step that depends on information an earlier step just destroyed. Named Entity Recognition (NER) — identifying that "Apple" refers to a company rather than a fruit — relies heavily on capitalization as a signal: a capitalized word mid-sentence is far more likely to be a proper noun. If lowercasing runs before NER, "Apple announced record profits" becomes "apple announced record profits," and the exact signal that would have flagged "Apple" as a likely named entity is gone before NER ever sees the text. The fix isn't a different NER algorithm — it's pipeline ordering: run NER (and generally, tokenization and part-of-speech tagging) on the original-case text first, and apply lowercasing, stemming, and stopword removal afterward, for whichever downstream task (like a bag-of-words frequency count) actually needs case-insensitive, stemmed tokens. Treating "normalize the text" as one atomic step obscures the fact that different downstream uses of the same text sometimes need it prepared in a different order.

Where the Rest of This Pipeline Is Covered

Tokenization, stopword removal, and the bag-of-words/TF-IDF representations built from the resulting tokens are covered in the lesson on stemming and text preprocessing; contraction expansion, including the genuine it's-style ambiguity a normalizer has to handle carefully, is covered in the lesson on expanding contractions in text mining; and part-of-speech tagging, the mechanism NER and chunking both depend on, is covered in the lesson on shallow parsing.