Data Preprocessing Techniques in Data Mining

Several lessons throughout this series — the noisy-age example in the lesson on the challenges of data mining, the missing-value and normalization operators in the lesson on data mining operators — have already shown that dirty data distorts results, sometimes drastically. This lesson covers the actual techniques used to fix that, each worked through on a small dataset so the effect of applying it is visible rather than asserted.

Handling Missing Data

Take six exam scores with two missing: 72, NULL, 85, 90, NULL, 78. Two common strategies fill the gaps using the four known values (72, 85, 90, 78):

Mean   = (72 + 85 + 90 + 78) / 4 = 325 / 4   = 81.25
Median = sort known values -> 72, 78, 85, 90
       = average of the two middle values (78, 85) = 81.5

Here the mean and median land close together, but that won't always be true. The median is robust to skew and extreme values because it only depends on the middle-ranked values, while the mean shifts toward whatever extreme values are present — which is exactly why median imputation is generally preferred over mean imputation once a dataset is known to contain outliers, a point that connects directly to outlier detection below.

Detecting and Treating Outliers

The interquartile range (IQR) method gives a precise, repeatable rule for flagging outliers rather than relying on eyeballing a dataset. Take ten test scores: 55, 60, 62, 65, 68, 70, 72, 75, 78, 150.

Q1 (25th percentile) = between the 3rd value (62) and 4th value (65)
                      = 62 + 0.25 × (65 − 62) = 62.75
Q3 (75th percentile) = between the 7th value (72) and 8th value (75)
                      = 72 + 0.75 × (75 − 72) = 74.25
IQR = Q3 − Q1 = 74.25 − 62.75 = 11.5
Upper fence = Q3 + 1.5 × IQR = 74.25 + 17.25 = 91.5

Any value above 91.5 (or, symmetrically, below Q1 − 1.5 × IQR = 45.5) is flagged as an outlier. The score of 150 clears the upper fence easily and gets flagged, while every other value in the dataset falls comfortably inside the normal range. Once flagged, an outlier can be removed, capped at the fence value, or investigated individually — which choice is correct depends entirely on whether the value represents a genuine data-entry error (like the invalid ages covered in the challenges lesson) or a rare but real event worth preserving, which is exactly the kind of judgment call the lesson on outliers in data mining covers for the case where the outlier itself is the finding rather than the noise.

Normalization and Standardization

Rescaling numeric features matters most when a dataset contains values on very different scales — and it's worth seeing two different rescaling methods respond differently to the same troublesome dataset. Take five household incomes: 30000, 45000, 60000, 75000, 500000 — four fairly typical values and one clear extreme. Min-max normalization rescales every value into a fixed 0–1 range using (x − min) / (max − min). Here min = 30000 and max = 500000, so the range is 470000:

Normalize 60000:  (60000 - 30000) / 470000  ≈ 0.0638
Normalize 500000: (500000 - 30000) / 470000 = 1.0000

Notice the problem: 60000 is a fairly ordinary income among the other four values, yet the single extreme value at 500000 compresses it down to 0.0638 — barely above the minimum. Min-max normalization is sensitive to outliers precisely because both endpoints of its range are set by the most extreme values in the data. Z-score standardization instead rescales using the mean and standard deviation: (x − mean) / standard deviation. The mean of this dataset is 142000, and the standard deviation works out to approximately 179630:

z(60000)  = (60000 - 142000) / 179630   ≈ -0.457
z(500000) = (500000 - 142000) / 179630  ≈  1.993

This doesn't force every value into a bounded 0–1 range, but it centers the data around 0 and expresses each value in terms of how many standard deviations it sits from the mean — a value's z-score stays meaningful and interpretable (how unusual is this value?) even though the extreme value still pulls the mean and standard deviation somewhat, the same statistical logic used for the fraud-detection z-score in the lesson on data mining applications. Neither method is universally correct: min-max is often preferred when the actual bounds of the data are known and stable, while z-score standardization is generally preferred when outliers are present or the true minimum and maximum aren't known in advance.

Encoding Categorical Data

Most mining algorithms expect numbers, not text labels, so a categorical feature like a customer's city has to be converted before it can be used. Two approaches handle this very differently, and picking the wrong one introduces a real error into the data. Ordinal encoding simply numbers the categories — New York = 1, London = 2, Paris = 3. This is a genuine mistake for a feature with no natural order: it tells any distance-based or magnitude-sensitive algorithm that Paris (3) is "more" than New York (1) by exactly the same amount that London (2) is "more" than New York — a relationship that doesn't exist and was never intended. One-hot encoding avoids this by creating one binary column per category instead:

City       ->  is_NewYork  is_London  is_Paris
New York        1            0          0
London          0            1          0
Paris           0            0          1
New York        1            0          0

No column implies an order or magnitude relationship between cities — each is simply present (1) or absent (0). Ordinal encoding is appropriate only when the categories genuinely have a natural order (low, medium, high); for unordered categories like city names, one-hot encoding is the technique that avoids injecting a false numeric relationship into the data.

Reducing Dimensionality: Two Different Approaches

Feature selection and Principal Component Analysis (PCA) both reduce the number of features a model has to work with, but they do it in fundamentally different ways, and conflating them is a common source of confusion. Feature selection chooses a subset of the original features and discards the rest — if a dataset has ten features and eight of them turn out to be irrelevant to the prediction target, feature selection keeps the two that matter and drops the other eight unchanged. PCA instead creates new features that are mathematical combinations of the original ones, chosen specifically to capture as much of the data's variance as possible in as few dimensions as possible. A simple, concrete case makes the motivation clear: if a dataset records a person's height in both inches and centimeters, these two features are perfectly correlated — one is always exactly 2.54 times the other — so they carry no more combined information than either one alone. PCA would detect this redundancy and combine both into a single new dimension that captures essentially all the original variance, rather than carrying two columns that say the same thing twice. This is a different strategy from the sampling and aggregation techniques covered in the lesson on numerosity reduction, which reduce the number of records, not the number of features — the two approaches are often used together in the same pipeline.

Discretization: Turning Continuous Data into Categories

Some algorithms, and some kinds of analysis, work better with categories than with raw continuous numbers. Equal-width binning divides a value's full range into a fixed number of equally sized intervals. Take seven ages: 12, 25, 34, 45, 52, 61, 70, ranging from 12 to 70 (a span of 58), divided into three bins:

Bin width = 58 / 3 ≈ 19.33
Bin 1: 12.00 - 31.33   -> 12, 25          (labeled "Young")
Bin 2: 31.33 - 50.67   -> 34, 45          (labeled "Middle-aged")
Bin 3: 50.67 - 70.00   -> 52, 61, 70      (labeled "Senior")

Each raw age is now replaced by a category label, which is often exactly what's needed for the kind of characteristic-pattern discovery covered in the lesson on data mining task primitives — "customers in their 50s and 60s show pattern X" is a more directly actionable finding than a pattern expressed in terms of exact ages that happen to correlate.

Weighing the Real Costs

None of the techniques above are free. Discarding features (in feature selection) or combining them (in PCA) can genuinely discard information that turns out to matter later; imputing missing values, encoding categories, and computing IQR fences all take real computation time that scales with dataset size, which matters at the scale discussed in the lesson on the challenges of data mining; and because these are all choices — which imputation method, which encoding, how many bins — two analysts working from the same raw dataset can reasonably make different preprocessing decisions and get different downstream results, which is exactly why documenting preprocessing choices carefully matters for anyone who needs to reproduce or audit a data mining result later.