Normalization in Data Mining

When a dataset mixes features with very different ranges — annual income in the tens of thousands and a satisfaction rating from 1 to 5 — algorithms that rely on distance or magnitude will let the large-range feature dominate the result, regardless of which feature actually matters more. Normalization (also called feature scaling) rewrites every feature onto a common scale so that no feature dominates simply because of the units it happens to be measured in. Min-max scaling and z-score standardization — the two most common methods — are covered with a full worked example, including how each behaves in the presence of an outlier, in the lesson on data preprocessing techniques. This lesson covers the methods from that broader list that aren't: decimal scaling, robust scaling, log transformation, and softmax scaling.

Decimal Scaling: The Simplest Possible Method

Decimal scaling divides every value by a power of 10 large enough to bring the largest absolute value below 1. If the maximum absolute value in the dataset has j digits, every value is divided by 10^j. Take the values {350, -970, 610, 220}:

Maximum absolute value = 970 -> 3 digits -> j = 3 -> divide by 10^3 = 1000

350  / 1000 = 0.35
-970 / 1000 = -0.97
610  / 1000 = 0.61
220  / 1000 = 0.22

This preserves the exact relative relationships between values (twice as large stays twice as large) using nothing more than a single division, which is why it's cheap to compute but rarely used where a dataset's scale changes over time — new data arriving with a larger maximum would silently change j, and therefore the scale, for every value computed before it.

Robust Scaling: Normalization That Resists Outliers

Min-max scaling and z-score standardization both use statistics — the minimum/maximum and the mean/standard deviation — that a single extreme outlier can distort dramatically, as shown in the lesson on data preprocessing techniques. Robust scaling replaces those statistics with the median and the interquartile range (IQR), which depend only on the rank order of values rather than their exact magnitude:

Robust score = (x - median) / IQR

The practical benefit becomes visible when the same small dataset is scaled two different ways as its one outlier grows more extreme. Take ages {22, 25, 28, 30, 32, 35} plus one outlier, and compare the z-score and robust score of the ordinary value 28 as that single outlier increases from 90 to 150 to 1000:

Dataset: {22, 25, 28, 30, 32, 35, outlier}
Median = 30, Q1 = 25, Q3 = 35, IQR = 35 - 25 = 10 (unchanged by the outlier's exact value)
Robust score of 28 = (28 - 30) / 10 = -0.20   <- stays exactly -0.20 in every case below

Outlier = 90:   mean = 37.43, std = 21.83   ->  z-score of 28 = (28 - 37.43) / 21.83  ≈ -0.43
Outlier = 150:  mean = 46.00, std = 42.65   ->  z-score of 28 = (28 - 46.00) / 42.65  ≈ -0.42
Outlier = 1000: mean = 167.43, std = 339.90 -> z-score of 28 = (28 - 167.43) / 339.90 ≈ -0.41

The robust score of 28 never moves, because the median and IQR only depend on where 28 ranks among the other ordinary values, not on how extreme the one outlier happens to be. The z-score, by contrast, drifts steadily toward zero as the outlier grows — the ballooning standard deviation squeezes every ordinary value's z-score closer together, making genuinely different values look artificially similar. This is precisely why robust scaling is the recommended choice specifically when a dataset is known to contain outliers that shouldn't be removed but also shouldn't be allowed to distort the scale of everything else.

Log Transformation: Compressing Skewed Data

Log transformation applies a logarithm to every value, which compresses large multiplicative differences into much smaller additive ones. This matters most for heavily right-skewed data — incomes, populations, transaction amounts — where a small number of very large values would otherwise stretch the scale for everyone else. Take four income values: {20000, 25000, 30000, 500000}:

log10(20000)  ≈ 4.301
log10(25000)  ≈ 4.398
log10(30000)  ≈ 4.477
log10(500000) ≈ 5.699

The largest raw value (500000) is 25 times the smallest (20000), but after the log transform the gap shrinks to 5.699 - 4.301 = 1.398 — a difference small enough that an algorithm sensitive to magnitude no longer treats the top earner as existing in an entirely different numeric universe from everyone else. The trade-off, worth stating plainly, is interpretability: a model trained on log10(income) produces coefficients and predictions in log-space, which have to be transformed back (10^value) before they mean anything to a non-technical reader.

Softmax Scaling: Turning Scores into Probabilities

Softmax scaling converts a set of raw numeric scores into values that are all positive and sum to exactly 1 — which is precisely why it's used to convert a classification model's raw output scores into class probabilities, rather than as a general-purpose feature-scaling method. Given raw scores 2.0, 1.0, and 0.1 for three classes:

e^2.0 ≈ 7.389
e^1.0 ≈ 2.718
e^0.1 ≈ 1.105
Sum   ≈ 11.212

P(class 1) = 7.389 / 11.212 ≈ 0.659
P(class 2) = 2.718 / 11.212 ≈ 0.242
P(class 3) = 1.105 / 11.212 ≈ 0.099

Check: 0.659 + 0.242 + 0.099 = 1.000

Notice that softmax doesn't just rank the classes the same way the raw scores did — it also produces a specific probability for each one, which is why it sits at the very end of a classification pipeline rather than being used to prepare input features the way the other three methods above are.

Choosing a Method

The right choice always depends on what the data's distribution actually looks like and what happens downstream — there is no single normalization method that is correct by default, which is exactly why checking a feature's distribution is a preprocessing step in its own right, covered in the lesson on data preprocessing techniques.