Feature Transformation in Data Mining

A dataset arrives with whatever distribution reality gave it — income figures cluster near a typical value with a long tail of high earners, word counts pile up at zero with occasional large bursts, and physical measurements often follow their own irregular shape. Many algorithms, though, quietly assume the numbers they receive look roughly bell-shaped, or at least well-behaved. Feature transformation is the practice of applying a mathematical function to a feature (a data column) so that its values take a form more suited to whatever comes next — a more symmetric shape, a smaller range, or a relationship that a downstream model can pick up more easily.

This is distinct from feature reduction, which cuts down the number of features, and from feature construction, which builds an entirely new feature out of others. Feature transformation instead reshapes the values of a single feature, and it is one of the core techniques grouped under feature engineering — the broader effort of shaping raw data into a form that helps a model learn.

Why Transform a Feature at All?

Several widely used models — linear regression and logistic regression among them — perform best when the input features are close to normally distributed (the familiar bell-shaped curve) and when extreme values do not dominate the numbers a coefficient has to be fit to. Real-world data routinely violates this: incomes, transaction amounts, word frequencies, and reaction times are all typically right-skewed — most values are small or moderate, and a long tail of large values stretches out to the right.

The effect of that skew is concrete and worth seeing directly rather than taking on faith. Simulating 1,000 income-like values (an exponential distribution shifted to a realistic range) and measuring skewness — a single number where 0 means perfectly symmetric, positive means a right-leaning tail, and negative means a left-leaning tail — shows exactly the kind of shape a linear model struggles with:

import numpy as np
from scipy import stats

np.random.seed(42)
income = np.random.exponential(scale=50000, size=1000) + 20000
print("skewness before:", round(stats.skew(income), 3))

log_income = np.log(income)
print("skewness after log transform:", round(stats.skew(log_income), 3))

Output:

skewness before: 1.865
skewness after log transform: 0.345

A skewness of 1.865 describes a distribution with a pronounced right tail — a small number of very large incomes pulling the shape far from symmetric. After a log transform, skewness drops to 0.345, much closer to the symmetric shape a linear model's assumptions are built around. This is the entire point of feature transformation: not to make the data "nicer" for its own sake, but to make it closer to the shape an algorithm was designed to work with, which in practice tends to produce more stable coefficients, faster and more reliable convergence during training, and better accuracy.

Transformation Techniques

1. Log Transformation

The log transform (y = log(x)) compresses large values far more than it compresses small ones, which is exactly why it works so well on right-skewed data: a jump from 20,000 to 220,000 (a difference of 200,000) becomes a jump from about 9.9 to about 12.3 (a difference of only 2.4) on the log scale. It cannot be applied to zero or negative values, since the logarithm is undefined there — a dataset containing zeros needs either a small constant added first (log(x + 1)) or a different transform entirely.

2. Reciprocal Transformation

The reciprocal transform (y = 1/x) inverts the ordering of magnitudes — large values become small, and small values become large:

vals = np.array([1, 2, 4, 10, 100])
print(1 / vals)

Output:

[1.   0.5  0.25 0.1  0.01]

Notice how aggressively this compresses the top end: 100 becomes 0.01, while 1 stays 1. This makes the reciprocal transform a much stronger, blunter tool than the log transform — it is useful when a feature represents something like "time to complete a task" and the analysis cares more about "rate" (tasks per unit time) than raw duration, but it is undefined at zero and can distort a distribution more than a milder transform would.

3. Square Transformation

The square transform (y = x^2) does the opposite job of the log transform: it stretches out large values and compresses small ones, which is useful for left-skewed data — where the long tail points toward small values instead of large ones. Simulating a left-skewed distribution and squaring it shows the effect, and also shows that this transform is comparatively mild:

left_skewed = 300000 - np.random.exponential(scale=50000, size=1000)
print("skewness before:", round(stats.skew(left_skewed), 3))
print("skewness after square transform:", round(stats.skew(left_skewed**2), 3))

Output:

skewness before: -1.992
skewness after square transform: -1.001

The skew moved from -1.992 to -1.001 — a real improvement, but the data is still clearly left-skewed, not symmetric. Squaring is a weaker corrective than the power transformations covered below, and on its own it will rarely be enough to fully normalize a heavily skewed feature.

4. Square Root Transformation

The square root transform (y = sqrt(x)) is a milder version of the log transform: it also compresses large values relative to small ones, but less aggressively, and it only works on non-negative values.

counts = np.random.poisson(lam=4, size=1000)**2 + 1
print("skewness before:", round(stats.skew(counts), 3))
print("skewness after sqrt transform:", round(stats.skew(np.sqrt(counts)), 3))

Output:

skewness before: 1.644
skewness after sqrt transform: 0.541

This is a common choice for count data (page visits, word occurrences, event counts) precisely because it is gentler than a log transform — useful when the log transform ends up over-correcting a distribution that is only mildly skewed.

5. Custom Transformation

Any mathematical function can be applied as a transformation when domain knowledge suggests it — dividing every value by a maximum to bring a feature into a fixed range, applying a log specifically to frequency counts before feeding them into a model, or applying a domain formula (such as converting a raw pixel intensity into a perceptual brightness scale). The techniques above are the common, general-purpose cases; a custom transformation is appropriate whenever the shape of a particular feature calls for something more specific than log, square, reciprocal, or square root.

6. Power Transformations

Log, square, reciprocal, and square root are all special or limiting cases of a more general family called power transformations, which search for the specific transformation (out of a continuous family of possibilities) that brings a feature as close to normally distributed as the data allows. Two variants dominate in practice.

Box-Cox Transformation searches over a parameter λ (lambda) for the power transformation that best normalizes the data, choosing λ automatically rather than requiring a person to guess between log, square root, or another fixed option:

bc_data, bc_lambda = stats.boxcox(income)
print("lambda:", round(bc_lambda, 4))
print("skewness after Box-Cox:", round(stats.skew(bc_data), 3))

Output:

lambda: -0.2759
skewness after Box-Cox: 0.054

A skewness of 0.054 is close to perfectly symmetric — noticeably better than the plain log transform's 0.345 on the same data — because Box-Cox is searching for the best power transformation for this specific dataset rather than committing to log in advance. The trade-off is that Box-Cox requires strictly positive data; it fails outright on any feature containing zero or negative values:

mixed = np.random.exponential(scale=5, size=1000) - 8   # contains negative values
stats.boxcox(mixed)

Output:

ValueError: Data must be positive.

Yeo-Johnson Transformation solves exactly this limitation. It is built on the same idea as Box-Cox — searching for a power-transformation parameter that normalizes the data — but its formula is defined piecewise so that it also accepts zero and negative values:

yj_data, yj_lambda = stats.yeojohnson(mixed)
print("lambda:", round(yj_lambda, 4))
print("skewness after Yeo-Johnson:", round(stats.skew(yj_data), 3))

Output:

lambda: 0.5858
skewness after Yeo-Johnson: 0.407

Skewness dropped from 1.865 to 0.407 on data that Box-Cox could not have touched at all. This is the practical reason Yeo-Johnson is often the default choice in general-purpose preprocessing pipelines: it behaves like Box-Cox on positive data, but does not fail the moment a feature contains a zero or a negative value — a very common situation for features like temperature readings, profit/loss figures, or any measurement centered on zero.

Choosing a Transformation

The right choice depends on two things: the direction of the skew, and whether the feature can contain zero or negative values. A right-skewed, strictly positive feature is a natural candidate for a log or square-root transform, or Box-Cox if an automatically-tuned result is worth the extra computation. A left-skewed feature calls for a square transform. Any feature that can be zero, negative, or both rules out the log, square-root, reciprocal, and Box-Cox transforms outright, leaving Yeo-Johnson (or a custom transformation built for that specific case) as the practical option. In every case, the way to confirm a transformation actually helped is the same one used throughout this lesson: measure skewness before and after, rather than assuming a named technique automatically fixes the data it is applied to.