Regression in Data Mining

A retailer wants to know how much revenue an extra ₹10,000 of ad spend will bring in. A hospital wants to estimate a patient's expected recovery time from lab values. A logistics company wants a formula for delivery time in terms of distance and package weight. None of these questions has a small set of categories as an answer — the answer is a number, and it can fall anywhere on a continuous scale. Regression is the family of data mining techniques built to answer exactly this kind of question.

What Is Regression?

Regression is a predictive data mining technique that models the relationship between a dependent variable (the number being estimated) and one or more independent variables (the inputs used to estimate it). Once that relationship is expressed as an equation, the equation can be applied to new inputs to produce a prediction.

This is the line that separates regression from classification, and it is worth stating precisely because the two are so often confused:

• "Will this customer churn?" → a category (yes/no) → classification
• "How many days until this customer churns?" → a number → regression

If the output is a label from a fixed set of options, it is classification. If the output is a quantity that could take on a range of values, it is regression.

Simple Linear Regression

The simplest case uses exactly one independent variable and assumes the relationship between it and the output is a straight line:

y = b₀ + b₁x

• y — the predicted value (dependent variable)
• x — the input (independent variable)
• b₀ — the intercept: the predicted value of y when x is 0
• b₁ — the slope: how much y changes for each one-unit increase in x

"Best fit" has a precise meaning here: among all possible straight lines, linear regression picks the one that minimizes the sum of the squared vertical distances between the line and the actual data points (the least squares method).

Worked Example

Suppose a company records monthly advertising spend (in thousands of rupees) against the resulting sales (in units):

Fitting a least-squares line to this data produces:

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[10], [20], [30], [40], [50]])
y = np.array([15, 25, 35, 45, 58])

model = LinearRegression().fit(X, y)
print(f"slope (b1) = {model.coef_[0]:.3f}")
print(f"intercept (b0) = {model.intercept_:.3f}")
print(f"predicted sales at spend=60: {model.predict([[60]])[0]:.1f}")

Output:

slope (b1) = 1.060
intercept (b0) = 3.800
predicted sales at spend=60: 67.4

The fitted equation is:

sales = 3.800 + 1.060 × ad_spend

Reading the coefficients: for every additional ₹1,000 spent on advertising, sales are predicted to rise by about 1.06 units, and at zero spend the model still predicts roughly 3.8 units of baseline sales — customers who buy without seeing an ad. Plugging in 60 for ad_spend gives a prediction of about 67.4 units, extending the pattern in the data one step beyond what was observed.

That last point is worth pausing on: the model is interpolating a trend, not looking anything up. It has no way of knowing whether spend of 60 behaves like spend of 10–50 or whether the relationship changes at that scale — a genuine limitation of applying regression outside the range of the training data.

Multiple Linear Regression

Real outcomes rarely depend on a single input. Multiple linear regression extends the same idea to several independent variables:

y = b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ

Predicting an employee's salary, for instance, might use experience, education level, and job role together rather than any one of them alone. The fitting method is the same least-squares idea; the difference is that the model now fits a plane (with two inputs) or a hyperplane (with more) instead of a line.

Adding more input variables almost always improves how well the line fits the training data — that alone is not a reason to add them. A variable only earns its place if it improves predictions on data the model has not already seen.

Polynomial Regression

Not every relationship is a straight line. Fuel consumption against speed is a common example: consumption is high at low speeds (idling, frequent braking), drops as speed increases into an efficient range, and rises again at high speed because of air resistance. A straight line cannot capture a curve that goes down and then up — a polynomial regression can, by adding powers of the input variable:

y = b₀ + b₁x + b₂x² + b₃x³ + ...

This is still classified as a form of linear regression, which surprises many learners the first time they see it. "Linear" refers to how the coefficients b₀, b₁, b₂, ... combine — they are added together, never multiplied by each other or raised to a power. The curve traced by x² is not linear, but the model is linear in its parameters, and that is what the term describes.

A polynomial model fits the training data more closely as its degree increases, which creates a real risk: a high-degree polynomial can bend itself through every training point exactly and still generalize badly to new data (overfitting). Degree is a choice to make deliberately, validated against data the model has not seen — not simply raised until the fit looks good.

Logistic Regression

Despite the name, logistic regression is used for classification, not for predicting a continuous number. It estimates the probability that a record belongs to a particular class — for example, whether an email is spam — and applies an S-shaped (sigmoid) function to constrain that probability between 0 and 1:

P(y=1) = 1 / (1 + e^(-(b₀ + b₁x)))

The name is a historical artifact: it borrows the linear combination b₀ + b₁x from linear regression and then passes it through the sigmoid function. Because the output is genuinely a probability used to assign a class label, logistic regression belongs conceptually with classification techniques even though its formula descends from regression.

Ridge and Lasso Regression: Solving Multicollinearity

Ordinary linear regression runs into a specific, well-defined problem when two or more independent variables are highly correlated with each other — a condition called multicollinearity. The clearest example is a dataset that records a person's height in both centimeters and inches: the two columns carry essentially the same information, just rescaled.

When this happens, ordinary least squares cannot decide how to split credit between the redundant variables, and the coefficients it produces become large, unstable, and sensitive to tiny changes in the data — technically correct on the training set, but unreliable and hard to trust.

Worked Example

Simulating 50 people's height (recorded in both units, with slightly different rounding noise) and weight demonstrates this directly:

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso

rng = np.random.default_rng(42)
height_cm = rng.uniform(150, 200, 50)
height_in = height_cm / 2.54 + rng.normal(0, 0.05, 50)   # same signal, tiny noise
weight = 0.5 * height_cm + rng.normal(0, 5, 50)

X = np.column_stack([height_cm, height_in])

lr = LinearRegression().fit(X, weight)
ridge = Ridge(alpha=10).fit(X, weight)
lasso = Lasso(alpha=0.5).fit(X, weight)

print("Ordinary regression coefficients:", np.round(lr.coef_, 3))
print("Ridge regression coefficients:   ", np.round(ridge.coef_, 3))
print("Lasso regression coefficients:   ", np.round(lasso.coef_, 3))

Output:

Ordinary regression coefficients: [ 5.458 -12.353]
Ridge regression coefficients:    [ 0.563   0.088]
Lasso regression coefficients:    [ 0.596   0.0  ]

Ordinary regression assigns height_cm a coefficient of 5.458 and height_in a negative coefficient of -12.353 — even though weight obviously does not decrease as height in inches increases. The two variables are fighting each other for credit over the same underlying signal, and the result is a formula that fits the training data but means nothing physically.

Ridge regression fixes this by adding a penalty term to the fitting process that discourages coefficients from growing large:

minimize: (sum of squared errors) + α × (sum of squared coefficients)

With that penalty applied (alpha=10 above), both coefficients shrink to small, stable, same-signed values — 0.563 and 0.088 — a far more trustworthy split of a redundant signal between two variables.

Lasso regression (Least Absolute Shrinkage and Selection Operator) uses the sum of absolute coefficient values as its penalty instead of squared values:

minimize: (sum of squared errors) + α × (sum of |coefficients|)

That difference in penalty shape has a distinctive practical effect: Lasso can shrink a coefficient all the way to exactly zero, effectively removing that variable from the model. In the output above, Lasso assigned height_in a coefficient of exactly 0.0, automatically recognizing it as redundant with height_cm and dropping it — Ridge only shrinks toward zero, while Lasso can eliminate a variable outright. This makes Lasso useful not just for stabilizing predictions but for automatic feature selection when a dataset has many candidate inputs and some are redundant or irrelevant.

Real-World Applications

• Financial forecasting — projecting revenue, stock prices, or expenses from historical trends.
• Business analytics — estimating sales from advertising spend, pricing, and seasonality, as in the worked example above.
• Marketing — predicting customer lifetime value or the return on a marketing campaign from spend and channel data.
• Environmental modeling — estimating pollutant concentration or crop yield from temperature, rainfall, and soil measurements.
• Healthcare — estimating recovery time or dosage response from patient and lab data.
• Manufacturing — estimating output quality or equipment wear from process measurements such as temperature and pressure.

Choosing Among Regression Techniques

The choice is rarely made from theory alone. Fitting a straight line first and inspecting how far the predictions miss the actual values — and whether independent variables are correlated with each other — is what tells you which of these to reach for next.