Lazy Learning in Data Mining
Every classification algorithm covered elsewhere in this series — C4.5, rule-based classification, Naive Bayes, Support Vector Machines — is an eager learner : it processes the entire training set onc
Every classification algorithm covered elsewhere in this series — C4.5, rule-based classification, Naive Bayes, Support Vector Machines — is an eager learner: it processes the entire training set once, up front, to build a model (a tree, a set of rules, a set of probabilities, a boundary), and then discards the training data, using only that compact model to classify new records. Lazy learning does the opposite: it stores the training data as-is and does no real work until a new record actually needs to be classified, at which point it searches the stored data directly. This lesson works through the most common lazy learner, k-Nearest Neighbors (k-NN), with a full worked classification example.
A Worked k-NN Example
Five training records describe a person's height and weight, along with whether they were classified Fit or Not Fit:
A: (170 cm, 65 kg) -> Fit
B: (160 cm, 55 kg) -> Fit
C: (180 cm, 90 kg) -> Not Fit
D: (175 cm, 85 kg) -> Not Fit
E: (165 cm, 60 kg) -> FitA new person, Q = (172 cm, 68 kg), needs a prediction, using k = 3 and ordinary Euclidean distance:
Distance(Q, A) = sqrt((172-170)^2 + (68-65)^2) = sqrt(4 + 9) = sqrt(13) ≈ 3.606
Distance(Q, E) = sqrt((172-165)^2 + (68-60)^2) = sqrt(49 + 64) = sqrt(113) ≈ 10.630
Distance(Q, D) = sqrt((172-175)^2 + (68-85)^2) = sqrt(9 + 289) = sqrt(298) ≈ 17.263
Distance(Q, B) = sqrt((172-160)^2 + (68-55)^2) = sqrt(144+169) = sqrt(313) ≈ 17.692
Distance(Q, C) = sqrt((172-180)^2 + (68-90)^2) = sqrt(64 + 484) = sqrt(548) ≈ 23.409Sorted from nearest to farthest: A (3.606, Fit), E (10.630, Fit), D (17.263, Not Fit), B (17.692, Fit), C (23.409, Not Fit). With k = 3, only the three nearest records vote — A, E, and D — giving 2 votes for Fit and 1 for Not Fit. Q is classified as Fit, by majority vote among its 3 nearest neighbors, and notice that B — which happens to share Q's actual class — never gets a vote at all, because it isn't among the 3 closest points; k-NN's prediction depends entirely on local proximity, not on which training records ultimately share the correct label. Choosing k as an odd number for a two-class problem, as used here, isn't a formality — it guarantees a clean majority and avoids the tie a k-NN vote could otherwise produce with an even k.
Why Feature Scaling Matters Here Specifically
Because k-NN's entire prediction mechanism is a distance calculation, any feature measured on a much larger numeric scale than the others will dominate that calculation regardless of how relevant it actually is — if this dataset had included income in dollars alongside height in centimeters, the income differences (potentially in the thousands) would swamp the height differences (at most a few tens) in the distance formula, making height essentially irrelevant to every prediction. This is exactly why the scaling techniques covered in the lesson on data preprocessing techniques and the lesson on normalization in data mining are a prerequisite for k-NN specifically, not just a generic good practice — an unscaled feature with a naturally larger range doesn't just contribute more to the result, it can silently dominate it completely.
Why "Lazy" Isn't a Criticism — and What It Costs Instead
Lazy learning's defining trade-off is when the computation happens, not whether it happens: an eager learner (like a decision tree) does expensive work once during training and then answers each new query almost instantly by walking a small tree; k-NN does almost no work during training (it just stores the data) and instead does its expensive work — computing a distance to every stored record — at every single prediction. This makes k-NN well suited to data that changes frequently (a new labeled record just gets added to storage, with no retraining step required at all) and correspondingly poorly suited to applications needing fast predictions against a very large stored dataset, where computing a distance to every stored record on every query becomes the bottleneck. Structures like KD-trees and ball trees exist specifically to avoid checking every stored record for every query, organizing the stored data so that most of it can be ruled out without a full distance calculation.
The Curse of Dimensionality
k-NN's reliance on distance has a specific, well-documented failure mode as the number of features grows: in a very high-dimensional space, the distance between the closest and farthest points in a dataset tends to shrink toward each other in relative terms, because there are so many dimensions along which two points can differ that most pairs of points end up looking roughly equally far apart. When "nearest" and "farthest" stop being meaningfully different, the entire basis for k-NN's vote — that the closest points are more informative than distant ones — breaks down. This is exactly why the dimensionality-reduction techniques (PCA) and feature-selection strategies covered in the lesson on preprocessing in data mining matter more for k-NN than for many other algorithms: reducing an irrelevant or redundant feature isn't just a minor cleanup step here, it's often the difference between a distance calculation that's actually meaningful and one that isn't.
Where This Leaves Lazy Learning
k-NN is the standard lazy learner, but the same family includes variants worth naming: Radius Neighbors votes using every training point within a fixed distance rather than a fixed count k; Locally Weighted Learning gives closer neighbors more influence over the vote instead of counting every one of the k nearest points equally; and Case-Based Reasoning applies the same "find similar past examples" logic outside strictly numeric feature spaces, to structured cases such as legal precedents or diagnostic histories.