Pattern Evaluation Methods in Data Mining
Accuracy, precision, recall, and the confusion matrix are covered with a full worked example in the lesson on data mining operators; information gain is covered with a full worked calculation in the l
Accuracy, precision, recall, and the confusion matrix are covered with a full worked example in the lesson on data mining operators; information gain is covered with a full worked calculation in the lesson on attribute selection measures; and the objective/subjective interestingness measures used to evaluate discovered patterns specifically (rather than trained classifiers) are covered in the lesson on interestingness of patterns in data mining. This lesson covers two evaluation tools that haven't been covered yet and that solve a real gap the metrics above leave open: ROC/AUC, which evaluates a classifier across every possible decision threshold at once rather than just one, and cost-sensitive evaluation, which accounts for the fact that not all mistakes cost the same.
ROC and AUC: Evaluating Every Threshold at Once
A classifier that outputs a score rather than a hard yes/no decision (most do, internally) only becomes a classification once you pick a threshold above which a record counts as "positive." Precision and recall, as computed in the lesson on data mining operators, describe performance at exactly one threshold — AUC (Area Under the ROC Curve) instead measures how well the classifier ranks positives above negatives, regardless of what threshold you'd eventually pick. The most transparent way to compute it directly: take the probability that a randomly chosen positive example scores higher than a randomly chosen negative example. Suppose a classifier scores 3 positive examples (P1, P2, P3) and 3 negative examples (N1, N2, N3), and ranking every one of the 6 examples from lowest score to highest produces this order:
Lowest score Highest score
N3 -> N2 -> P3 -> N1 -> P2 -> P1For every possible (positive, negative) pair — there are 3 x 3 = 9 such pairs — check whether the positive scored higher:
P3 vs N3: P3 scores higher -> counts
P3 vs N2: P3 scores higher -> counts
P3 vs N1: N1 scores higher -> does NOT count
P2 vs N3: P2 scores higher -> counts
P2 vs N2: P2 scores higher -> counts
P2 vs N1: P2 scores higher -> counts
P1 vs N3: P1 scores higher -> counts
P1 vs N2: P1 scores higher -> counts
P1 vs N1: P1 scores higher -> counts
Pairs where the positive outranks the negative = 8 out of 9
AUC = 8 / 9 ≈ 0.889An AUC of 0.889 means an example drawn at random from the positive class outscores a randomly drawn negative about 89% of the time — well above the 0.5 an AUC would show for a classifier no better than random guessing, but not the 1.0 of a perfect ranking, since N1 outranked P3 here. This single number summarizes the classifier's discriminative ability across every threshold simultaneously, which is exactly why it's used to compare two classifiers before either one has actually been deployed at a specific cutoff.
Cost-Sensitive Evaluation: Not All Mistakes Are Equal
Precision, recall, and even AUC treat every misclassification as equally bad, which is rarely true in practice. Cost-sensitive evaluation assigns an explicit dollar (or other unit) cost to each kind of error and compares options by total expected cost rather than by an error rate that quietly assumes every error is interchangeable. Take a fraud-detection model where missing an actual fraud case (a false negative) costs the business $500 in unrecovered loss, while flagging a legitimate transaction for manual review (a false positive) costs $5 in review labor. Comparing two threshold choices on the same 1,000 transactions:
Threshold A (more sensitive): FN = 2, FP = 50
Threshold B (more strict): FN = 10, FP = 5
Expected cost A = (2 x $500) + (50 x $5) = $1,000 + $250 = $1,250
Expected cost B = (10 x $500) + (5 x $5) = $5,000 + $25 = $5,025Threshold B produces far fewer false positives (5 versus 50) and might look better on a precision-focused report, but it costs four times more overall, because it misses far more of the actually-costly fraud cases. This is exactly the kind of decision the ROI framing in the lesson on the strategic value of data mining depends on: the "best" threshold is whichever one minimizes total expected cost given the actual cost of each error type, not whichever one produces the most favorable-looking precision or recall number in isolation.
A Naming Collision Worth Flagging: "Lift" in Two Different Contexts
A lift chart, used to evaluate a ranking model, compares how many true positives you'd catch by acting on the top-scored X% of records against how many you'd catch by selecting X% at random — a completely different measurement from the lift used to score association rules, covered in the lesson on association rule mining, which compares the observed co-occurrence of two items against what independence would predict. Both use the word "lift" to mean "how much better than a baseline," but a classification lift chart's baseline is random selection while an association rule's lift baseline is statistical independence between two items — worth keeping straight specifically because the same term appears in both contexts with genuinely different formulas behind it.
Where the Rest of Pattern Evaluation Is Covered
Accuracy, precision, recall, F1, and the confusion matrix are covered with a full worked example in the lesson on data mining operators; information gain and the Gini index are covered with a full worked example in the lesson on attribute selection measures; cross-validation is covered with a hand-verified 5-fold example in the lesson on major issues in data mining; and the objective/subjective distinction for evaluating discovered patterns (as opposed to trained classifiers) is covered in the lesson on interestingness of patterns in data mining.