Classification Algorithms in Data Mining

Classification itself — assigning records to known categories — along with the specific algorithms that do it (C4.5, rule-based classification, Naive Bayes, Support Vector Machines) and the concepts surrounding it (attribute types, bias-variance trade-off and cross-validation, imbalanced-data handling, evaluation metrics, ensemble methods, ethics) are each covered with their own worked examples elsewhere in this series. Rather than repeating a shorter version of all of that here, this lesson focuses in depth on one distinction between classifiers that hasn't been covered yet: the difference between discriminative and generative models, using the spam-classification example already built in the lesson on data mining projects as the concrete anchor.

Two Different Ways to Learn the Same Boundary

Every classifier ultimately needs to answer the same question for a new record: which class does this belong to? Discriminative models answer it directly — they learn a boundary or a function that maps input features straight to a class, without ever modeling how the data for each class was generated. Logistic regression is the standard example: it learns a set of weights on the input features and combines them into a single score that determines the predicted class, with no intermediate step that represents "what spam emails typically look like" as its own object.

Generative models take the opposite route — they model each class's own data distribution first, then use Bayes' theorem to work out which class is more probable for a new record. Naive Bayes, covered with a full worked spam-classification example in the lesson on data mining projects, is the clearest possible illustration: recall that the model there directly computed P(free | spam) and P(free | not spam) — the probability of the word "free" appearing given each class — as explicit, separately estimated quantities, and only combined them into a final posterior probability (P(spam | free) = 75%) as a second, later step using Bayes' theorem. A discriminative model solving the exact same spam-classification problem would skip that intermediate step entirely: it would fit weights directly to separate spam from non-spam based on the training labels, without ever computing or storing a number that specifically means "how likely is the word 'free' to appear in a spam email."

Why the Distinction Matters in Practice

Because a generative model builds an explicit model of each class's data distribution, it can do things a discriminative model structurally cannot — most usefully, generate new synthetic examples that resemble a given class, since it has already modeled what that class's data looks like. A discriminative model, having learned only the boundary between classes rather than the shape of either one, is often more accurate for pure classification specifically, precisely because it spends its entire modeling effort on the one thing that determines classification performance — the boundary itself — rather than on modeling the full data distribution on each side of it, most of which is irrelevant to where the boundary should sit.

Specialized Classification Scenarios

A handful of scenario names extend the core classification task to situations the basic setup doesn't directly cover, without requiring a fundamentally different algorithm:

• Multi-label classification assigns more than one label to a single record at once (a single news article tagged as both "Politics" and "Economy"), rather than choosing exactly one class as ordinary classification does.
• Real-time classification applies the same trained model to a continuous incoming stream rather than a static batch, which is where the concept-drift monitoring covered in the lesson on data stream mining becomes directly relevant, since a model deployed once and never re-checked will gradually stop matching the data it's scoring.
• Transfer learning reuses a model already trained on one task as the starting point for a related task, rather than training a new model completely from scratch — most commonly seen with large pretrained neural networks, covered in the lesson on neural networks in data mining.
• Active learning selects which unlabeled records are most worth having a human label next, prioritizing the records the current model is least confident about, so that labeling effort — often the most expensive part of building a training set — is spent where it improves the model the most.

Where Everything Else Is Already Covered

• Attribute types (binary, nominal, ordinal, discrete, continuous) — covered in full in the lesson on types of attributes in data mining.
• Bias-variance trade-off, overfitting, and cross-validation — covered with a hand-verified 5-fold example in the lesson on major issues in data mining.
• Imbalanced data (oversampling, undersampling, SMOTE) — covered in the lesson on partitioning methods in data mining.
• Evaluation metrics (accuracy, precision, recall, F1) — covered with a full confusion-matrix worked example in the lesson on data mining operators.
• Ensemble methods and hyperparameter tuning — covered in the lesson on data mining concepts and techniques.
• Ethical issues (bias, privacy) — covered in the lesson on the challenges of data mining.
• C4.5, rule-based classification, Naive Bayes, and Support Vector Machines — each covered with a dedicated worked example in their own lessons.