Bayesian Classifiers

In many real-world applications, the relationship between input attributes and the class label is not certain. Even when a test record has exactly the same attributes as records in the training set, the class label cannot always be predicted with confidence.

This happens for two main reasons: the data contains noise, or important factors affecting the outcome were never recorded in the first place.

Consider predicting whether a person is at risk of liver disease from their diet and exercise habits. People who eat well and exercise regularly generally carry lower risk — but some develop the disease anyway, through factors the dataset never captured, such as alcohol consumption or a genetic predisposition. On top of that, the recorded attributes are themselves imprecise: deciding whether a diet counts as "healthy" or exercise as "sufficient" involves judgement that two people would make differently.

Classification under this kind of uncertainty is exactly what Bayesian methods are built for. Rather than producing a flat yes/no answer, they produce a probability for each class, which is a more honest output when the evidence genuinely is ambiguous.

Bayesian Classification

Bayesian classification is a statistical approach to classification, based on Bayes' Theorem — a rule for calculating the probability of an event using prior knowledge and observed evidence. A Bayesian classifier uses probability theory to determine how likely it is that a record belongs to each possible class, then assigns the most likely one.

The underlying idea comes from Thomas Bayes, who introduced a method of using conditional probability to estimate unknown quantities from observed evidence.

Bayes' Theorem

The theorem is written as:

             P(Y|X) × P(X)
  P(X|Y)  =  ─────────────
                  P(Y)

where X and Y are events and P(Y) ≠ 0.

Prior and Posterior

In the Bayesian view, a probability represents a degree of belief, and the theorem describes how that belief should change when new evidence arrives.

For a hypothesis X and evidence Y:

• P(X) is the prior — what you believed before seeing the evidence.
• P(X|Y) is the posterior — what you should believe after seeing it.

The rest of the formula controls how much the evidence moves your belief. If the evidence is much more likely under the hypothesis than in general, belief rises sharply; if the evidence is equally likely either way, belief barely moves at all.

A medical test makes this concrete. Suppose a disease affects 1 in 1,000 people, so the prior is 0.001. A person tests positive on a test that is 99% accurate. The posterior is not 99% — because the disease is rare, false positives from the 999 healthy people outnumber true positives from the 1 sick person. Bayes' theorem combines both the test result and the rarity, which intuition alone tends to get badly wrong.

The Naive Bayes Classifier

The most widely used Bayesian classifier in data mining is Naive Bayes. For a record with attributes x₁, x₂, … xₙ, we want the probability of each class C:

  P(C | x₁, x₂, ..., xₙ)

Computing this directly requires knowing how every combination of attributes behaves together, which needs far more training data than is ever available. Naive Bayes solves this with one simplifying assumption: all attributes are conditionally independent given the class. That turns the hard joint probability into a simple product:

  P(C | x₁...xₙ)  ∝  P(C) × P(x₁|C) × P(x₂|C) × ... × P(xₙ|C)

Each term on the right can be counted directly from the training data. The classifier computes this score for every class and picks the highest.

The assumption is called naive because it is almost always false — in a weather dataset, humidity and outlook are clearly related. Naive Bayes nevertheless works remarkably well in practice, because picking the largest score doesn't require the probabilities themselves to be accurate, only their ranking to be right.

A Worked Example

Take this small training set predicting whether a game is played, from the outlook and whether it is windy:

There are 10 records: 7 "Yes" and 3 "No". The priors are:

  P(Yes) = 7/10 = 0.70        P(No) = 3/10 = 0.30

Now predict for a new day: Outlook = Sunny, Windy = No.

Count the conditional probabilities from each class separately:

  P(Sunny | Yes)    = 1/7        P(Sunny | No)    = 2/3
  P(Windy=No | Yes) = 6/7        P(Windy=No | No) = 1/3

Multiply each class's terms together with its prior:

  score(Yes) = 1/7 × 6/7 × 7/10 = 3/35 = 0.0857
  score(No)  = 2/3 × 1/3 × 3/10 = 1/15 = 0.0667

Normalising so the two sum to 1:

  P(Yes | Sunny, not windy) = 0.562
  P(No  | Sunny, not windy) = 0.438

Prediction: Yes.

Notice what happened. "Sunny" on its own points strongly toward "No" — most sunny days in this data were no-play days. But the calm conditions and the higher base rate of playing outweigh it, and the final answer flips. Notice also how close the result is, at 0.562 against 0.438. A classifier that only output "Yes" would hide that this was nearly a coin flip; the probability tells you not to trust this particular prediction very far.

The Zero-Frequency Problem

There is a trap in that multiplication. If an attribute value never appears with a class in training, its conditional probability is zero — and a single zero wipes out the entire product, no matter how strongly every other attribute points that way.

The standard fix is Laplace smoothing (add-one smoothing): add 1 to every count so nothing is ever exactly zero. With three outlook values, P(Sunny|Yes) would become (1+1)/(7+3) = 0.2 rather than 1/7. This barely affects well-populated counts while preventing any single unseen combination from vetoing the result.

Where Naive Bayes Is Used

• Spam filtering — the classic application, treating each word as an attribute. It is fast, handles tens of thousands of features, and works well even though words are obviously not independent.
• Document and topic classification, for the same reasons.
• Medical and risk screening, where a calibrated probability is more useful than a bare label.

Its main strengths are speed, small training data requirements, and simplicity. Its weakness is that when attributes are strongly correlated, the probabilities it reports become overconfident — the ranking usually survives, but the numbers should not be read as precise.

Bayesian Networks

A Bayesian Network is a Probabilistic Graphical Model (PGM) that represents uncertain relationships between variables, also known as a Belief Network. Where Naive Bayes assumes every attribute is independent of every other, a Bayesian network lets you state which variables actually depend on which.

Bayesian networks are represented using a Directed Acyclic Graph (DAG), consisting of:

• Nodes — random variables
• Edges (directed links) — dependencies between variables, pointing from cause to effect

"Acyclic" means the arrows never form a loop, so no variable ends up indirectly depending on itself. Together these model how the probability of one event depends on other related events.

Conditional Probability in Bayesian Networks

Uncertainty in the network is modelled using Conditional Probability Distributions (CPDs). Each variable carries a Conditional Probability Table (CPT) giving the probability of that variable for each combination of its parent variables' values.

A node with no parents simply holds its prior probability. A node with two binary parents needs a row for each of the four parent combinations. This is also the practical limit of the approach: the table grows exponentially with the number of parents, which is why networks are kept sparse.

Seen this way, Naive Bayes is the simplest possible Bayesian network — one class node pointing to every attribute, with no links between the attributes at all. That is precisely the independence assumption, drawn as a picture.

Related Concepts

Classification as a general technique, and the supervised-learning framing that goes with it, are covered in this series' lesson on data mining techniques. Naive Bayes is one of the standard algorithms for document classification described in the text data mining lesson, where the attributes are the word features produced by TF-IDF.