Interestingness of Patterns in Data Mining

A mining algorithm can easily surface thousands of technically valid patterns from a large dataset — and the overwhelming majority of them will be useless: obvious restatements of what's already known, or statistically real but practically meaningless. Interestingness is the set of criteria used to separate the handful of patterns actually worth a person's attention from the flood of ones that aren't. The support, confidence, lift, leverage, and conviction measures that do most of this filtering for association rules are already covered with a full worked example in the lesson on association rule mining; this lesson focuses on what those measures leave out — the difference between a statistically strong pattern and a genuinely interesting one.

Objective Measures vs. Subjective Measures

It's worth organizing interestingness measures along a line the source material for this topic tends to blur: objective measures are computed purely from the data itself, without any reference to what a person already believes or wants — support, confidence, lift, leverage, and conviction all fall here, since each is a pure calculation over transaction counts. Subjective measures instead depend on the specific analyst's existing beliefs and goals, which is precisely why the same pattern can be genuinely interesting to one team and worthless to another: a rule showing that new parents buy diapers more than average is statistically real but surprises no one, while a rule showing that a specific, non-obvious pair of unrelated products sell together consistently is far more actionable precisely because it violates what anyone would have expected in advance — its interest comes from unexpectedness, not from a higher support or confidence score. This is why domain knowledge matters as much as it does: judging whether a pattern is unexpected, novel, or actionable requires knowing what "expected" already looks like in that specific field.

Minimum Description Length: Interestingness as Compression

One more principled way to formalize "interesting" comes from information theory: a pattern is interesting to the extent that using it makes the dataset cheaper to describe. This is easiest to see with a simplified illustration. Suppose 40 of 100 transactions contain both bread and butter. Describing those 40 transactions without referencing any pattern requires recording two separate item-flags per transaction — a "bread present" bit and a "butter present" bit — for 2 × 40 = 80 bits total. Describing the same 40 transactions using a discovered pattern P = {bread, butter} instead requires a small fixed cost to define P once (say 4 bits), plus a single "does this transaction match P" flag per transaction (40 bits):

Without the pattern: 2 bits x 40 transactions        = 80 bits
With the pattern:     4 bits (define P) + 40 bits    = 44 bits

The pattern-based description is shorter — 44 bits instead of 80 — which is exactly what the Minimum Description Length (MDL) principle looks for: a pattern earns its keep by making the data it applies to more compact to represent, not merely by appearing frequently. (Real implementations compute this compression more rigorously using formal coding theory; the arithmetic above is a simplified illustration of the underlying logic, not a specific published algorithm's exact formula.) This principle also explains why simpler patterns are generally preferred over more complex ones covering the same cases: a pattern with fewer conditions costs less to define in the first place, so it needs less data-compression benefit to justify its own existence.

Redundancy: When Discovered Rules Say the Same Thing Twice

A mining run frequently returns multiple rules that are all really the same underlying signal restated with a minor variation. If {bread} → {butter} already has strong support and confidence, a longer rule like {bread, jam} → {butter} will often show nearly identical numbers simply because adding jam to the antecedent barely changes which transactions qualify — jam doesn't meaningfully explain the bread-butter relationship, it's just riding along with it. A genuinely non-redundant discovery adds real, independent information beyond what a simpler rule already captured; a redundant one adds length and apparent specificity without adding anything a decision-maker didn't already know from the shorter rule.

Ethical Use of Discovered Patterns

Because interesting patterns are frequently discovered in personal data — purchase histories, health records, financial activity — the same privacy and data-protection obligations covered in the lesson on the challenges of data mining apply directly here: a pattern being statistically and practically interesting does not by itself make it acceptable to act on or disclose without regard for the people it describes.

Where Interestingness Actually Gets Applied

The application areas most commonly used to illustrate interestingness — market basket analysis, healthcare diagnosis support, cybersecurity anomaly detection — are covered with their own concrete worked examples in the lesson on data mining applications; what interestingness measures determine, in every one of those applications, is simply which of the many patterns a mining run turns up are worth a person's limited attention at all.