Tasks and Functionalities of Data Mining

Data mining is not one operation — it is a toolbox of distinct tasks, each answering a different kind of question about the same underlying data: What does this group look like? What tends to happen together? What causes what? Which record doesn't belong? This lesson maps out that toolbox as a whole, so that the more detailed, fully worked-out treatments of individual techniques found elsewhere in this series slot into a clear overall picture rather than feeling like an unordered list of unrelated topics.

What Counts as Data Mining, and What Doesn't

Data mining, precisely, is the step where an algorithm searches already-prepared data for patterns — groupings, relationships, unusual records, or sequences — automatically or semi-automatically. Data collection, cleaning, and reporting are necessary parts of a real project, but they are not the mining step itself; they are the surrounding preparation and communication work that makes mining possible and useful, a distinction covered in full in the lesson on KDD versus data mining. Keeping this boundary in mind matters here because the functionalities below are all things a mining algorithm does — not things a data pipeline does before or after it runs.

Data Mining vs. Data Analysis

The two terms are often used interchangeably, but they lean in different directions. Data analysis typically starts from a specific hypothesis and uses statistical methods to test it — "does average spend differ between two regions?" is a question data analysis answers directly. Data mining is more often exploratory: it searches for patterns without a hypothesis stated in advance, discovering relationships a person may not have thought to ask about at all. This maps onto the verification-versus-discovery distinction discussed in the KDD lesson — data analysis leans toward verification, data mining leans toward discovery — though in practice the two overlap heavily and a real project usually uses both.

Two Broad Categories: Descriptive and Predictive

Every specific functionality below falls into one of two broad categories, covered in full in their own lesson: descriptive tasks characterize the general properties of the data as it already exists (counts, averages, groupings, relationships), while predictive tasks use existing data to infer values that are not yet known — a future sales figure, an unlabeled record's category. The functionalities that follow are organized under whichever of these two goals they primarily serve.

The Core Functionalities

1. Class and Concept Description

Before comparing or predicting anything, it often helps simply to describe a group. Data characterization summarizes the typical features of a single class or dataset — average sales, total customer count, the typical profile of a category — while data discrimination compares two or more classes against each other to find what actually distinguishes them, such as comparing buyers against non-buyers. Both are descriptive tasks, and both are covered with fully worked numeric examples (including the d-weight measure used to quantify how strongly an attribute discriminates between classes) in the lesson on class comparison methods.

2. Mining Frequent Patterns

A frequent pattern is any combination that shows up often enough in the data to be worth noting, and it comes in three structural flavors depending on what kind of combination is being counted. A frequent itemset is a set of items that often occur together with no regard for order, such as milk and bread appearing in the same shopping basket. A frequent subsequence is a pattern that recurs in a specific order over time, such as a customer buying a phone and later buying a phone case. A frequent substructure is a recurring pattern in more complex, structured data — a shape that shows up repeatedly within trees or graphs, such as a particular sub-molecule appearing across many chemical compounds. All three are searched for using the same underlying primitives — support and confidence thresholds, and the counting techniques applied to transaction data — covered in depth with a fully worked support/confidence calculation in the lesson on data mining task primitives.

3. Association Analysis

Association analysis, often called market basket analysis, is the specific application of frequent-itemset mining that looks for "if this, then that" relationships between items — customers who buy coffee also tend to buy biscuits, for instance. Its two defining measures are support (how often the items appear together across all transactions) and confidence (how often the "then" item appears given that the "if" item is present). This technique gets a complete worked treatment, including how support and confidence are computed on real transaction data, in the dedicated lesson on market basket analysis.

4. Classification

Classification assigns each record to one of a set of predefined categories — spam or not spam, approved or denied — by training a model (a decision tree, a set of if-then rules, or a neural network, among other approaches) on records whose true category is already known. This is a predictive task, and it is covered with a full worked example (including a real confusion matrix and accuracy score computed on a bank-loan dataset) in the lesson on classification and prediction.

5. Prediction

Prediction estimates a value that is not yet known, and comes in two forms depending on what kind of value is being estimated. Numeric prediction estimates a continuous number, typically using regression — forecasting next quarter's sales, for instance — and is covered with verified regression examples in the lesson on regression in data mining. Class prediction estimates a category rather than a number and is, in practice, the same operation as classification applied to a case where the true label is not yet available; the distinction between the two names is more about framing (are you evaluating a known-answer test set, or genuinely predicting an unknown future case?) than about a different underlying technique.

6. Cluster Analysis

Cluster analysis groups similar records together without any predefined categories at all — the groupings themselves are discovered from the data's structure, such as segmenting customers by buying behavior with no prior labels for what the segments should be. This is the core unsupervised counterpart to classification's supervised approach, and it is covered in depth (hierarchical clustering, density-based clustering, and BIRCH's summarize-then-cluster strategy) across several dedicated lessons elsewhere in this series.

7. Outlier Analysis

Outlier analysis identifies records that deviate sharply from the rest of the data — a sudden, unusually large transaction on a bank account, for instance — which matters because outliers can signal errors, fraud, or genuinely rare but important events, and because failing to account for them can distort other analyses run on the same data. That last point is worth demonstrating directly rather than stating abstractly, because it connects outlier analysis to the very next functionality on this list.

8. Evolution and Deviation Analysis

Evolution analysis studies how data changes over time — a sales trend across months, or growth in website traffic — treating time itself as a dimension of the pattern being searched for, rather than analyzing a single static snapshot of the data. It relies on the same descriptive and predictive functionalities above, applied repeatedly across a time-ordered sequence of data rather than once.

9. Correlation Analysis

Correlation analysis measures the strength and direction of the relationship between two variables — for example, whether an increase in advertising spend tends to accompany an increase in sales, and how strongly. A positive correlation means the two variables tend to move together; a negative correlation means one tends to rise as the other falls. This gets a full dedicated treatment, including how correlation is computed and what its numeric value means, in its own lesson — but it is worth previewing here specifically because of how easily it interacts with outlier analysis.

Why These Functionalities Are Not Independent: A Worked Example

The functionalities above are often presented as a flat list, which can obscure how directly they interact in practice. Take a small dataset of 15 customers with their age and monthly spend, computed with age and spend genuinely related, plus one customer with an unusually large spend value:

import pandas as pd

print(df)
print("mean spend:", round(df["monthly_spend"].mean(), 2))
print("correlation(age, spend) including outlier:", round(df["age"].corr(df["monthly_spend"]), 3))

Output (the relevant figures):

mean spend: 1358.52
correlation(age, spend) including outlier: 0.117

A correlation of 0.117 looks like essentially no relationship between age and spending. But running outlier analysis on the same data first — flagging any record whose z-score exceeds 2 — identifies exactly one problem record:

z = (df["monthly_spend"] - df["monthly_spend"].mean()) / df["monthly_spend"].std()
print(z.round(2).to_dict())

Output:

{..., 3: 3.52, ...}

Row 3 has a z-score of 3.52 — a monthly spend of 5,200 against a typical range of roughly 700–1,400 for every other customer in the data. Recomputing the correlation with that single flagged record removed changes the picture entirely:

df_no_outlier = df.drop(index=3)
print("correlation(age, spend) excluding outlier:", round(df_no_outlier["age"].corr(df_no_outlier["monthly_spend"]), 3))

Output:

correlation(age, spend) excluding outlier: 0.882

One outlier changed the measured correlation from 0.117 (essentially nothing) to 0.882 (a strong relationship) — the exact same underlying data, with a completely different conclusion depending on whether outlier analysis ran first. This is the practical reason these functionalities are described together rather than as isolated techniques: descriptive statistics, outlier analysis, and correlation analysis are not independent tools applied in any order to a dataset — the output of one directly changes what the next one finds, and a data mining project that treats them as separable steps run in isolation risks reporting a strong relationship that is really just one bad data point, or missing a real one that a single outlier happened to be masking.