KDD: Knowledge Discovery in Databases

KDD (Knowledge Discovery in Databases) is the overall process of discovering useful knowledge from large amounts of data, using data mining techniques to find patterns, relationships, and meaningful information along the way. It's an interdisciplinary field, drawing on artificial intelligence, machine learning, pattern recognition, databases, statistics, expert systems, and data visualization — KDD is what ties all of these together into a systematic process for turning raw data into something an organization can actually act on.

An earlier lesson in this series introduced KDD as a seven-step process (Data Cleaning, Data Integration, Data Selection, Data Transformation, Data Mining, Pattern Evaluation, Knowledge Presentation). That version is a widely taught simplification. This lesson covers the process in its original, more detailed form, as laid out by Gregory Piatetsky-Shapiro, Usama Fayyad, and Padhraic Smyth in the paper that introduced the term — which breaks the same journey into nine steps. Different sources compress or expand this count (you'll see five-, seven-, and nine-step versions depending on the textbook), but they all describe the same underlying journey; the step count is an editorial choice, not a disagreement about what KDD actually involves.

The nine-step version is worth knowing because it makes two things explicit that the seven-step summary compresses: an upfront step for understanding the business or research problem (similar in spirit to CRISP-DM's Business Understanding phase, covered earlier in this series), and a three-way split of "Data Mining" into choosing a task, choosing an algorithm, and actually applying it.

The process itself is interactive and iterative — steps can be, and often are, repeated. It begins with understanding the problem and ends with putting the discovered knowledge to use in a real application.

The Nine Steps of the KDD Process

1. Understanding the Application Domain

This first step is where the people working on the project build a shared understanding of the problem to be solved, the end user's goals, and the environment the resulting system will operate in. Getting this right shapes every decision that follows — which data gets collected, which methods are appropriate, and which algorithms are even worth considering.

2. Selecting and Creating the Dataset

With the objectives defined, the next step is assembling the data that analysis will actually run on: identifying what's available, collecting what's relevant, and combining data from different sources into a single working dataset.

The quality of this dataset matters more than it might seem, because data mining can only learn patterns that are actually present in the data it's given. If an important attribute is missing from the dataset entirely, no amount of clever analysis later in the process can recover it.

3. Data Preprocessing and Cleaning

Here the data is cleaned and made ready for analysis. This typically includes:

• Handling missing values
• Removing noise and outliers
• Correcting inconsistent data

Statistical techniques or data mining algorithms are sometimes used to improve data quality directly — for instance, a prediction model can be used to estimate a reasonable value for a field that's missing, rather than simply discarding the record.

4. Data Transformation

The data is converted into a format suitable for data mining. Common techniques include:

• Feature selection — keeping only the important attributes
• Feature extraction — creating new, more useful attributes from existing ones
• Data sampling — working with a representative subset of records rather than the full dataset
• Discretization — converting continuous numerical data into categories (for example, turning an age value into an age bracket)

This step matters more than its short description suggests: poor transformation choices here limit what every later step can discover, no matter how good the mining algorithm is.

5. Choosing the Type of Data Mining Task

Now the project decides what kind of data mining it actually needs. There are two broad goals:

Prediction — estimating future or unknown values from existing data, through techniques like classification and regression. This is usually called supervised learning, because the algorithm is trained on examples where the correct answer is already known.

Description — finding patterns and relationships within the data itself, through techniques like clustering, association rules, and data visualization. This is usually called unsupervised learning, because there's no predefined "correct answer" to train against — the algorithm surfaces structure on its own.

(These map directly onto the seven techniques covered earlier in this series: classification, regression, and prediction fall under Prediction; clustering, association rule mining, and sequential pattern mining fall under Description.)

6. Selecting the Data Mining Algorithm

Once the task is chosen, the next decision is which specific algorithm to use — and different algorithms trade off different things. A neural network, for example, can offer high prediction accuracy but is hard to interpret; a decision tree is usually less accurate but far easier for a human to read and explain. Each algorithm also comes with its own parameters and training methods, such as cross-validation for testing how well it will generalize to new data.

7. Applying the Data Mining Algorithm

The chosen algorithm is run against the prepared dataset — often more than once, with parameters adjusted between runs to improve performance. In a decision tree, for instance, this might mean changing the minimum number of records required at a node before it's allowed to split further.

8. Evaluation and Interpretation

Once results come back, the patterns that were found have to be evaluated and interpreted, not just accepted at face value. This step checks whether the results actually meet the original objectives from step 1, whether the model is accurate and useful, and whether the results are something a person can actually understand and explain. Whatever knowledge survives this check gets documented for future reference.

9. Using the Discovered Knowledge

The final step is putting that knowledge to work in real systems — improving business strategies, supporting decisions, or updating existing processes. After deployment, results are monitored over time, and because KDD is iterative, the whole process may be run again as new data becomes available.

Related Concepts

Step 1 here plays the same role as Business Understanding in CRISP-DM, and step 5's Prediction/Description split lines up with the classification-vs-clustering distinction from the Data Mining Techniques lesson — different frameworks, same underlying ideas, described at different levels of detail.