Data Mining Systems Architecture

Earlier lessons in this series covered data mining as a process — the KDD steps, the CRISP-DM project methodology. This lesson looks at it from a different angle: the actual software components a data mining system is built from, and how data flows between them when someone actually runs a mining task.

A data mining system is made up of several components that work together to collect, store, process, analyze, and present data. Together, these form what's usually called the data mining system architecture.

Data Sources

Data sources are where the data is originally collected: databases, data warehouses, the World Wide Web, text files, spreadsheets, and other documents. For data mining to produce anything useful, it needs a large amount of historical data to learn patterns from — which is why organizations typically consolidate it into databases or data warehouses rather than mining scattered files directly.

A single data warehouse might combine data pulled from multiple databases, spreadsheets, and text files. Even something as ordinary as an Excel sheet can turn out to hold useful information, and increasingly, the internet and the web are data sources in their own right.

Data Preprocessing: Cleaning, Integration, and Selection

Data pulled from multiple sources and formats rarely arrives ready to use — it usually has errors, missing values, or irrelevant information mixed in. Before it can be mined, it has to be:

• Cleaned — errors, missing values, and incorrect entries are removed or corrected.
• Integrated — data from different sources is combined into a single, consistent dataset.
• Selected — only the data actually relevant to the analysis is kept.

This preprocessing stage is where the accuracy and meaningfulness of everything downstream gets decided. It's also often the most labor-intensive part of the architecture, since different data sources tend to need different cleaning and integration methods.

Database or Data Warehouse Server

The database or data warehouse server stores the data once it's been processed and is ready for analysis. It manages that data and retrieves whatever a user's data mining task requests — effectively acting as the main storage layer the rest of the system reads from.

Data Mining Engine

The data mining engine is the core of the system — the component that actually performs the analysis. It's typically made up of several modules, each handling a different kind of mining task:

• Association — finding relationships between data items
• Characterization — summarizing the general features of a dataset
• Classification — assigning data into predefined categories
• Clustering — grouping similar data together
• Prediction — forecasting future values
• Time-series analysis — analyzing data collected over time

These map directly onto the techniques covered earlier in this series — the engine is where classification, clustering, association rule mining, and the rest actually run against the stored data.

Pattern Evaluation Module

Not every pattern the mining engine finds is worth keeping. The pattern evaluation module checks the patterns produced during mining and determines which ones are actually interesting or useful, using evaluation measures or thresholds — support, confidence, and lift for association rules, for instance. Patterns that don't clear the bar are discarded.

This module works closely with the mining engine, in both directions: it filters the engine's output, and its evaluation criteria can also help steer which patterns the engine looks for in the first place. The point is to keep the system focused on patterns that matter rather than flooding the user with statistically present but practically meaningless noise.

Knowledge Base

The knowledge base stores background information that helps guide and improve the mining process — domain knowledge, user preferences, results from previous mining runs, and rules or patterns discovered earlier. Rather than sitting in the data's path, it works alongside both the mining engine and the pattern evaluation module, feeding them context and getting updated with newly discovered knowledge in return.

Graphical User Interface (GUI)

The GUI is how a person actually operates the system: submitting queries or tasks, controlling how the mining process runs, and viewing the results and visualizations it produces. It hides the technical machinery underneath — the engine, the evaluation module, the storage layer — so users can work with the system without needing to understand how any of it works internally.

How the Pieces Fit Together

It's tempting to read the sections above as a strict pipeline — sources feed the server, the server feeds the engine, and so on — but the GUI actually sits on top of the whole system rather than at the end of it. A user's query enters through the GUI, travels down through the engine and evaluation module (consulting the knowledge base along the way), and the results travel back up to the same GUI the query came from.

Data Sources → Preprocessing → Database / Data Warehouse Server
                                            │
                                            ▼
                              Data Mining Engine ⇄ Pattern Evaluation Module
                                            ▲                  ▲
                                            └── Knowledge Base ┘
                                            │
                                            ▼
                              Graphical User Interface (GUI)
                                     ↑ query in / results out ↓

Related Concepts

The cleaning, integration, and selection steps described here are the same ones covered as part of the KDD process and the CRISP-DM Data Preparation phase earlier in this series — this lesson just shows where they physically sit inside a running system, rather than as a stage in a project plan.