Data Mining Implementation Process

Data Mining Process

Many industries — manufacturing, marketing, aerospace, and chemicals among them — rely on data mining to improve business performance and decision-making. But without a shared process, one team's data mining project can look nothing like another's, with wildly inconsistent quality and easy-to-miss blind spots. The Cross-Industry Standard Process for Data Mining (CRISP-DM) exists to solve exactly that: a structured, repeatable methodology for running a data mining project end to end, independent of whatever specific tools or algorithms a team happens to use.

CRISP-DM was conceived in 1996 and developed into a full methodology under a European Commission-funded project starting in 1997, led by a consortium of organizations that included NCR, SPSS (then known as Integral Solutions Ltd), Daimler-Benz, and the insurance company OHRA. Despite its age, it remains the most widely used data mining and analytics methodology in industry today — the same methodology referenced briefly in this series' earlier lesson on the practical data mining workflow.

The Six Phases of CRISP-DM

CRISP-DM organizes a project into six phases:

1. Business Understanding
2. Data Understanding
3. Data Preparation
4. Modeling
5. Evaluation
6. Deployment

The most important thing to understand about this list is that it isn't a strict, one-way sequence. CRISP-DM is cyclical: insights from Data Understanding often send you back to revise Business Understanding, and the Evaluation phase frequently sends a project back to Data Preparation or Modeling rather than straight through to Deployment.

1. Business Understanding
        ↕
2. Data Understanding
        ↕
3. Data Preparation
        ↕
4. Modeling
        ↕
5. Evaluation ── often loops back to phase 1, 3, or 4
        ↓
6. Deployment

1. Business Understanding

This phase focuses on understanding the business problem and the project's objectives. Its main output is a business problem translated into a data mining problem, plus a plan for reaching that goal.

Key tasks:

• Determine Business Objectives — understand what the organization actually wants to achieve, identify the factors that could affect the project's results, and state the business goal clearly.
• Assess the Situation — analyze available resources, project constraints, assumptions, and risks before committing to an approach.
• Determine Data Mining Goals — translate the business goal into a specific, technical objective that a model can actually be built to achieve.
• Produce a Project Plan — create a roadmap for the project, and identify the tools, techniques, and steps needed to complete it.

Example

This translation step matters more than it looks: "increase sales" isn't something an algorithm can act on directly. A data mining goal has to be specific and measurable enough that you can later judge, objectively, whether the project succeeded.

2. Data Understanding

This phase begins with collecting data and learning about its structure and quality, so the team understands what it's actually working with before committing to a modeling approach.

Key tasks:

• Collect Initial Data — gather data from available sources, such as databases or files, and load it into whatever analysis tools the project will use.
• Describe Data — examine basic properties: the number of records, the number of attributes, and the data types involved.
• Explore Data — use visualization, queries, and statistical analysis to look for patterns, such as the distribution of a variable, summary statistics, or relationships between attributes.
• Verify Data Quality — check for missing values, errors, inconsistencies, or duplicate records.

3. Data Preparation

Data preparation is where raw, messy data becomes a clean dataset a model can actually be trained on. It's routinely the most time-consuming phase of a data mining project — many practitioners report that data preparation alone consumes the majority of total project time, with some citing figures as high as 80–90%.

Key tasks:

• Select Data — choose the datasets and attributes actually required for the analysis; more data isn't automatically better if most of it is irrelevant to the goal.
• Clean Data — handle missing values and remove incorrect or duplicate records.
• Construct Data — create new variables or features from existing data. For example, calculating total purchase value from quantity and price.
• Integrate Data — combine data from different sources, tables, or systems into one consistent dataset.
• Format Data — convert data into whatever format the modeling tool requires. For example, converting text categories into numerical values.

4. Modeling

In this phase, machine learning or statistical models are applied to the prepared data to identify patterns and make predictions.

Key tasks:

• Select a Modeling Technique — choose an algorithm suited to the data mining goal, such as a decision tree, a neural network, a classification algorithm, or a regression model.
• Generate a Test Design — split the dataset into a training set (used to build the model) and a testing set (used to evaluate it on data it hasn't seen before).
• Build the Model — run the selected algorithm on the prepared training data.
• Assess the Model — evaluate how well the model performs technically (its accuracy or error rate), and sanity-check whether the results make sense from a business perspective.

5. Evaluation

It's easy to conflate this phase with the "Assess the Model" task in Modeling, but they check different things. Assessing the model in phase 4 asks a technical question — is this model statistically accurate? Evaluation asks a business question — does a statistically accurate model actually solve the problem defined back in phase 1? A churn-prediction model can be highly accurate and still fail here, for instance, if its predictions arrive too late for the retention team to act on them.

Key tasks:

• Evaluate Results — measure how well the model achieves the original business objectives, not just its technical performance.
• Review the Process — look back across every phase to confirm nothing important was missed or done out of order.
• Determine Next Steps — decide whether to deploy the model, improve it, collect more data, or start a new data mining project altogether.

6. Deployment

In the deployment phase, the final model and its insights get put to work in real business operations. Deployment can be as simple as a written report handed to decision-makers, or as complex as a model scoring live transactions in real time — the right level of complexity depends entirely on the business need.

Key tasks:

• Plan Deployment — decide how the results will actually be used in the organization: generating reports, integrating the model into existing business systems, or feeding its predictions directly into a decision-making workflow.
• Plan Monitoring and Maintenance — track the model's performance over time, and update it as business conditions or the underlying data change. A model trained on last year's customer behavior can quietly go stale.
• Produce a Final Report — document the project's results, process, and findings.
• Review the Project — assess what worked well and what could be improved on the next project.

How CRISP-DM Relates to the KDD Process

This series' first lesson introduced the Knowledge Discovery in Databases (KDD) process, a more academic, seven-step description of the same overall journey from raw data to knowledge. CRISP-DM covers the same ground but frames it around a real project team, adding explicit business-facing phases that KDD doesn't call out on its own:

The two aren't competing standards — CRISP-DM is best thought of as the KDD idea made concrete enough for a project team to actually follow, with Business Understanding added at the front because, in practice, no data mining project should start with the data at all — it should start with the question.