The Data Mining Process: From Raw Data to Deployed Model

A full data mining project is not "run an algorithm on a spreadsheet" — it is a pipeline of distinct stages, most of which have nothing to do with the algorithm itself and everything to do with getting the data, the problem definition, and the evaluation right. This lesson walks through that pipeline as a single coherent sequence, using one running scenario throughout to keep every stage concrete, and points to the lessons in this series that cover each stage's techniques in depth rather than repeating them here.

The Scenario Used Throughout This Lesson

A telecom company wants to identify, in advance, which customers are likely to cancel their subscription within the next 30 days — a problem known as churn prediction.

Stage 1: Problem Definition

Before touching any data, the target has to be defined precisely: not "understand our customers" but "predict whether will_cancel_within_30_days is True or False for each current customer." This framing already tells you which function this project needs — classification, covered in the lesson on categories of functions in data mining — well before any algorithm is chosen.

Stage 2: Data Collection

Data is gathered from every source that might carry a signal: the billing system (payment history, plan type), the CRM (support tickets, complaint history), and usage logs (call minutes, data usage, app logins). At this stage the data is typically scattered across systems that were never designed to be joined together.

Stage 3: Data Cleaning and Integration

Records from billing, CRM, and usage logs are merged on customer ID, missing values are handled, duplicate records are removed, and inconsistent formats (a phone number stored as text in one system and as a number in another) are reconciled — the full set of techniques for this stage is covered in the lesson on data preprocessing techniques.

Stage 4: Data Transformation and Reduction

Categorical fields (plan type, region) are encoded, and numeric fields with very different scales (monthly bill in dollars versus call minutes in the thousands) are normalized so that no single field dominates the model purely because of its scale — covered in the lesson on data preprocessing techniques and, for methods such as robust and log scaling, the lesson on normalization in data mining. Redundant or irrelevant features are removed at the same stage, using the feature-selection and dimensionality-reduction techniques covered in the same preprocessing lesson.

Stage 5: Exploratory Data Analysis (EDA)

Before any model is trained, the cleaned data is examined directly — churn rate by plan type, churn rate by tenure, correlation between support-ticket volume and cancellation. This step often surfaces a data quality issue that sends the project back to Stage 3, which is why the pipeline is better understood as iterative than strictly linear (more on this below).

Stage 6: Model Selection and Training

Since the target is a yes/no outcome, a classification algorithm is the right family of choice — any of the algorithms covered in the lessons on C4.5, rule-based classification, Naive Bayes (in the lesson on data mining projects), or Support Vector Machines. The data is split into training and test sets before training begins; because churn is typically rare (most customers do not cancel in any given month), the stratified sampling approach covered in the lesson on partitioning methods in data mining — which preserves the same proportion of churners in both the training and test sets — matters more here than a plain random split would.

Stage 7: Evaluation

The trained model is scored against the held-out test set using the confusion-matrix-based metrics (precision, recall, F1) covered with a full worked example in the lesson on data mining operators, and validated further with the cross-validation approach and overfitting checks covered in the lesson on major issues in data mining. For a churn model specifically, recall (catching as many actual churners as possible) is usually weighed more heavily than precision, since the cost of a missed churner (lost revenue) tends to be higher than the cost of a false alarm (an unnecessary retention offer).

Stage 8: Deployment and Monitoring

Once the model performs acceptably, it is deployed to score current customers on an ongoing basis. This is not the end of the pipeline: customer behavior shifts over time (a new competitor enters the market, a pricing change goes into effect), which is exactly the concept drift problem covered in the lesson on data stream mining — a model deployed once and never re-evaluated will quietly degrade as the population it was trained on stops matching the population it's scoring.

Why This Pipeline Is a Loop, Not a Straight Line

The eight stages above are numbered for clarity, but real projects move backward through them constantly: an EDA finding sends the project back to cleaning, a poor evaluation result sends it back to feature transformation or a different model choice entirely, and ongoing monitoring can trigger a full retraining cycle starting again from Stage 2 with fresh data. Treating this as a strict one-way sequence is the most common way a first data mining project underestimates how much of the total effort — commonly cited as the majority of it — is spent on Stages 2 through 5, well before any algorithm is trained at all.